Skip to main content

EBS R12.2 : Managed servers failed to start after server crash

By Gowthami | apps-dba.com | E-Business Suite Series

After an unexpected server crash or hard reboot, Oracle E-Business Suite R12.2 managed servers (WebLogic AdminServer, oacore, forms, oafm) often fail to start due to stale lock files, incomplete shutdowns, or corrupted state files. This post covers the diagnostic steps and fixes to get EBS back online quickly.

What You Will Learn: How to diagnose and resolve managed server startup failures in Oracle EBS R12.2 after a server crash, including clearing lock files, cleaning WebLogic state, and restarting services in the correct order.

Common Symptoms

  • adstrtal.sh completes but managed servers show FAILED or UNKNOWN state
  • WebLogic AdminServer starts but oacore/forms/oafm fail to start
  • Error: RUNNING mode failed - server is already started
  • Error: Lock file exists or PID file found

Step 1: Check Running Processes

First, verify that no stale Oracle/WebLogic processes are still running:

# Check for any running WLS or EBS processes
$ ps -ef | grep -E "weblogic|oacore|opmn|java" | grep -v grep

# Kill any stale processes found (replace PID)
$ kill -9 <PID>

Step 2: Remove Lock Files and PID Files

After a crash, lock files may prevent services from starting:

# Navigate to instance home
$ cd $INST_TOP

# Remove WLS lock files
$ find . -name "*.lck" -delete
$ find . -name "*.pid" -delete

# Remove OPMN lock files
$ rm -f $INST_TOP/ora/10.1.3/opmn/logs/*.pid
$ rm -f $INST_TOP/ora/10.1.3/opmn/logs/opmn.pid

Step 3: Clean WebLogic Server State

Corrupted WebLogic state files must be removed before restarting:

# Navigate to domain directory
$ cd $EBS_DOMAIN_HOME

# Remove server state directories (replace SERVER_NAME)
$ rm -rf servers/AdminServer/tmp
$ rm -rf servers/oacore_server1/tmp
$ rm -rf servers/forms_server1/tmp
$ rm -rf servers/oafm_server1/tmp

# Remove .lok files
$ find $EBS_DOMAIN_HOME -name "*.lok" -delete

Step 4: Clean Deadlock Files

$ rm -f $EBS_DOMAIN_HOME/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok

Step 5: Restart in Correct Order

Always start EBS R12.2 services in this sequence:

  1. Database (if not running): startup from SQL*Plus
  2. Database listener: lsnrctl start
  3. Application tier—run as applmgr:
$ source /u01/install/APPS/EBSapps.env run
$ $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/<APPS_PASSWORD>

Step 6: Verify Managed Server Status

# Check status of all services
$ $ADMIN_SCRIPTS_HOME/adstatus.sh

# Or check via WebLogic console
# http://<hostname>:7001/console

Troubleshooting Tips

IssueLikely CauseFix
AdminServer won't startStale .lok or .pid fileRemove lock files, clean tmp
oacore FAILED stateCorrupted server stateRemove servers/oacore*/tmp
OPMN not startingStale opmn.pid fileDelete opmn.pid and restart
Forms server errorsDatabase not accessibleVerify DB and listener are up first

Prevention

  • Always use adstpall.sh for graceful shutdown before patching or rebooting
  • Configure EBS services as OS startup scripts (rc.local or systemd) for automatic, ordered startup
  • Monitor disk space—full filesystems are a leading cause of crash-related startup failures

Master Oracle Exadata

This post is part of our E-Business Suite administration series. Get our comprehensive Exadata guide with architecture, performance tuning, and practical DBA troubleshooting techniques.

Get the Exadata PDF Guide
Labels:

Comments

Post a Comment

Popular posts from this blog

Data Safe - Introduction

Oracle Data Safe - Practical Guide Oracle Data Safe learner guide Oracle Data Safe Assess risk, discover sensitive data, audit activity, and mask safely It focuses on what Data Safe helps you do operationally: review security posture, find risky identities, centralize auditing, locate sensitive data, and produce safer non-production copies. Contents 01 Why Data Safe matters 02 Where it fits 03 Capability map 04 Assessments 05 Activity Auditing 06 Discovery and Masking 07 Operating model 08 First 30 days 09 Knowledge check Section 01 Why Data Safe matters Database security work is often fragmented. One process checks configuration drift, another stores audit logs, another team scans for PII, and another team writes masking logic for test refreshes. Data Safe is useful because it turns those separate jobs into one security workflow. Key idea The best way to think about Data Safe is as a control plane for database security posture: assess the target, identify risky accounts, d...
Post a Comment
Read more

Testing Different Access Paths : Concatenated Index

Oracle Concatenated Indexes - Practical Deep Dive Oracle concatenated index deep dive Concatenated Indexes How composite indexes really work, why column order matters, and when skip scan changes the story Concatenated indexes, also called composite indexes, are easy to explain badly and surprisingly rich to explain well. The usual summary is “Oracle can use the index only when the leading column is present,” but that is only the starting point. To design them properly, you need to think about leading portions, equality versus range predicates, ordering requirements, skip scan eligibility, covering behavior, and whether one composite index can replace several single-column indexes in a given workload. Contents 01 What concatenated indexes are 02 Leading edge and leading portion 03 Why column order matters 04 Skip scan and when it helps 05 Access patterns and plan reading 06 Covering and sort elimination 07 Design rules that actually hold 08 Common mistakes 09 End-to-end demo 1...
Post a Comment
Read more

Database Replay - Real Application Testing (RAT)

Oracle Database Replay and RAT - Practical Deep Dive Oracle Database Replay deep dive Database Replay and Real Application Testing How to validate upgrades, patches, migrations, and risky changes with real workload behavior Database Replay is one of the most practical risk-reduction tools in the Oracle DBA toolbox. Instead of trusting synthetic benchmarks, isolated SQL tests, or intuition, you capture a real production workload, restore a test system to the same logical starting point, replay that workload, and analyze whether performance, errors, timing, and transactional behavior still look safe. Contents 01 What RAT actually is 02 Why Database Replay matters 03 End-to-end workflow 04 Capture design and prerequisites 05 Preprocess, calibrate, replay 06 Reading the results well 07 Pitfalls and unreplayable work 08 Database Replay vs SPA 09 Practical playbooks 10 Knowledge check Section 01 What Real Application Testing actually is Real Application Testing, usually shortened...
Post a Comment
Read more