=> Scenario : Secondary application tier node in EBS database was restarted without bringing down EBS services. Managed servers on second node are in FAILED_NOT_RESTARTABLE state in weblogic console. Managed servers failed to start either from console or from backend.
=> Cause : Issue is due to existence of lock files due to improper shutdown of managed servers. So we should clear the lock files and start the managed servers
=> Error 01 :
<Nov 1, 2022 11:00:51 AM GST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Nov 1, 2022 11:00:55 AM GST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/tmp/oacore_server2.lok. Server may already be running
<Nov 1, 2022 11:00:55 AM GST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
=> Fix :
Backup and delete contents of below tmp directory
=> /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/tmp
=> Error 02 :
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
=> Fix :
Backup and remove the below DAT files. I moved them with .OLD extension
[oracle@ebs-host-002 default]$ pwd
/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/data/store/default
[oracle@ebs-host-002 default]$ ls -ltr
total 2560
-rw-r-----. 1 oracle oinstall 1049088 Oct 17 06:41 _WLS_OACORE_SERVER2000000.DAT.OLD
-rw-r-----. 1 oracle oinstall 1049088 Nov 1 14:45 _WLS_OACORE_SERVER2000000.DAT
[oracle@ebs-host-002 default]$ cd ../diagnostics/
[oracle@ebs-host-002 diagnostics]$ pwd
/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/data/store/diagnostics
[oracle@ebs-host-002 diagnostics]$ ls -ltr
total 2560
-rw-r-----. 1 oracle oinstall 1052672 Oct 17 06:41 WLS_DIAGNOSTICS000000.DAT.OLD
-rw-r-----. 1 oracle oinstall 1052672 Nov 1 11:45 WLS_DIAGNOSTICS000000.DAT
[oracle@ebs-host-002 diagnostics]$
Error 03 :
<Nov 1, 2022 11:33:07 AM GST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
Fix :
Backup and delete the lok file under below directory. I moved it with .old extension
[oracle@ebs-host-002 ldapfiles]$ pwd
/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server2/data/ldap/ldapfiles
[oracle@ebs-host-002 ldapfiles]$ ls -ltr
total 544
-rw-r-----. 1 oracle oinstall 1104 Oct 5 05:45 EmbeddedLDAP.index
-rw-r-----. 1 oracle oinstall 108 Oct 5 05:45 EmbeddedLDAP.delete
-rw-r-----. 1 oracle oinstall 0 Nov 1 11:35 EmbeddedLDAP.lok.old
-rw-r-----. 1 oracle oinstall 0 Nov 1 11:44 EmbeddedLDAP.lok
-rw-r-----. 1 oracle oinstall 8 Nov 1 11:54 EmbeddedLDAP.trpos
-rw-r-----. 1 oracle oinstall 8 Nov 1 11:54 EmbeddedLDAP.twpos
-rw-r-----. 1 oracle oinstall 269401 Nov 1 11:54 EmbeddedLDAP.tran
-rw-r-----. 1 oracle oinstall 199020 Nov 1 11:54 EmbeddedLDAP.data
[oracle@ebs-host-002 ldapfiles]$
=> Start the managed servers from node-02
[oracle@ebs-host-002 ldapfiles]$ cd $ADMIN_SCRIPTS_HOME
[oracle@ebs-host-002 scripts]$ ./admanagedsrvctl.sh start oacore_server2
=> Apply the same fix for all the managed servers on second node
No comments:
Post a Comment