apps-dba journal

A working journal for Oracle DBAs.

March 30, 2022

fs_clone

EBS R12.2 · ADOP

After AD-TXK Delta 12, adop phase=fs_clone can fail with s_appltmp not set correctly. The dual file system, why temp must be node-local, what Delta 12 tightened, and the fix: backup context → edit → mkdir → AutoConfig → fs_clone — on every node.

Dual file system — quick recap

Oracle EBS R12.2 requires two independent application-tier file systems so you can patch online without taking the run edition away from users:

  • fs1 (run edition) — active APPL_TOP / INST_TOP serving users.
  • fs2 (patch edition) — idle or receiving patches; brought in sync from run via fs_clone when the two trees diverge or after abort / config change.
  • fs_ne (non-editioned) — shared data across both editions; never copied during patching or fs_clone.

In a normal ADOP cycle, PREPARE, APPLY, FINALIZE, and CLEANUP all run with users fully online. Only CUTOVER — the edition switch between fs1 and fs2 — needs a brief service restart (often on the order of tens of minutes). That is the online-patching model that separates R12.2 from earlier EBS releases.

fs_clone is not one of those five phases. It is an out-of-cycle resynchronization: recreate the patch edition as a complete, synchronized copy of the run edition after configuration changes, after abort, or when the two file systems have drifted far enough that a full sync is cheaper than chasing every delta.

What fs_clone does / when to run

adop phase=fs_clone recreates the patch edition file system as a complete, synchronized copy of the run edition. Treat it as out-of-cycle maintenance, not as a step you run in every patch cycle.

When to run it:

  • After Autoconfig on the run edition (and after cutover when relevant) — so configuration changes reach the patch edition before the next PREPARE.
  • After adop phase=abort — to restore the patch file system to a clean, synchronized state.
  • When run and patch have diverged significantly — a full sync can be faster than applying every accumulated delta.
  • Before cloning the entire EBS environment — both file systems should be aligned.

What it does internally, in order: validate the run edition context file (including s_appltmp — this is where the Delta 12 error fires); back up the current patch APPL_TOP; copy the run APPL_TOP to the patch location, preserving permissions, ownership, and timestamps; generate the patch edition context from the run context; update patch-specific paths (fs1 → fs2 substitution).

Because fs_clone backs up the entire patch APPL_TOP before overwriting it, plan for at least about 25 GB free on the patch file-system mount and about 15 GB in /tmp. On large APPL_TOPs the phase can take 30–90 minutes.

s_appltmp must be node-local

s_appltmp is an XML parameter in the node’s context file. It defines the application temporary directory — where EBS writes temporary files during concurrent processing, form operations, and ADOP phases. Context file location:

$INST_TOP/appl/admin/$CONTEXT_NAME.xml
# e.g. /u01/oracle/fs1/inst/apps/PROD_appnode1/appl/admin/PROD_appnode1.xml

Correct vs wrong:

<!-- Correct — node-local -->
<parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter>

<!-- Incorrect — shared NFS/cluster -->
<parameter name="s_appltmp" itsm="false">/shared/nfs/appltmp</parameter>

In a multi-node application tier, each node runs independent concurrent manager processes. When s_appltmp points at a shared NFS location, every node writes temporary files into the same directory at once. That produces file collisions (same temp name, one overwrite), PID and lock-file conflicts, and ADOP concurrency failures when multiple ADOP processes across nodes share temporary state.

The correct value is always node-local under $INST_TOP — which is per-node by definition — typically $INST_TOP/appltmp (for example /u01/oracle/fs1/inst/apps/PROD_appnode1/appltmp).

Do not confuse s_appltmp with APPLPTMP. s_appltmp is application-layer temp and must be node-local. APPLPTMP is for PL/SQL concurrent program output and must be shared and reachable from the database server — the opposite requirement.

What Delta 12 changed

AD-TXK Delta releases are cumulative updates to the Application DBA (AD) and Applications Technology (TXK) stack. Under Delta 6, validation was comparatively light: confirm the parameter existed and was non-empty. Shared paths were often tolerated silently.

Delta 12 introduced a stricter ADOP validation framework in txkADOPValidations.pm and related modules: node-local path enforcement for s_appltmp, cross-node value consistency checks, and fail-fast behavior before any file-system operations begin. That validation runs at the start of adop phase=prepare, fs_clone, cutover, and abort.

Systems that ran for years with a shared s_appltmp under Delta 6 hit the error on the first ADOP phase after upgrading to Delta 12. The upgrade itself does not reconfigure your context files — you still have to correct them. Early rejection is intentional: discovering a shared temp mid-clone, with multiple nodes writing the same directory, produces corrupted temps, half-cloned trees, and painful ADOP state recovery.

[ERROR]: The value of s_appltmp is not set correctly in atleast one of the context files.

The fix — step by step

Remediation has three parts: correct the context file, run AutoConfig so the change propagates into dependent runtime configs, then run fs_clone so the patch edition inherits the corrected setup. Repeat the context / AutoConfig work on every application node before you rely on fs_clone — ADOP validates the context on all registered nodes, and a single leftover shared path fails the same check.

1. Back up the context file

find $INST_TOP/appl/admin -name "*.xml" -type f

cp $INST_TOP/appl/admin/$CONTEXT_NAME.xml \
   $INST_TOP/appl/admin/$CONTEXT_NAME.xml.bak_$(date +%Y%m%d_%H%M%S)

2. Check / edit s_appltmp

grep "s_appltmp" $INST_TOP/appl/admin/$CONTEXT_NAME.xml

# Set to node-local (variable form or fully expanded path):
# <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter>
vi $INST_TOP/appl/admin/$CONTEXT_NAME.xml

3. Create appltmp if missing

mkdir -p $INST_TOP/appltmp
chmod 755 $INST_TOP/appltmp
chown applmgr:dba $INST_TOP/appltmp

4. AutoConfig on the run edition

cd $APPL_TOP/ad/bin
./adautocfg.sh

grep -i "AutoConfig completed" $INST_TOP/appl/admin/log/adautocfg*.log | tail -1

AutoConfig reads the updated context and regenerates dependent configs (environment scripts, service configs, and related files). Without this step, the XML edit does not reach the actual runtime environment.

5. Run fs_clone (after steps 1–4 on every apps node)

cd $APPL_TOP/ad/bin
./adop phase=fs_clone
# Expect: fs_clone phase completed successfully

Verification

grep "s_appltmp" $PATCH_INST_TOP/appl/admin/$CONTEXT_NAME.xml
ls -ld $INST_TOP/appltmp

# Definitive: clean prepare with no s_appltmp validation errors
adop phase=prepare

Pre-fs_clone checklist

  • Patch FS free space ≥ ~25 GB — df -h on the patch mount.
  • /tmp ≥ ~15 GB free.
  • No active ADOP — ps -ef | grep adop.
  • Database reachable — tnsping $ORACLE_SID.
  • s_appltmp is $INST_TOP/appltmp on every node.
  • AutoConfig run after the last context edit.
  • No form compiler active; log dir has headroom (≥ ~5 GB).

Optional MOS: 2424019.1 / 2220669.1 (s_appltmp validation on fs_clone / cutover / abort).

Gotchas

  • Fixing only one node is not enough — any registered node still on a shared path fails the same check. If the error persists after “fixing,” use verbose ADOP logging; validation results are reported early, per node.
  • Editing the XML without AutoConfig leaves runtime configs stale — always run adautocfg.sh after the context change.
  • s_appltmp (node-local app temp) ≠ APPLPTMP (shared PL/SQL concurrent output visible to the database).
  • Delta 12 fails early on purpose — better than mid-clone temp collisions and half-synced trees.
  • fs_clone needs substantial free space and wall-clock time; check patch FS and /tmp before you start.

Quick quiz

Choose one answer per question, then submit. You’ll see the correct answer and a short why.

1. In R12.2 dual FS layout, which component is never cloned during patching?

2. What is the correct s_appltmp setting for ADOP?

3. Why does fs_clone often fail right after an AD-TXK upgrade to Delta 12?

4. Correct remediation order for the s_appltmp error?

No comments:

Post a Comment

Oracle DBA Lesson 24 Part 3 — Verify the Database Created by DBCA

DBCA creates and configures a database. Post-creation checks tell you which database you are connected to, the state of its containers,...