Wednesday, March 30, 2022

fs_clone issue

EBS R12.2 ADOP fs_clone Troubleshooting Cloud Manager ⏱ ~12 min read

You've just upgraded AD-TXK from Delta 6 to Delta 12, and the very first adop phase=fs_clone you run hits a wall:

Error
[ERROR]: The value of s_appltmp is not set correctly in atleast one of the context files.

This post explains exactly what triggered it, why Delta 12 introduced this check, and how to fix it permanently.


1. The Dual File System Architecture — Quick Recap

Oracle EBS R12.2 requires two independent application-tier file systems to enable zero-downtime patching — the run edition file system (fs1) and the patch edition file system (fs2). A third component, fs_ne (non-editioned), holds data shared across both editions and is never copied during patching.

USERS Online 24x7 Run Edition (fs1) Active — serving users APPL_TOP / INST_TOP s_appltmp → $INST_TOP/appltmp Context: PROD_host1.xml Patch Edition (fs2) Idle / Receiving patches Sync'd via fs_clone ↑ fs_clone fs_ne Non-editioned Shared, never cloned ADOP Patching Cycle ① PREPARE — Sync & create patch edition ② APPLY — Patch the patch edition ③ FINALIZE — Validate & compile ④ CUTOVER — Switch fs1↔fs2 (brief downtime) ⑤ CLEANUP — Drop old edition, reclaim space fs_clone — Re-sync after config change / abort
Zero-Downtime Patching PREPARE, APPLY, FINALIZE, and CLEANUP all run with users fully online. Only CUTOVER — the edition switch — requires a brief service restart (typically 10–30 minutes). This is what makes R12.2 online patching fundamentally different from earlier EBS versions.

2. What fs_clone Actually Does

adop phase=fs_clone recreates the patch edition file system as a complete, synchronized copy of the run edition. It is not part of the standard patching cycle — it is an out-of-cycle resynchronization operation.

When to run it:

  • After configuration changes (autoconfig on run edition) and cutover — to propagate those changes to the patch edition before the next PREPARE
  • After adop phase=abort — to restore the patch file system to a clean, synchronized state
  • When run and patch file systems have diverged significantly — faster than applying all accumulated delta patches
  • Before cloning the entire EBS environment — ensures both file systems are perfectly aligned

What fs_clone does internally, in sequence:

  1. Validates the run edition context file — including s_appltmp. This is where our error fires.
  2. Backs up the current patch APPL_TOP
  3. Copies the run edition APPL_TOP to the patch edition location, preserving permissions, ownership, and timestamps
  4. Generates the patch edition context file from the run edition context file
  5. Updates patch-edition-specific paths in the cloned context file (fs1 → fs2 path substitution)
Resource Requirements fs_clone backs up the entire patch APPL_TOP before overwriting it. Plan for at least 25 GB of free space on the patch file system mount and 15 GB in /tmp. On large APPL_TOPs, this can take 30–90 minutes.

3. s_appltmp: Purpose, Location, and Why Shared Paths Break ADOP

s_appltmp is an XML parameter in the node's context file that defines the application temporary directory — where EBS writes temporary files during concurrent processing, form operations, and ADOP phases.

Context file location:

Path
$INST_TOP/appl/admin/$CONTEXT_NAME.xml # Example on a typical EBS R12.2 install: /u01/oracle/fs1/inst/apps/PROD_appnode1/appl/admin/PROD_appnode1.xml

What the parameter looks like in the context file:

XML — Context File
<!-- Correct — node-local path --> <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter> <!-- Incorrect — shared NFS/cluster path --> <parameter name="s_appltmp" itsm="false">/shared/nfs/appltmp</parameter>

Why a shared path breaks things:

In a multi-node EBS application tier, each node runs independent concurrent manager processes. When s_appltmp points to a shared NFS location, every node writes its temporary files to the same directory simultaneously. This causes:

  • File collisions — two nodes create a temp file with the same name, one overwrites the other
  • PID conflicts — lock files with identical names written by different nodes' processes
  • ADOP concurrency failures — multiple ADOP processes across nodes collide on shared temporary state
s_appltmp vs APPLPTMP — Don't Confuse These s_appltmp is for application-layer temporary files and must be node-local (one per node). APPLPTMP is for PL/SQL concurrent program output files and must be shared and accessible from the database server — the exact opposite requirement. Mixing these up is a common source of confusion.

The correct value is always node-local, under $INST_TOP — which is per-node by definition:

Shell
$INST_TOP/appltmp # Expands to something like: /u01/oracle/fs1/inst/apps/PROD_appnode1/appltmp # Each node's $INST_TOP is unique — so this is inherently node-local

4. What Delta 12 Changed That Surfaced This

AD-TXK Delta releases are cumulative updates to the Application DBA (AD) and Applications Technology (TXK) modules. Delta 12 introduced a significantly enhanced ADOP validation framework in txkADOPValidations.pm and related modules.

AD-TXK Delta 6 ✓ Basic validation — parameter exists? ✓ s_appltmp: non-empty check only ✗ Shared-path enforcement: absent ✗ Shared s_appltmp silently tolerated Upgrade AD-TXK Delta 12 ✓ All Delta 6 checks retained ✓ Node-local path enforcement added ✓ Cross-node value consistency check ✓ Fails fast — before any FS operations

The key change: Delta 12's validation explicitly checks that s_appltmp is a node-local path, not a shared location. This check runs at the start of adop phase=prepare, fs_clone, cutover, and abort.

Systems that ran with a shared s_appltmp under Delta 6 — where this check did not exist — hit this error the first time they run any ADOP phase after upgrading to Delta 12. The upgrade itself does not reconfigure your context files.

Why "Fail Fast" Is Good Engineering The validation deliberately aborts before touching the file system. A shared s_appltmp discovered mid-clone — when multiple ADOP processes on different nodes simultaneously write to the same temp directory — produces far harder-to-diagnose failures: corrupted temp files, half-cloned directories, unrecoverable ADOP state. Early rejection is the right design.

5. The Fix — Step by Step

The remediation has three parts: fix the context file, run AutoConfig to propagate the change to all dependent configs, then run fs_clone to synchronize the file systems. Repeat on every application node.

1

Identify and back up the context file

Shell
# Locate all context files on this node find $INST_TOP/appl/admin -name "*.xml" -type f # e.g. /u01/oracle/fs1/inst/apps/PROD_appnode1/appl/admin/PROD_appnode1.xml # Back up before editing cp $INST_TOP/appl/admin/$CONTEXT_NAME.xml \ $INST_TOP/appl/admin/$CONTEXT_NAME.xml.bak_$(date +%Y%m%d_%H%M%S)
2

Check the current s_appltmp value

Shell
grep "s_appltmp" $INST_TOP/appl/admin/$CONTEXT_NAME.xml # If it shows a shared or NFS path, it needs fixing: <parameter name="s_appltmp" itsm="false">/shared/appltmp</parameter> # Correct value — node-local: <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter>
3

Edit the context file

Find the s_appltmp parameter and replace the shared path with the node-local value:

Shell
vi $INST_TOP/appl/admin/$CONTEXT_NAME.xml # Replace the value with the node-local path: <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter> # Fully expanded path is also valid (and preferred for clarity): <parameter name="s_appltmp" itsm="false">/u01/oracle/fs1/inst/apps/PROD_appnode1/appltmp</parameter>
4

Create the appltmp directory if it doesn't exist

Shell
mkdir -p $INST_TOP/appltmp chmod 755 $INST_TOP/appltmp chown applmgr:dba $INST_TOP/appltmp
5

Run AutoConfig on the run edition

AutoConfig reads the updated context file and regenerates all dependent configuration files — shell environment scripts, service configs, port assignments, and so on. Without this step, the context file change does not propagate to the actual runtime environment.

Shell — Run as applmgr
cd $APPL_TOP/ad/bin ./adautocfg.sh # Confirm context file path when prompted. # AutoConfig regenerates all configuration files from the updated context. # Verify successful completion: grep -i "AutoConfig completed" $INST_TOP/appl/admin/log/adautocfg*.log | tail -1
6

Run fs_clone

With the context file corrected and AutoConfig run, fs_clone can now validate cleanly and propagate the correct configuration to the patch edition.

Shell — Run as applmgr
cd $APPL_TOP/ad/bin ./adop phase=fs_clone # Monitor progress — can take 30–90 minutes depending on APPL_TOP size. # On completion: fs_clone phase completed successfully
Multi-Node Environments Repeat Steps 1–5 on every application node before running fs_clone. ADOP validates the context file on all registered nodes — a single node still pointing to a shared s_appltmp will cause the same error.

6. Verification

After fs_clone completes, confirm the patch edition inherited the corrected value:

Shell
# Check s_appltmp in the patch edition context file grep "s_appltmp" $PATCH_INST_TOP/appl/admin/$CONTEXT_NAME.xml # Must return the node-local path, not a shared one # Confirm the appltmp directory exists and is accessible ls -ld $INST_TOP/appltmp # Definitive test — run prepare and confirm no validation errors adop phase=prepare # A clean prepare is proof the fix is complete.

7. Pre-fs_clone Checklist

Regardless of why you're running fs_clone, work through these checks first to avoid mid-run failures:

CheckCommandRequirement
Patch FS disk space df -h $PATCH_FS_MOUNT ≥ 25 GB free
/tmp free space df -h /tmp ≥ 15 GB free
No active ADOP ps -ef | grep adop No running adop processes
Database reachable tnsping $ORACLE_SID OK response from listener
s_appltmp value correct grep s_appltmp $INST_TOP/appl/admin/$CONTEXT_NAME.xml Must be $INST_TOP/appltmp
AutoConfig recently run ls -lt $INST_TOP/appl/admin/log/adautocfg*.log | head -1 After last context file edit
No form compiler active ps -ef | grep f90 | grep -v grep Empty output
Log directory space df -h $INST_TOP/appl/admin/log/ ≥ 5 GB free

Root Cause — At a Glance

Summary AD-TXK Delta 12 introduced strict pre-execution validation in txkADOPValidations.pm that rejects any s_appltmp value pointing to a shared or non-node-local path. Systems running with a shared s_appltmp under Delta 6 — where this check did not exist — encounter the error on the first ADOP phase after the upgrade. The fix is deterministic: correct the context file → run AutoConfig → run fs_clone. Repeat on every node.

The error message is deliberately explicit. The fix is a three-command sequence per node. And the validation failing before any file system operations begin is intentional — it prevents far messier mid-clone failures that would require manual state recovery.

If you fixed the context file on only some nodes and the error persists, run fs_clone with verbose logging to identify which node is still misconfigured. ADOP logs per-node validation results early in the output.

References
  • MOS Note 2424019.1 — fs_clone Fails With s_appltmp Error
  • MOS Note 2220669.1 — ADOP Cutover / Abort / fs_clone Fails s_appltmp Validation
  • Oracle E-Business Suite Maintenance Guide (Doc ID 1982566.1)
  • AD and TXK Delta 12 Release Notes (Oracle EBS Blog)

No comments:

Post a Comment