19C : Pluggable database in restricted mode due to datapatch failure

By Gowthami | apps-dba.com | Oracle Administration Series

A common issue after Oracle Database 19c patching is a Pluggable Database (PDB) opening in RESTRICTED mode due to a datapatch failure. This occurs when datapatch — the tool that applies SQL-based patch changes — fails or is not run after the binary patch is applied. This post explains root causes and step-by-step resolution.

Key Insight: After applying Oracle patches (OPatch), you MUST run datapatch to apply the SQL-based portions of the patch to each database. If datapatch fails mid-way, PDBs may open in RESTRICTED mode until the SQL patches are successfully applied.

Symptoms

-- PDB shows RESTRICTED in open mode
SELECT con_id, name, open_mode, restricted
FROM v$pdbs;

-- Output:
-- CON_ID  NAME    OPEN_MODE   RESTRICTED
-- 3       PROD    READ WRITE  YES        <-- Problem!

-- Alert log shows:
-- "PDB PROD is restricted because datapatch has not been run"
-- or errors in /oracle/diag/rdbms/db/trace/ datapatch logs

Root Causes

• OPatch was applied but datapatch was not run afterward
• datapatch ran but failed with errors (DB not open, network issue, ORA- error)
• PDB was closed when datapatch ran and didn't get the SQL changes
• Registry mismatch between CDB and PDB patch levels

Diagnosing the Issue

-- Check registry status in the affected PDB
ALTER SESSION SET CONTAINER = PROD;

SELECT comp_id, comp_name, status, version, modified
FROM dba_registry
ORDER BY comp_id;
-- Look for: STATUS = 'INVALID' or version mismatch

-- Check datapatch history
SELECT patch_id, patch_uid, action, status, action_time, description
FROM dba_registry_sqlpatch
ORDER BY action_time DESC;
-- Look for BOOTSTRAP or WITH ERRORS status

-- Check CDB vs PDB versions
SELECT con_id, version, status
FROM cdb_registry
WHERE comp_id = 'CATPROC'
ORDER BY con_id;

Resolution: Re-run datapatch

-- Step 1: Ensure the PDB is open (READ WRITE)
-- Connect as SYSDBA to CDB
ALTER PLUGGABLE DATABASE PROD OPEN;

-- Step 2: Run datapatch from OS (as oracle user)
cd $ORACLE_HOME/OPatch
./datapatch -verbose

-- For a specific PDB only:
./datapatch -pdbs PROD -verbose

-- Step 3: Monitor datapatch output
-- Look for: "Patch application complete" for each PDB
-- datapatch log: $ORACLE_BASE/cfgtoollogs/sqlpatch/

After datapatch Completes

-- Verify PDB is no longer restricted
SELECT con_id, name, open_mode, restricted FROM v$pdbs;
-- RESTRICTED should now show NO

-- Verify registry is valid
ALTER SESSION SET CONTAINER = PROD;
SELECT comp_id, status, version FROM dba_registry;
-- All components should show 'VALID'

-- If still restricted, try restarting the PDB
ALTER PLUGGABLE DATABASE PROD CLOSE;
ALTER PLUGGABLE DATABASE PROD OPEN;

Prevention Best Practices

• Always run datapatch immediately after OPatch apply, before starting services
• Ensure ALL PDBs are open (READ WRITE) before running datapatch
• Run datapatch in a maintenance window, not during production hours
• Review datapatch logs even when it reports success
• Test the patching procedure in a non-production environment first

Summary

PDBs opening in RESTRICTED mode after patching is a common Oracle 12c/19c issue. The fix is straightforward: ensure the PDB is open, re-run datapatch targeting the affected PDB, and verify the registry shows all components as VALID. Build datapatch into your standard patching runbook to prevent this issue in future patch cycles.

Oracle Exadata - The Complete Guide

Master Oracle administration, patching, and Exadata-specific management with Gowthami's complete guide.

Get the Book

AWR Comparison Report

The awrddrpt.sql SQL script generates a report that compares performance attributes and configuration settings between two selected time periods on the local database instance. 

The report compares top wait events, SQL performance, parameter changes etc between two snapshot intervals.

Sample Test Case : Change SGA of PDB from 6 GB to 3 GB and verify it in AWR comparison report

=> For the purpose of testing, I am changing the snapshot interval to 15 minutes

SQL> col SNAP_INTERVAL for a20
SQL> col retention for a20
SQL> select snap_interval, retention from dba_hist_wr_control;

SNAP_INTERVAL        RETENTION
-------------------- --------------------
+00000 01:00:00.0    +00008 00:00:00.0

SQL> execute dbms_workload_repository.modify_snapshot_settings(interval => 15,retention => 11520);

PL/SQL procedure successfully completed.

SQL> select snap_interval, retention from dba_hist_wr_control;

SNAP_INTERVAL        RETENTION
-------------------- --------------------
+00000 00:15:00.0    +00008 00:00:00.0
SQL>

=> Change SGA of PDB

SQL> alter session set container=SITPDB;

SQL> show parameter sga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
allow_group_access_to_sga            boolean     FALSE
lock_sga                             boolean     FALSE
pre_page_sga                         boolean     TRUE
sga_max_size                         big integer 11G
sga_min_size                         big integer 0
sga_target                           big integer 6G
unified_audit_sga_queue_size         integer     1048576

SQL> alter system set sga_target=3g;
System altered.

SQL>

=> Wait till snapshot generation and run awr comparison report 

SQL> alter session set container=SITPDB;

SQL> @$ORACLE_HOME/rdbms/admin/awrddrpt.sql

Review the init.ora parameters section in the report, to verify the sga change 

=> Useful scripts:

awrddrpi.sql => for specific instance

awrgdrpt.sql => for all instances in RAC database

awrgdrpi.sql => for a set of instances in RAC database 

fs_clone issue

EBS R12.2 ADOP fs_clone Troubleshooting Cloud Manager ⏱ ~12 min read

You've just upgraded AD-TXK from Delta 6 to Delta 12, and the very first adop phase=fs_clone you run hits a wall:

Error

[ERROR]: The value of s_appltmp is not set correctly in atleast one of the context files.

This post explains exactly what triggered it, why Delta 12 introduced this check, and how to fix it permanently.

In This Article

1. The Dual File System Architecture — Quick Recap
2. What fs_clone Actually Does
3. s_appltmp: Purpose, Location, and Why Shared Paths Break ADOP
4. What Delta 12 Changed That Surfaced This
5. The Fix — Step by Step
6. Verification
7. Pre-fs_clone Checklist

---

1. The Dual File System Architecture — Quick Recap

Oracle EBS R12.2 requires two independent application-tier file systems to enable zero-downtime patching — the run edition file system (fs1) and the patch edition file system (fs2). A third component, fs_ne (non-editioned), holds data shared across both editions and is never copied during patching.

Zero-Downtime Patching PREPARE, APPLY, FINALIZE, and CLEANUP all run with users fully online. Only CUTOVER — the edition switch — requires a brief service restart (typically 10–30 minutes). This is what makes R12.2 online patching fundamentally different from earlier EBS versions.

---

2. What fs_clone Actually Does

adop phase=fs_clone recreates the patch edition file system as a complete, synchronized copy of the run edition. It is not part of the standard patching cycle — it is an out-of-cycle resynchronization operation.

When to run it:

• After configuration changes (autoconfig on run edition) and cutover — to propagate those changes to the patch edition before the next PREPARE
• After adop phase=abort — to restore the patch file system to a clean, synchronized state
• When run and patch file systems have diverged significantly — faster than applying all accumulated delta patches
• Before cloning the entire EBS environment — ensures both file systems are perfectly aligned

What fs_clone does internally, in sequence:

1. Validates the run edition context file — including s_appltmp. This is where our error fires.
2. Backs up the current patch APPL_TOP
3. Copies the run edition APPL_TOP to the patch edition location, preserving permissions, ownership, and timestamps
4. Generates the patch edition context file from the run edition context file
5. Updates patch-edition-specific paths in the cloned context file (fs1 → fs2 path substitution)

Resource Requirements fs_clone backs up the entire patch APPL_TOP before overwriting it. Plan for at least 25 GB of free space on the patch file system mount and 15 GB in /tmp. On large APPL_TOPs, this can take 30–90 minutes.

---

3. s_appltmp: Purpose, Location, and Why Shared Paths Break ADOP

s_appltmp is an XML parameter in the node's context file that defines the application temporary directory — where EBS writes temporary files during concurrent processing, form operations, and ADOP phases.

Context file location:

Path

$INST_TOP/appl/admin/$CONTEXT_NAME.xml # Example on a typical EBS R12.2 install: /u01/oracle/fs1/inst/apps/PROD_appnode1/appl/admin/PROD_appnode1.xml

What the parameter looks like in the context file:

XML — Context File

<!-- Correct — node-local path --> <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter> <!-- Incorrect — shared NFS/cluster path --> <parameter name="s_appltmp" itsm="false">/shared/nfs/appltmp</parameter>

Why a shared path breaks things:

In a multi-node EBS application tier, each node runs independent concurrent manager processes. When s_appltmp points to a shared NFS location, every node writes its temporary files to the same directory simultaneously. This causes:

• File collisions — two nodes create a temp file with the same name, one overwrites the other
• PID conflicts — lock files with identical names written by different nodes' processes
• ADOP concurrency failures — multiple ADOP processes across nodes collide on shared temporary state

s_appltmp vs APPLPTMP — Don't Confuse These s_appltmp is for application-layer temporary files and must be node-local (one per node). APPLPTMP is for PL/SQL concurrent program output files and must be shared and accessible from the database server — the exact opposite requirement. Mixing these up is a common source of confusion.

The correct value is always node-local, under $INST_TOP — which is per-node by definition:

Shell

$INST_TOP/appltmp # Expands to something like: /u01/oracle/fs1/inst/apps/PROD_appnode1/appltmp # Each node's $INST_TOP is unique — so this is inherently node-local

---

4. What Delta 12 Changed That Surfaced This

AD-TXK Delta releases are cumulative updates to the Application DBA (AD) and Applications Technology (TXK) modules. Delta 12 introduced a significantly enhanced ADOP validation framework in txkADOPValidations.pm and related modules.

The key change: Delta 12's validation explicitly checks that s_appltmp is a node-local path, not a shared location. This check runs at the start of adop phase=prepare, fs_clone, cutover, and abort.

Systems that ran with a shared s_appltmp under Delta 6 — where this check did not exist — hit this error the first time they run any ADOP phase after upgrading to Delta 12. The upgrade itself does not reconfigure your context files.

Why "Fail Fast" Is Good Engineering The validation deliberately aborts before touching the file system. A shared s_appltmp discovered mid-clone — when multiple ADOP processes on different nodes simultaneously write to the same temp directory — produces far harder-to-diagnose failures: corrupted temp files, half-cloned directories, unrecoverable ADOP state. Early rejection is the right design.

---

5. The Fix — Step by Step

The remediation has three parts: fix the context file, run AutoConfig to propagate the change to all dependent configs, then run fs_clone to synchronize the file systems. Repeat on every application node.

1

Identify and back up the context file

Shell

# Locate all context files on this node find $INST_TOP/appl/admin -name "*.xml" -type f # e.g. /u01/oracle/fs1/inst/apps/PROD_appnode1/appl/admin/PROD_appnode1.xml # Back up before editing cp $INST_TOP/appl/admin/$CONTEXT_NAME.xml \ $INST_TOP/appl/admin/$CONTEXT_NAME.xml.bak_$(date +%Y%m%d_%H%M%S)

2

Check the current s_appltmp value

Shell

grep "s_appltmp" $INST_TOP/appl/admin/$CONTEXT_NAME.xml # If it shows a shared or NFS path, it needs fixing: <parameter name="s_appltmp" itsm="false">/shared/appltmp</parameter> # Correct value — node-local: <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter>

3

Edit the context file

Find the s_appltmp parameter and replace the shared path with the node-local value:

Shell

vi $INST_TOP/appl/admin/$CONTEXT_NAME.xml # Replace the value with the node-local path: <parameter name="s_appltmp" itsm="false">$INST_TOP/appltmp</parameter> # Fully expanded path is also valid (and preferred for clarity): <parameter name="s_appltmp" itsm="false">/u01/oracle/fs1/inst/apps/PROD_appnode1/appltmp</parameter>

4

Create the appltmp directory if it doesn't exist

Shell

mkdir -p $INST_TOP/appltmp chmod 755 $INST_TOP/appltmp chown applmgr:dba $INST_TOP/appltmp

5

Run AutoConfig on the run edition

AutoConfig reads the updated context file and regenerates all dependent configuration files — shell environment scripts, service configs, port assignments, and so on. Without this step, the context file change does not propagate to the actual runtime environment.

Shell — Run as applmgr

cd $APPL_TOP/ad/bin ./adautocfg.sh # Confirm context file path when prompted. # AutoConfig regenerates all configuration files from the updated context. # Verify successful completion: grep -i "AutoConfig completed" $INST_TOP/appl/admin/log/adautocfg*.log | tail -1

6

Run fs_clone

With the context file corrected and AutoConfig run, fs_clone can now validate cleanly and propagate the correct configuration to the patch edition.

Shell — Run as applmgr

cd $APPL_TOP/ad/bin ./adop phase=fs_clone # Monitor progress — can take 30–90 minutes depending on APPL_TOP size. # On completion: fs_clone phase completed successfully

Multi-Node Environments Repeat Steps 1–5 on every application node before running fs_clone. ADOP validates the context file on all registered nodes — a single node still pointing to a shared s_appltmp will cause the same error.

---

6. Verification

After fs_clone completes, confirm the patch edition inherited the corrected value:

Shell

# Check s_appltmp in the patch edition context file grep "s_appltmp" $PATCH_INST_TOP/appl/admin/$CONTEXT_NAME.xml # Must return the node-local path, not a shared one # Confirm the appltmp directory exists and is accessible ls -ld $INST_TOP/appltmp # Definitive test — run prepare and confirm no validation errors adop phase=prepare # A clean prepare is proof the fix is complete.

---

7. Pre-fs_clone Checklist

Regardless of why you're running fs_clone, work through these checks first to avoid mid-run failures:

Check	Command	Requirement
Patch FS disk space	df -h $PATCH_FS_MOUNT	≥ 25 GB free
/tmp free space	df -h /tmp	≥ 15 GB free
No active ADOP	ps -ef | grep adop	No running adop processes
Database reachable	tnsping $ORACLE_SID	OK response from listener
s_appltmp value correct	grep s_appltmp $INST_TOP/appl/admin/$CONTEXT_NAME.xml	Must be $INST_TOP/appltmp
AutoConfig recently run	ls -lt $INST_TOP/appl/admin/log/adautocfg*.log | head -1	After last context file edit
No form compiler active	ps -ef | grep f90 | grep -v grep	Empty output
Log directory space	df -h $INST_TOP/appl/admin/log/	≥ 5 GB free

---

Root Cause — At a Glance

Summary AD-TXK Delta 12 introduced strict pre-execution validation in txkADOPValidations.pm that rejects any s_appltmp value pointing to a shared or non-node-local path. Systems running with a shared s_appltmp under Delta 6 — where this check did not exist — encounter the error on the first ADOP phase after the upgrade. The fix is deterministic: correct the context file → run AutoConfig → run fs_clone. Repeat on every node.

The error message is deliberately explicit. The fix is a three-command sequence per node. And the validation failing before any file system operations begin is intentional — it prevents far messier mid-clone failures that would require manual state recovery.

If you fixed the context file on only some nodes and the error persists, run fs_clone with verbose logging to identify which node is still misconfigured. ADOP logs per-node validation results early in the output.

References

• MOS Note 2424019.1 — fs_clone Fails With s_appltmp Error
• MOS Note 2220669.1 — ADOP Cutover / Abort / fs_clone Fails s_appltmp Validation
• Oracle E-Business Suite Maintenance Guide (Doc ID 1982566.1)
• AD and TXK Delta 12 Release Notes (Oracle EBS Blog)