We’ve had issues in the past patching the Oracle Homes on our Exadata nodes due to the fact that they were defined as part of a RAC cluster.
We don’t use RAC, so we have made sure we don’t encounter such problems in the past by unlinking them as described in the documentation.
However I have maintained a separate pair of linked Oracle homes for testing, and I have a reproducible test case of OPatch getting into a mess under certain conditions.
Running this testcase, I have noticed some change in behavior in this area with latest version of OPatch 12.2.0.1.9 compared to previous version 12.2.0.1.8.
Review the following testcase using OPatch 12.2.0.1.8. A few things to note.
First by default patches are applied to both nodes (this can be overridden by specifying -local flag).
Second the confirmation prompt regarding the remote node takes place after the local node has been patched.
[oracle@hkexdb01 22652097]$ $ORACLE_HOME/OPatch.backup/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.8
Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0.2/dbhome_2
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0.2/dbhome_2/oraInst.loc
OPatch version : 12.2.0.1.8
OUI version : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/dbhome_2/cfgtoollogs/opatch/opatch2017-05-25_15-36-03PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 22652097
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'hkexdb02'
Local node: 'hkexdb01'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0.2/dbhome_2')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '22652097' to OH '/u01/app/oracle/product/12.1.0.2/dbhome_2'
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0.2/dbhome_2/rdbms/admin/catfusrg.sql" because it is the same as
the file in incoming patch "/home/oracle/patchtest/22652097/files/rdbms/admin/catfusrg.sql"
Patching in rolling mode.
The node 'hkexdb02' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'hkexdb02'.
(Oracle Home = '/u01/app/oracle/product/12.1.0.2/dbhome_2')
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'hkexdb02'
Apply-related files are:
FP = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/copy_files.txt"
DP = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/make_cmds.txt"
RC = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-36-03PM/rac/make_cmds.txt" with actual path.
Running command on remote node 'hkexdb02':
cd /u01/app/oracle/product/12.1.0.2/dbhome_2/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2 || echo REMOTE_MAKE_FAILED::>&2
The node 'hkexdb02' has been patched. You can restart Oracle instances on it.
Finished patching in rolling mode.
Patch 22652097 successfully applied.
Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_2/cfgtoollogs/opatch/opatch2017-05-25_15-36-03PM_1.log
OPatch succeeded.
[oracle@hkexdb01 22652097]$
Compare that with an attempt to apply same patch using OPatch 12.2.0.1.9 using the same command; by default, only the local node will be patched.
[oracle@hkexdb01 22652097]$ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.9 Copyright (c) 2017, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0.2/dbhome_2 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0.2/dbhome_2/oraInst.loc OPatch version : 12.2.0.1.9 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0.2/dbhome_2/cfgtoollogs/opatch/opatch2017-05-25_15-39-06PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 22652097 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.1.0.2/dbhome_2') Is the local system ready for patching? [y|n] n User Responded with: N NApply exits on request UtilSession: NApply exits on user's request. UtilSession system modification phase did not start: NApply exits on request Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_2/cfgtoollogs/opatch/opatch2017-05-25_15-39-06PM_1.log OPatch stopped on request. [oracle@hkexdb01 22652097]$
To get 12.2.0.1.9 to patch remote nodes as well, -all_nodes switch must be provided, as demonstrated below. Observe also that confirmation regards patching all nodes takes place before patching any of them.
[oracle@hkexdb01 22652097]$ $ORACLE_HOME/OPatch/opatch apply -all_nodes
Oracle Interim Patch Installer version 12.2.0.1.9
Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0.2/dbhome_2
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0.2/dbhome_2/oraInst.loc
OPatch version : 12.2.0.1.9
OUI version : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/dbhome_2/cfgtoollogs/opatch/opatch2017-05-25_15-40-05PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 22652097
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'hkexdb02'
Local node: 'hkexdb01'
Please shut down Oracle instances running out of this ORACLE_HOME on all the nodes.
(Oracle Home = '/u01/app/oracle/product/12.1.0.2/dbhome_2')
Are all the nodes ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '22652097' to OH '/u01/app/oracle/product/12.1.0.2/dbhome_2'
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patching in all-node mode.
Updating nodes 'hkexdb02'
Apply-related files are:
FP = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/copy_files.txt"
DP = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/make_cmds.txt"
RC = "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/12.1.0.2/dbhome_2/.patch_storage/NApply/2017-05-25_15-40-05PM/rac/make_cmds.txt" with actual path.
Running command on remote node 'hkexdb02':
cd /u01/app/oracle/product/12.1.0.2/dbhome_2/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2 || echo REMOTE_MAKE_FAILED::>&2
All nodes have been patched. You may start Oracle instances on the local system and nodes 'hkexdb02'
Patch 22652097 successfully applied.
Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_2/cfgtoollogs/opatch/opatch2017-05-25_15-40-05PM_1.log
OPatch succeeded.
[oracle@hkexdb01 22652097]$
I think the change to prompt for confirmation before patching any nodes is an improvement. Regarding the change of defaults for patching the local node vs patching all nodes, this doesn’t affect us (as we’ve unlinked our homes) but the change might cause some issues for some.
Update 2017-05-26, this change in behaviour is documented in the following MOS note:
OPatch: Behavior Changes starting in OPatch 12.2.0.1.9 and 11.2.0.3.16 releases (Doc ID 2232156.1)