Ad Online Patching (ADOP) Known Issues & Fixes

Start Here

Get in touch with a
TriCore Solutions specialist

Blog | Aug 14, 2017

Ad Online Patching (ADOP) Known Issues & Fixes

Learn about few ADOP known issues and their fixes from the Part 2 of the blog series. Click here to read part 1. 

 Learn about a few ADOP known issues and their fixes from the Part Two of the blog series. Click here to read Part One.  

In the Part Two of the ADOP series, I have covered some of the common issues, their fixes and ADOP logs location details image. The details are given below.  

The following image describe ADOP log files details, green color boxes indicates location of log files.

adop_new.png

Image Source: http://onlineappsdba.com/index.php/2015/10/25/r12-2-log-files-location-apps-every-oracle-apps-dba-r12-must-know/

Issue-1

ADOP prepare fails with below data dictionary corruption error, this is a generic common corruption error.

Error:

Note: Error may vary for different environment.

[EVENT]     Verifying data dictionary.

    [UNEXPECTED]Data dictionary corrupted:

    [UNEXPECTED]Data dictionary corruption - missing parent

    5608975 ORA$BASE        IMAT            V_WORKFLOWWORKITEMII           VIEW

    5608973 ORA$BASE        IMAT            V_WFSTAGETIME                  VIEW

    5608973 ORA$BASE        IMAT            V_WFSTAGETIME                  VIEW

    [UNEXPECTED]Data dictionary corruption detected. Provide details to

    [UNEXPECTED]Oracle Support and ask for a bug to be opened against the

    [UNEXPECTED]Online patching component of Oracle Application Install.

/apps1/SID/fs_ne/EBSapps/log/adop/18/adop_xxxxx_xxxxx.log: 

Cause

Issues related to the data dictionary will happen when a developer promotes customizations improperly, which violates the online patching standards.

Fix

Fix Logical Data Dictionary corruption (missing-parent)

  1. Run the $AD_TOP/sql/ADZDDBCC.sql script as apps user to identify whether the logical data dictionary corruption exists. Check corruption in spool log.
  2. Run corruption fix script $AD_TOP/patch/115/sql/adzddmpfix.sql as sys user.

SQL> @adzddmpfix.sql

 "---- Fixing Data Dictionary Corruptions (missing parent) ----"

12 rows deleted.

Commit complete.

System altered.

 "---- Compiling invalids ----"

PL/SQL procedure successfully completed.

Commit complete. 

Note: In above output 12 corruption objects are fixed.

  1. Run the $AD_TOP/sql/ADZDDBCC.sql script again as apps user to identify whether the logical data dictionary corruption is still present.
  2. If no corruption is found, proceed with the upgrade or adop patching-cycle.
  3. If corruption is still present, contact Oracle Support and log a bug. 

Retry adop prepare once fix is applied.

Issue-2:

In general ADOP prepare fails with below error due to some Oracle bug.

Error:

Lines #(47-50):

runMSSrvPortsVal : oacore_server1:7252

ERROR: Run fs Context variable s_oacore_server_ports value cannot be NULL for oacore_server2

ERROR: Derived Patch managed server oacore_server2 port : NULL

ERROR: Failed to clone Run Context file to refresh Patch context file

Fix

Change the port reported in error and paths accordingly target server & SID in below command and run to fix this issue.

perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \

ebs-delete-managedserver \

-contextfile=/apps1/SID/fs1/inst/apps/SID_server/appl/admin/SID_server.xml -managedsrvname=oacore_server2 \

-servicetype=oacore -logfile=$APPLRGF/TXK/delMS_oacore_server2.log 

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=/apps1/SID/fs1/inst/apps/SID_server/appl/admin/SID_server.xml \

-configoption=removeMS -oacore=server.cm.charter.com:7252 

Retry ADOP prepare once fix is applied.

Issue-3

While applying patches with adop phase=apply patches=123456, sometime Forms objects did not generate successfully and ADOP session quit without patch continue prompt Y/N as not like previous EBS R12.1, 11i.

Error

For example, the following Oracle Forms objects did not generate successfully:

inv     forms/US        INVMWBIV.fmx

Fix

Restart the failed patch session from the patch directory with below command.

cd /apps1/SID/fs_ne/EBSapps/patch

adop phase=apply patches=20609071 restart=yes flags=autoskip 

Issue-4

If you encounter ADOP cutover hung or server crashes/rebooted issues in the middle of ADOP cutover phase, below action plan steps need to be performed to fix the issue and continue ADOP further.

Fix:

1) Make sure no services or processes are running from the PATCH file system

2) Ensure the Weblogic Admin Server and Node Manager are running on the Run File system. You can run the following commands to check the status:

 $ adadminsrvctl.sh status

 $ adnodemgrctl.sh status

3) Run these in order:

 $ adop phase=abort

 $ adop phase=cleanup cleanup_mode=full

 $ adop phase=fs_clone force=yes 

4) Run an empty ADOP cycle to make sure there is no issue in the ADOP cutover.

$adop phase=prepare, finalize, cutover, cleanup cleanup_mode=full

5) Next continue with fresh ADOP prepare and apply patches, any deployment again which were lost after abort and cleanup and complete rest of ADOP phases like finalize, cutover and cleanup.

Issue-5

Aborting an Online Patching Cycle for any reasons.

If a patching cycle is failing and the issue cannot be resolved quickly, it is possible to abort the patching cycle and return to normal runtime operation. The patch edition will be dropped.

You can abandon a patching cycle (without applying any patches) by running the command:

$ adop phase=abort

Important: This abort command can only be used before successful completion of the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle.

Aborting a patching cycle will drop the patch edition, but you must then run the cleanup and fs_clone phases before starting a new patching cycle. The cleanup must be a full cleanup.

For example:

$ adop phase=prepare

$ adop phase=apply patches=123456

[Patch application encounters problems and you want to abort]

$ adop phase=abort

$ adop phase=cleanup cleanup_mode=full

$ adop phase=fs_clone

Optionally, you can combine the abort and cleanup commands as follows:

 $ adop phase=abort,cleanup cleanup_mode=full 

Note: You cannot abort application of a patch applied in hotpatch mode (adop phase=apply apply_mode=hot patch). 

Conclusion:
As discussed above, known ADOP issues and fixes are helpful for AppsDBAs when they encounter similar issues. I hope that you found the ADOP series helpful. If you have any questions on the topic feel free to send in your queries by clicking on the button below. We encourage you to leave any comments that you may have for the blog series, you can do so by filling out the fields given below:

Ask Nagunaik