Patching Steps

How To Check If a Certain Patch Was Applied to Oracle E-Business Suite Instance (11i or R12) ? (Doc ID 443761.1)

***********************************************************************************

In releases prior to 12.2, we used to query ad_bugs or ad_applied_patches to find out if a patch is applied or not.

In 12.2, this is no longer the right way to do it due to the implications of online patching.

For example, if you apply a patch in an online patching cycle and aborted the cycle before it is complete, the records inserted in the above two tables will not be updated or removed.

The following query helps in identifying the patches applied:

SELECT adop_session_id, bug_number, session_type, 

       DECODE(status,'N','Applied on other nodes',

                     'R','Running',

                     'H','Failed (Hard)',

                     'F','Failed (Jobs Skipped)',

                     'S','Success (Jobs Skipped)',

                     'Y','Success',

                     'C','Clone Complete') status,

       applied_file_system_base, patch_file_system_base,

       node_name, start_date, end_date, 

       ROUND((end_date - start_date) * 24*60,2) exec_time,

       adpatch_options, autoconfig_status, driver_file_name

FROM ad_adop_session_patches

WHERE session_type IN ('ADPATCH','HOTPATCH','DOWNTIME','ONLINE')

ORDER BY adop_session_id, start_date, end_date;

Session type column was restricted to the following values:

ADPATCH: Patches applied using the traditional adpatch tool. It is used right after installing 12.2 and before online patching is enabled by applying patch 13543062.

HOTPATCH: Patches applied in hotpatch mode (aka. no online patching cycle is involved). It should be used only when instructed by Oracle.

DOWNTIME: Patches applied when all application services are down. Only used when instructed by Oracle, such as when applying 12.2.6 RUP.

ONLINE: Patches applied in a normal online patching cycle. This is the recommended method.

 

If you want to find out which patch fixed a specific bug, you can use the following query:

SELECT b.bug_number, asp.adop_session_id, asp.bug_number patch#,

       asp.session_type, asp.applied_file_system_base,

       asp.node_name, asp.start_date, asp.end_date

FROM ad_bugs b, ad_patch_run_bugs prb, ad_patch_runs pr,

     ad_patch_drivers pd, ad_adop_session_patches asp

WHERE b.bug_id = prb.bug_id

  AND prb.patch_run_id = pr.patch_run_id

  AND pr.patch_driver_id = pd.patch_driver_id

  AND pr.patch_run_id = asp.patchrun_id

  AND prb.applied_flag = 'Y'

  AND b.bug_number = '&bug_num';

*************************************** 1 *****************************************

STEP 1:

Patch Impact Analysis:

Login as the owner of application tier eg: " applmgr "

$ . ./EBSapps.env run

$ echo $PATCH_TOP

$ cd  $PATCH_TOP/

$ mkdir ad

Put the patch whose impact analysis is to be done without unzipping it.

Run the patch impact analysis.

NOTE: Request will run and complete with warning. 

STEP 2:

Check invalids and take backup of the same.

Run utlrp.sql.

STEP 3:

Patch Staging:

Login as the owner of application tier eg: " applmgr "

$ . ./EBSapps.env run

$ echo $PATCH_TOP

$ cd $PATCH_TOP/

Unzip the patch that needs to be applied.

NOTE: If the application is multinode need to unzip the patch on all the nodes in the PATCH_TOP directory.

STEP 4:

Patch Application:

Login as the owner of application tier eg: " applmgr "

$ . ./EBSapps.env run

$ echo $RUN_BASE

output would be run file system either fs1 or fs2

adop -status -details

adop phase=prepare

adop -status -details

adop phase=apply patches=patch_number

adop -status -details

adop phase=finalize

adop -status -details

adop phase=cutover

echo $RUN_BASE

echo $FILE_EDITION

adop -status -details

adop phase=cleanup cleanup_mode=full

adop -status -details

adop phase=fs_clone

IMP_POINTS:

tail the log file location after the apply phase has started.

It internally run ETCC and recommends for data fixes to be applied on database ORACLE_HOME as adop starts.

 *************************************** 2 *****************************************

###########  FILE_SYSTEM  ###############

s_base: /a01/R122_EBS/fs1

s_patch_base: /a01/R122_EBS/fs2

s_ne_base: /a01/R122_EBS/fs_ne

s_base:

-->EBSapps->appl

          ->comn

          ->10.1.2

-->11.2.0

-->inst

-->FMW_HOME

s_patch_base:

-->EBSapps->appl

          ->comn

          ->10.1.2

-->inst

-->FMW_HOME

s_ne_base:

-->EBSapps->appl

          ->patch (patch_top where all patches are downloaded)

          ->log->adop->adop_session_id->phase_T.s->context_name         

There are below 5 Phases of ADOP(online patching):

1) PREPARE

2) APPLY

3) FINALIZE

4) CUTOVER

5) CLEANUP

$ echo $FILE_EDITION

Output: either RUN or PATCH

$ echo $TWO_TASK

Output: dbSID (PROD)

$ echo $RUN_BASE

Output: fs1 or fs2 file system path

if your RUN file system is fs1 the it will show fs1 path or if your RUN file system is fs2 it will show fs12 path

$ echo $PATCH_BASE

Output: fs1 or fs2 file system path

if your RUN file system is fs1 the it will show fs1 path or if your RUN file system is fs2 it will show fs12 path

$ echo $NE_BASE

Check if the particular patch has been applied before:

SQL> select * from ad_bugs where bug_number='&num';

SQL> select * from ad_applied_patches where patch_name='&num';

Check if there are any long running requests,if any let them complete.

Keep APPS password: SYSTEM password: WLSADMIN password: handy.

$ adop -help                 (Shows the help screen)

$ adop -examples             (Displays some commonly used adop sample commnads)

MAIN STEPS:

STEP 1: 

Login as the owner of application tier eg: " applmgr "

source the run file system env file:

$ cd /u01/app

$ . ./EBSapps.env run

        OR      

. /u01/oracle/dba/EBSapps.env RUN

It will set environment and display below information on screen.

E-Business Suite Environment Information

 —————————————-

 RUN File System : /u01/oracle/dba/fs2/EBSapps/appl

 PATCH File System : /u01/oracle/dba/fs1/EBSapps/appl

 Non-Editioned File System : /u01/oracle/dba/fs_ne

 DB Host: dbhost.hertz.com Service/SID: dba

 Sourcing the RUN File System …

$ echo $FILE_EDITION

$ echo $TWO_TASK

$ echo $RUN_BASE

$ echo $PATCH_TOP

$ echo $NE_BASE

$ echo $PATCH_BASE

NOTE:

1.

Run the Run environment file(optional)

The adop utility sets its own environment. There is therefore no need to source the environment before running it.

2.

We always run adop utility from the run edition file system. The adop process automatically sets its environment correctly, regardless of the edition it is run from.

STEP 2: 

Download patch to directory, represented by $PATCH_TOP variable 

echo $PATCH_TOP

Output: $NE_BASE/EBSapps/patch

$ cd $PATCH_TOP

$ unzip <patch>.zip

NOTE: Must go through Readme.html or readme.txt.

STEP 3:

a. 

$ adop phase=prepare

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Wait for the prepare phase to be complete.

Please check the status from below points:

$ adop -status 

Enter the APPS username: apps 

Enter the APPS password: passwd

After completion Prepare phase, go to next phase called Apply phase:

b. 

$ adop phase=apply patches=1234,7891 workers=8

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

After completion of apply phase, goto next phase called Finalize phase:

i.e like Ready the instance for cutover

c. 

$ adop phase=finalize

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

VERY IMPORTANT 1: 

Up to this phase, you can run a special phase called abort, which will undo the changes made so far in the patching cycle. After cutover is complete, however, you cannot do this.

VERY IMPORTANT 2: 

In an online patching cycle, the requisite JAR files are initially stored in the $APPL_TOP/admin/<SID>/out directory, and then uploaded into the database during the cutover phase. Therefore, the out directory must not be deleted at least until cutover (next phase) is complete.

d.

$ adop phase=cutover

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

NOTE:

Shut down internal concurrent manager. The adop utility signals the internal concurrent manager to shut down, but will wait for any existing concurrent requests to finish before it proceeds with cutover actions.

Cutover will take longer if it has to wait for long-running concurrent processes to complete. In such a case, you can expect to see an informational message of the form: [STATEMENT] [END 2013/10/28 23:47:16] Waiting for ICM to go downIf you do not want to wait for in-progress concurrent requests to finish normally, you can terminate the internal concurrent manager by executing the adcmctl.sh abort command from a different shell.

After completion of Cutover phase, run next Cleanup Phase

e.

$ adop phase=cleanup

NOTE: We can do this step after instance is released to Customer.

NOTE: Various actions are performed during cleanup, including dropping (removing) obsolete: Cross edition triggers, Seed data, Editioned code objects (covered objects), Indexes, Columns, Editions.

Using parameter cleanup_mode:

a) cleanup_mode=quick – Performs minimum cleanup, including removal of obsolete crossedition triggers and seed data.

Use quick cleanup when you need to start the next patching cycle as soon as possible. For example, if you want to start a new patching cycle right away, but have not yet run cleanup from the previous patching cycle, you can use quick cleanup mode to complete the essential cleanup tasks as fast as possible.

b) cleanup_mode=standard – Does the same as quick mode, and also drops (removes) obsolete editioned code objects (covered objects).This is the default mode , so does not need to be specified.

c) cleanup_mode=full – Performs maximum cleanup, which drops all obsolete code and data from earlier editions

Use full cleanup when you want to recover the maximum amount of space in the database. If you have run a large number of patching cycles, or applied a very large patch such as a rollup, significant space may be consumed by obsolete table columns and recovered by running a full cleanup. A full cleanup should only be performed when there is no immediate need to start a new patching cycle.

IMP: Keep checking Status after every Phase by using below command:

$ adop -status 

Enter the APPS username: apps 

Enter the APPS password: passwd

STEP 4:

NOTE:

After successful patch completion always check adop logfiles for any errors. Even if adop completes successfully, there may be some hidden warnings. I prefer to use this egrep pattern, catching most common errors:

The adop log files are located on the non-editioned file system (fs_ne), under:

$ cd $NE_BASE/EBSapps/log/adop/6

$ egrep -i "error|fail|ora-|ac-|rc-" ./*

STEP 5:

Verify that patch was successfully applied using query:

SQL> select * from ad_bugs where bug_number='&num';

SQL> select * from ad_applied_patches where patch_number='&num';

STEP 6:

It's a good practice to verify login to the EBS after patching to make sure that everything still works as it did.

Check if all manager are up and the actual and target values match.

Submit a request like " Active users " to check if requests are getting completed

#######  IMPORTANT   ##########

Note: 

THERE ARE TWO SPECIAL PHASES:

A) ABORT PHASE DETAILS:

Abort PHASE is conditional phase. This phase cannot be specified with any other phase.

If for some reason either the prepare or apply phase failed or gave problems, you can abort the patching cycle at either of these points by running a special phase with the Command. The actions taken will be discarded (rollbacked).

IMPORTANT:

This abort command is only available up to (but not including) the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle.

How to execute:

The command to perform this operation is:

$ adop phase=abort

What it will do:

• Confirms that there is an in-progress online patching cycle, so the abort call is therefore valid.

• Checks for the existence of a patch edition and drops one if it exists.

• Cancels the ADZDPATCH concurrent program, if it is running.

• Deletes the rows inserted for the pending session ID from the ad_adop_sessions and ad_adop_session_patches tables.

VERY IMPORTANT: 

After running abort, a full cleanup must be performed. The cleanup command is: adop phase=cleanup cleanup_mode=full). This will remove any columns that were added by the patch but are no longer needed because of the abort. If they are not removed, they may cause problems in a later patching cycle.

Alternatively, you can run a combined command to abort the patching cycle and perform a full cleanup:

$ adop phase=abort,cleanup cleanup_mode=full

If any attempt was made to apply patches to the patch edition, after abort you must run the fs_clone phase (adop phase=fs_clone) to recreate the patch file system.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

B) FS_CLONE PHASE DETAILS:

The fs_clone phase is a command (not related to adcfgclone.pl) that is used to synchronize the patch file system with the run file system. The fs_clone phase should only be run when mentioned as part of a specific documented procedure.

How to execute:

The fs_clone phase is run using the following command:

$ adop phase=fs_clone

What it will do:

This phase is useful if the APPL_TOPs have become very unsynchronized (meaning that there would be a large number of delta patches to apply). It is a heavyweight process, taking a backup of the entire current patch APPL_TOP and then cloning the run APPL_TOP to create a new patch APPL_TOP. As this method requires more time and disk space, it should only be used when the state of the patch file system is unknown. This command must be invoked from the run file system, before the next prepare phase is run.

NOTE: 

The patch file system requires at least 25 GB of free disk space to be available for adop operations, including fs_clone. If there is insufficient free space, the adop operation will fail.

If an fs_clone operation fails, you can rerun it with the option force=yes to restart it from the beginning (with the same session ID), or force=no to restart it from the point where it failed.

###IMPORTANT POINTS REGARDING ONLINE PATCHING ####

1. adop utility is put under $APPL_TOP_NE/ad/bin. It is a wrapper script which calls internally the perl script $AD_TOP/bin/adzdoptl.pl which does actual work of applying the patch.

2. adop will automatically set its environment as required, but it is the user’s responsibility to set the environment correctly for any other commands that may be run. Set the run edition environment whenever executing commands that you intend to affect the run edition.

For example:

$ . <EBS_ROOT>/EBSapps.env run

$ adstrtal.sh

Set the patch edition environment whenever you intend to execute commands that affect the patch edition.

For example:

$ . <EBS_ROOT>/EBSapps.env patch

$ sqlplus apps/apps @my_custom_patch_script.sql

3. All the phases need to be completed and you can’t skip any of these. For example, if you try to skip prepare phase, you may get error message like “Apply phase can only be run while in a patching cycle, i.e. after prepare phase.”

4. After an online patching cycle is started, you should not perform any configuration changes in the run edition file system. Any that are made will not be propagated and will therefore be lost after cutover is complete.

5. You should not attempt to clone an Oracle E-Business Suite system while an online patching cycle is in progress.

6. The prepare, apply, and fs_clone phases all require at least 10GB of free disk space. All other phases require 1GB of free space. A warning message will be displayed if less than the needed amount is available.

7. The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes. This is also a requirement for patches applied in a hotpatch session.

8. Maintenance Mode is not needed for online patching, and so Maintenance Mode is not available in Oracle E-Business Suite Release 12.2.

##### ADOP ON MULTI-NODE   #######

In a multi-node environment, one application tier node will be designated as the primary node. This is the node where the Admin Server is located, and will usually also be the node that runs Oracle HTTP Server. All other application tier nodes are designated as secondary nodes.

adop commands are invoked by a user on the primary node. Internally, adop uses Secure Shell (ssh) to automatically execute required patching actions on all secondary nodes. You must set up passwordless ssh connectivity from the primary node to all secondary nodes.

IMP:

If a node unexpectedly becomes inaccessible via ssh, it will be abandoned by adop, and the appropriate further actions taken. Consider a scenario where the adop phase=prepare command is run in a system with ten application tier nodes. The command is successful on nine nodes, but fails on the tenth. In such a case, adop will identify the services enabled on nodes 1-9. If they are sufficient for Oracle E-Business Suite to continue to run normally, adop will mark node 10 as abandoned and then proceed with its patching actions. If they are not sufficient, adop will proceed no further.

 *************************************** 3 *****************************************

Select bug_number,creation_date from ad_bugs where bug_number='bug_number';

select patch_name,creation_date from ad_applied_patches where patch_name ='patch_name';

prepare       : Prepare the instance for online patching.

apply         : Apply patch(es) to the Patch Edition.  

finalize      : Ready the instance for cutover.  

abort         : Abort the patching cycle.  

cutover       : Promote the Patch Edition to Run Edition.   

cleanup       : Drop obsolete objects and seed data from Old Editions.

actualize_all : Actualize all objects in the Patch Edition.   

cleanup_full  : Cleanup and drop Old Editions.   

abandon       : yes|no - Abandon failed patches.

Download any required technology patches and unzip the contents. The patch contents may be unzipped into  $NE_BASE/EBSapps/patch.

1. Prepare the system for patching. 

   source <EBS_ROOT>/EBSapps.env run

   $ adop phase=prepare 

2. Apply technology patches to the Oracle Home under the Patch f/s using the information below. 

   source <EBS_ROOT>/EBSapps.env patch 

3. Apply any Oracle E-Business Suite patches planned for this patching cycle 

   $ adop phase=apply patches=<patch_list> 

4. After all patches have been successfully applied, complete the patching cycle. 

   $ adop phase=finalize 

   $ adop phase=cutover 

Note: Below steps can be done when applications are up and running 

5. source <EBS_ROOT>/EBSapps.env run 

   $ adop phase=cleanup

6. To complete the process and synchronize the technology level between patch and run f/s. 

   $ adop phase=fs_clone

------------------------------------------HOT PATCHING WITH ADOP--------------------------------------------

Hotpatch which can apply directly on run fs

$ adop phase=apply patches=<patch_list> hotpatch=yes

After hotpatch please run phase=cleanup and phase=fs_clone to sync the run fs with patch fs to prepare for next patching cycle

-------------------------------------------REAPPLY PATCH FORCEFULLY---------------------------------------

$ adop phase=apply patches=<patch list> hotpatch=yes options=forceapply

-------------------------------CONTINUE AS IF PATCH APPLY WAS SUCCESSFUL------------------------

adop phase=apply patches=<patch list> abandon=no restart=yes flags=autoskip

--------------------------------TO DEFINE WORKERS IN ADOP-------------------------------------------------

$ adop phase=apply patches=<patch list> workers=5

-----------------------------TO DEFINE LOCATION OF PATCH TOP--------------------------------------------

$ adop phase=apply patches=<patch list> patchtop=<patch location base>

-------------------------------------------MERGE PATCHES----------------------------------------------------------

$ adop phase=apply patches=<patch list> merge=yes

-----------------------------RESTARTING ADOP FROM A FAILED SESSION---------------------------------

 $ adop phase=abort

 $ adop phase=cleanup cleanup_mode=full

 $ adop phase=fs_clone 

Then reapply the patch

-------------------------------------NON-INTERACTIVE------------------------------------------------------------

$ adop phase=apply options=nocopyportion patchtop=$XLA_TOP/patch/115 patches=driver:xla5584908.drv

-----------------------STEPS TO FOLLOW TO SKIP FAILED WORKERS-------------------------------------

1. Use adctrl and select option#8 (This will not be visible) to skip the failed jobs

2. Restart adop using "restart=yes" parameter

** If the failed jobs are numerous, then better to re-start this patch once again with autoskip option.

ie.  adop restart=no abandon=yes flags=autoskip 

This command will restart the patch once again from starting onwards and will skip all the failures if any comes. But make sure to review the log file at the end of the patch application that you have skipped the failures that you want to.

How To Automatically Set the Current Run or Patch Edition / File System for EBS 12.2 

For 12.2.2:

Change directory to the Base directory and run script EBSapps.env giving "run" or "patch" as argument, eg:

cd /u01/oracle/EBS122

. ./EBSapps.env run

  E-Business Suite Environment Information

  ----------------------------------------

  RUN File System : <EBS base dir>/fs1/EBSapps/appl

  PATCH File System : <EBS base dir>/fs2/EBSapps/appl

  Non-Editioned File System : <EBS base dir>/fs_ne

  DB Host: <hostname.domain name>  Service/SID: <SID>

  Sourcing the RUN File System ...

 If EBSapps.env is run without the file system as argument this will be prompted for, eg:

. ./EBSapps.env

  E-Business Suite Environment Information

  ----------------------------------------

  RUN File System : <EBS base dir>/fs1/EBSapps/appl

  PATCH File System : <EBS base dir>/fs2/EBSapps/appl

  Non-Editioned File System : <EBS base dir>/fs_ne

  DB Host: <hostname.domain name>  Service/SID: <SID>

  E-Business Suite Environment Setting

  ------------------------------------

  - Enter [R/r] for sourcing Run File System Environment file, or

  - Enter [P/p] for sourcing Patch File System Environment file, or

  - Enter anything else to exit

  Would you like to set the E-Business Suite environment [R/P]:R

  Sourcing the RUN File System ...