Oracle Patching Fundamentals For 12.2.x

PATCHING

 

Patching Principles and Nomenclature

Patching Principles

As described in more detail later in this chapter, all Oracle E-Business Suite Release 12.2 patches are applied online: that is, to a running system with users logged on and working in their normal way. Use of online patching is mandatory in Release 12.2.

Online Patching

Traditionally, Oracle E-Business Suite has needed to be shut down to apply patches and bug fixes of various kinds. Scheduling the sometimes-lengthy downtimes needed can prove difficult to combine with requirements for the highest possible availability of business-critical systems. This is exacerbated by the increasing move to consolidate systems into a single instance that may support worldwide operations, and which consequently does not have a natural downtime window.

The introduction of online patching in Oracle E-Business Suite Release 12.2 greatly reduces the need for such downtime. All Oracle E-Business Suite patches in Release 12.2 are applied online.

Key features of online patching include:

·       Oracle E-Business Suite patching operations are carried out while the applications are in use and users are online. This is accomplished by means of a copy of the application components that are stored in the database and file system.

·       Oracle E-Business Suite patching is performed using the new adop (AD Online Patching) utility.

·       A short period of downtime is required, but this amounts to little more than a restart of the application tier services: the time the applications are unavailable is measured in minutes rather than hours, and this can be set to be at the most convenient time, for example outside normal business hours.

·       Maintenance Mode is not needed or used in Release 12.2.

The patching model used before Release 12.2 was designed to minimize downtime by performing its operations in the shortest time possible, using whatever resources are needed. In contrast, the online patching model is designed to minimize downtime by allowing patching operations to be performed while users remain on the system.

Overview

In traditional patching, application of a patch is a single logical operation. In online patching, it can be thought of as a series of related steps:

1.     A copy is made of the code of the running system that resides in both the file system and database.

2.     Patches are applied to the copy while users continue to access the running system.

3.     The copy becomes the new running system.

4.     The original running system (now obsolete) is recycled for use in future patching cycles.

These steps constitute the online patching cycle. To implement this mechanism, various changes have been made to the Oracle E-Business Suite infrastructure.

Any method used to create a copy of the running application must take into account that an Oracle E-Business Suite application comprises both code and data, stored in the database and the file system. A key concept is the edition as a copy of the application code: the running application executes on the run edition, while any patching activity takes place in the patch edition. The implementation mechanism uses the Oracle Database Edition-Based Redefinition (EBR) feature to cater for database objects, and a dual file system to cater for objects stored in the file system.

Two complete file systems are always present in an Oracle E-Business Suite Release 12.2 system. As noted above, the run file system is the one currently in use by the running application, while the patch file system is the either being patched or awaiting the start of the next online patching cycle. In other words, the two file systems swap roles with every online patching cycle.

In the database, there is also a run edition and patch edition, but they do not swap roles back and forth as in the file system: the patch edition only comes into existence during a patching cycle, and becomes the new run edition at the end of the cycle. The former database run edition (the old edition) and the obsolete objects it contains are discarded at the end of an online patching cycle, and the space reclaimed during the cleanup phase.

These concepts will all be described in more detail after a discussion of the way they are used in the online patching cycle.

The Online Patching Cycle

The online patching cycle is divided into five phases:

 

Patching Cycle Overview

The key actions in the various stages of the online patching cycle can be summarized as follows:

• Prepare

o   Synchronizes patch edition and run edition on the file system.

o   Creates a new patch edition in the database.

• Apply

o   Executes patch drivers to update patch edition.

o   Patches applied: can be one or many, including customizations.

• Finalize

o   Compiles invalid objects.

o   Generates derived objects.

• Cutover

o   Configures patch edition file system to be the new run edition file system.

o   Configures patch edition of database to be the new run edition.

o   Restarts application tier services.

• Cleanup

o   Delete obsolete code and seed data to recover space.

Each of the phases will now be discussed in more detail.

Prepare

The following actions are taken in this phase.

On the file system:

1. The patch file system is synchronized with the run file system.
2. The patch edition files become an exact copy of the run edition files.

Note: By default, synchronization is incremental: that is, only files that were changed in the last patch application are copied.

In the database:

1. A patch edition is created in the database.
2. All code objects in the patch edition begin as pointers to code objects in the run edition. Code objects in the patch edition begin as lightweight "stub objects" that point to the actual object definitions, which are inherited from earlier editions. Stub objects consume minimal space, so the database patch edition is initially very small in size.
3. As patches are applied to the patch edition, code objects are actualized (have a new definition created) in that edition.

Note: Storage objects (such as transaction tables) are not copied.

Stub Objects.

A stub object is a shared object, built entirely from mapfiles, that supplies the same linking interface as the real object, while containing no code or data. Stub objects cannot be used at runtime.

However, an application can be built against a stub object, where the stub object provides the real object name to be used at runtime.

Apply

The following actions are taken in this phase:

1. Patches are applied to the patch edition. During this process, any changed stub objects will become actual code objects in the patch edition.
2. The changes introduced by the patches are made in the isolation of the patch edition.

o   Changes to code objects are only visible in the patch edition of the file system and database.

o   Changes to tables are stored in new columns or rows that are only visible to the patch edition.

Note: At this point, users still remain connected to the application and performing their work.

Finalize

This phase is used to perform the final operations that can be executed while the application is online:

1. Invalid objects are compiled.
2. Derived objects are generated.
3. Any actions that must be performed at cutover are pre-computed and stored for quick execution at that time.

Cutover

This phase involves:

1. Shutdown of application tier services.
2. Execution of any required cutover actions to maintain non-editioned objects and data.
3. Configuration of the patch edition of the file system as the new run edition.
4. Configuration of the patch edition of the database as the new run edition.
5. Restart of application tier services on the new run edition.

Although the cutover phase does require a short period of application tier services downtime, the online patching cycle can be paused for as long as required prior to running this phase. You could, for example, add such a pause to ensure that the downtime period will be outside business hours.

Note: The database remains open throughout the entire online patching cycle, including cutover.

Cleanup

The following database actions are taken in this phase, which occurs after users have been brought back online to the newly-patched application:

1. Old code objects that are no longer visible in the run edition are dropped.
2. Old data (rows or columns) that is no longer visible in the run edition is deleted or dropped.
3. Old database editions that no longer contain actual objects are dropped.

Database Implementation

Creating a copy of the database part of the running system has been accomplished by taking advantage of the Oracle Database Edition-Based Redefinition (EBR) feature. This allows an application to efficiently store multiple copies of its application definition in the same database, and thereby enables online upgrade of the database tier.

The term edition refers to the isolation mechanism that allows pre-upgrade and post-upgrade schemas to co-exist. The simplest way to think of an edition is as a separate (isolated) copy of all database code objects that are changed by a patch.

The three types of database edition are:

·       Run: Online users connect to this edition, which is always the default database edition. The run edition will always exist.

·       Patch: Patching tools connect to this edition. A child of the run edition, the patch edition exists only while a patching cycle is in progress.

·       Old: When a patch edition is promoted to be the run edition, the previous run edition is now regarded as an old edition. There may be zero or more old editions at a given time. They are discarded when a full cleanup (described later) is performed. You cannot connect to an old edition.

File System Implementation

Creating and using two file systems allows one (run file system) to be part of the running system, while the other (patch file system) is either being patched or waiting to be patched during the next patching cycle.

Note: The file systems are designated File System 1 and File System 2 in Rapid Install. These are abbreviated to fs1 and fs2 respectively.

The two file systems are sometimes referred to as a dual file system, and swap roles at the end of each patching cycle. That is, the file system that has just been patched is put into use as part of the running system (becoming the new run file system), and the previous run file system takes over the the role of patch file system (in readiness for commencement of the next patching cycle).

Important: The existence of the dual file system has significant implications for general (non-patching) maintenance activities. For information on understanding how to decide which file system to run AD tools from, refer to: Choosing the Correct File System For Maintenance Tasks, Oracle E-Business Suite Maintenance Guide (Chapter 7, page 7-1).

However, two file systems are not sufficient to meet all the practical needs of an online patching environment for Oracle E-Business Suite. A third file system, described in the next section, is also required.

Non-Editioned File System

The dual file system approach caters for application code, but applications also use the file system to read and write business data. In Release 12.2, application data files are stored in a third area, the non-editioned file system (fs_ne), which is used to store data that is needed across all file systems. Non-editioned files are not copied or moved during patching: their location remains constant across online patching cycles.

The Three File Systems: fs1, fs2, and fs_ne

The non-editioned file system is therefore completely separate from file system 1 and file system 2.

Context Variables for Online Patching File Systems

Several AutoConfig context variables support the file systems used in online patching. For example, the s_ne_base context variable stores the location of the non-editioned file system, and AutoConfig sets the environment variable $NE_BASE to the corresponding value. AutoConfig also sets the environment variables $RUN_BASE and $PATCH_BASE, so you can easily tell which is currently the run file system and which the patch file system.

Files Stored in the Non-Editioned File System

The non-editioned file system is designed to store files that contain transactional data and reference data. Examples include: import, export files, general log files, and concurrent manager log and out files. These are all examples of files that are not modified during an online patching cycle.

More specialized examples include:

• Batch upload and download files.
• Files used to transfer transactional data from processes external to Oracle E-Business Suite (for example, where a third-party order entry system delivers orders via order import files).
• Files containing transactions that are needed across all file systems.

Note: The non-editioned file system is not designed to store shared files, because initially identical files can become non-identical during patching life cycles. Nor is it designed to store code, which is editioned (one copy can be in the run file system while the other is in the patch file system).

Concurrent Processing and the Non-Editioned File System

As noted above, concurrent processing in particular makes significant use of the non-editioned file system, with all concurrent manager log and out files being stored there. This has some additional implications, as described below.

Before Release 12.2, there were two choices for the location of log and out files. The first was $APPL_TOP, and the second was $APPLCSF (which pointed to $APPL_TOP/admin/$TWO_TASK). The location used was determined by whether the $APPLCSF environment variable was set.

With Release 12.2, the $APPL_TOP contains only code and configuration files. Therefore, it is mandatory for the $APPLCSF environment variable to be set. Its default value points to the non-editioned file system, specifically to <s_ne_base>/inst/<EBS_Instance_ID>/logs/appl/conc: for example, /u01/R122_EBS/fs_ne/inst/VIS_app101/logs/appl/conc.

In addition, a new environment variable, $APPLLDM, has been introduced to provide the option of organizing storage of log and out files on a product-specific basis.

In Release 12.2, $APPLLDM environment variable can have multiple possible values. The two main ones are 'single' and 'product'.

If $APPLLDM is set to 'single', concurrent processing and AD Administration log files will both be stored under $APPLCSF/log. Concurrent processing out files will be stored under $APPLCSF/out. This is the default setting.

If $APPLLDM is set to 'product', the concurrent processing log and out files will be written to product-specific directories under $APPLCSF. This is in contrast to previous releases, where the directories were under $APPL_TOP. For example, concurrent manager log files will be located under $APPLCSF/<product>/log, AD Administration log files will be located under $APPLCSF/log (i.e. the same location as with the 'single' setting), and PO reports will be located under $APPLCSF/po/out.

Beginning with Release 12.2.6, Oracle E-Business Suite provides additional storage strategies for management of large numbers of concurrent processing log and output files. For example, concurrent request log and out files can be organized by date or by user name. These storage strategies are called schemes. For more information, refer to the Oracle E-Business Suite Setup Guide.

In summary, the default behavior has been designed to be exactly the same as in previous releases: that is to say, if you did not set $APPLCSF before, and you do not set it or $APPLDLM now, you will not notice any difference in concurrent processing log file management or storage.

Editioned and Non-Editioned Objects

EBR automatically manages versioning of objects that support editioning, such as code objects. However, not all objects can be editioned: most notably, this includes transactional data, of which there is only ever one copy.

Oracle E-Business Suite Release 12.2 introduces a new concept, the logical view of the data model. This is implemented via synonyms and editioning views, which isolate the running application from changes to the data model that may be introduced by a patch. That is to say, data model changes are guaranteed to not affect the running application. Data model changes are implemented as new columns on the table, which are exposed to the patch edition of the application via the editioning view. New tables can be introduced by an extension of the same principle.

 ORACLE E-BUSINESS SUITE RELEASE 12.2: ONLINE PATCHING FAQ (DOC ID 1583902.1)

 

What are the phases that make up the Online Patching cycle?
The Online Patching cycle consists of the following phases:

1. Prepare a virtual copy (patch edition) of the running application (run edition).
2. Apply patches to the patch edition of the application.
3. Finalize the system in readiness for the cutover phase.
4. Cutover to the patch edition and make it the new run edition.
5. Cleanup obsolete definitions or data to recover space.

How does Online Patching work on the application tier?
During Release 12.2 installation, Rapid Install will lay down two copies of the application tier file system. One of the copies will be labeled as the run file system, and the other as the patch file system. Subsequently, when a patch is applied, adop will:

1. Synchronize the contents of the run file system to the patch file system. This happens during the prepare phase.
2. Perform patching actions on the patch file system. This happens during the apply phase.
3. Finally, during the cutover phase, the adop utility restarts the application tier services. The patch file system is then promoted to be the new run file system, and the old run file system becomes the patch file system for the next patching cycle.

Note:

There is a third file system, the non-editioned file system (fs_ne), is created to store files containing data that is needed across all file systems, such as log files.

How do I apply Oracle Fusion Middleware patches in Oracle E-Business Suite Release 12.2?
During the apply phase of an Online Patching cycle, you apply Oracle Fusion Middleware patches to the Oracle homes of the patch edition file system. Then, after the cutover phase is complete, you synchronize the file systems by performing an fs_clone operation.

Is it possible to apply patches in advance of the downtime period required by cutover?
Yes, you can apply online patches at any time in advance of a suitable cutover point, and then run in that state (with cutover pending) for as long as you need to.

Does the hotpatch option still exist in an Online Patching environment?
Yes, but it is only supported for use with patches that have been designed and tested to be applied in this way. This is because hotpatch mode applies changes to the run edition while this edition is in active use, which may result in one or more of the following issues:

• Runtime transactions may fail due to invalid objects.
• Runtime transactions may fail due to loss of PL/SQL package state.
• Application code and database objects may be temporarily inconsistent.
• Seed data may change, and may be temporarily inconsistent.
• Tables that are patched will be temporarily inconsistent.
• Code and data cached in application tier server memory may be inconsistent with changes made by the hotpatch.
• Runtime processing may hold long-term locks on code or data, leading to execution failures in the hotpatch.

What are the key differences between the DBA_OBJECTS, DBA_OBJECTS_AE, and AD_OBJECTS tables?

• DBA_OBJECTS shows object information for the current edition, but the STATUS column in this view may show the object as VALID even if the object actually needs to be compiled before use. This is a side effect of how the database handles objects in old editions that have been inherited into the current edition. The issue is tracked in database Bug 17777718.
• DBA_OBJECTS_AE is similar to DBA_OBJECTS, but shows object information across all editions. This has the drawback of showing objects in old editions that are no longer accessible to the application.
• AD_OBJECTS is the Oracle E-Business Suite workaround to the unreliable STATUS column in DBA_OBJECTS. AD_OBJECTS shows the correct status for each object visible in the current edition. It also shows whether the object is “actual” (a real object) in the current edition, or a "stub" object (the object definition was inherited from a previous edition). You can query AD_OBJECTS to locate objects that need to be recompiled before use:

SQL>select owner, object_name, object_type
from ad_objects
where status = 'INVALID'
order by 1,2,3
/

The same logic can be applied by running the script:

SQL>sqlplus apps/apps @$AD_TOP/sql/ADZDSHOWINVALID

Does Online Patching have any performance implications?
Online patching is designed to run efficiently alongside the running application. While an Online Patching cycle is in progress, there is a small additional processing overhead. The degree of this additional overhead can be managed by controlling the number of parallel workers used by Online Patching.

Does Online Patching increase the network port requirements on an Oracle E-Business Suite instance?
Yes. Online patching requires an additional set of network ports for the Oracle WebLogic Server managed servers on the second file system. During the cutover phase, the managed servers run simultaneously on the patch file system and run file system for a brief period, in a rolling transition process.

Can I apply multiple patches in an Online Patching cycle?
Yes. We recommend that if you have multiple patches to apply, you apply them in a single Online Patching cycle. This will minimize the overall time needed.

Is it possible to abort an Online Patching session?
Yes. Up to cutover, you can run the abort phase to undo the changes made so far in the patching cycle. It is not possible to back out patches once cutover is complete. Note that you should always follow an abort directly with a full cleanup.

Is there any limitation on the locations of the patch and run file systems?
Both the patch and run file systems must be located on the same node (machine).

Is the shared APPL_TOP configuration supported with Online Patching?
Yes. A shared APPL_TOP configuration is supported and recommended for multi-node application tier implementations in Release 12.2.

Does the adop utility support non-interactive mode?
Yes. The adop utility can be executed non-interactively, using an input file to supply the parameters that you would have supplied on the command line. However, using this option means you will have to allow for the downtime of the cutover phase occurring automatically - perhaps at a time that is not convenient.

 

What is duplicated on the second file system used by Online Patching?
The second file system contains a copy of all the components that make up an application tier file system, including:

• APPL_TOP - Oracle E-Business Suite code
• INST_TOP- Instance Configuration Home
• FMW_HOME - Oracle Weblogic Server Home and Oracle E-Business Suite Domain
• ORACLE_HOME - Oracle Application Server Home, Forms, Reports
• IAS_ORACLE_HOME - Oracle OHS Home
• COMMON_TOP - Oracle E-Business Suite Java code, third-party libraries

How does adop work in a multi-node environment?
The adop Online Patching tool uses remote APIs and ssh login to execute patching operations on remote nodes in a multi-node environment. The node that launches adop becomes the ‘master’ node, and the remote nodes are referred to as ‘slaves’.

How do I determine the status of my Online Patching session?
You can run the adop -status command. This will display information that includes phases completed and the time taken. If you want additional details of operations performed, you can run the adop -status -detail command.

What is downtime mode and when can it be used?
To optimize the process of upgrading to Oracle E-Business Suite Release 12.2, the AD Delta 5 Release Update Pack introduced downtime mode, which is employed as follows:

$ adop phase=apply patches=<patch_number> apply_mode=downtime

 

Downtime mode does not use an online patching cycle. As a result, less time is needed for the actual application of a patch than in online mode, but at the cost of increased system downtime. When applying Oracle E-Business Suite patches in this mode, adop will first confirm that the application tier services are down, and will then proceed to apply the patch to the run edition of the Oracle E-Business Suite database and file system.

Downtime mode is supported for:

• All patching (including post-upgrade patching) that forms part of the Release 12.2 upgrade process and is completed before the system is scaled up, the application tier services are started, and users log in to the upgraded system.
• Single-node development or test environments, where production support and high availability are not required.

Downtime mode allows the 12.2 upgrade process to be completed as quickly as possible. Once the upgrade is complete and users are online, all subsequent patching on a production system should use online mode, not downtime mode, unless the patch readme states otherwise. 

Several restrictions apply to the use of downtime mode:

• You cannot validate successful patch application before cutover to the updated code takes place.
• There is no capability to abort a failed patch and return to the existing run edition.
• Release 12.2 patches are not normally tested in downtime mode.
• Use of downtime mode in a multi-node application tier environment is not tested or supported.

 

Note: Downtime mode is only supported for production use where explicitly documented, or otherwise directed by Oracle.

Should I source the environment before executing adop?
This is not required, as adop will automatically source the correct environment for each phase. However, you must source the correct environment when using most other admin scripts or tools.

Do I need to stop services during an Online Patching cycle?
This is not required. The adop utility will automatically stop and restart the services during the cutover phase.

Which services need to be running for adop to work?
The adop utility requires that the database is up and running when the various Online Patching phases are being executed. However, adop will work if the application tier services are down. When necessary, it will start these services.

What can I do to reduce the time required for cutover?
It is important to distinguish between the time needed for the whole cutover phase, and the downtime period within the phase. The actual downtime (during which users cannot log in) is significantly shorter than the whole phase. To help reduce the overall time taken by cutover, you can do three things:

• Run the finalize phase explicitly, to obviate the need for cutover to do so.
• Shut down the concurrent managers before running cutover, to avoid having to wait for concurrent requests to complete. Alternatively, ensure no long-running concurrent jobs are submitted while a patching cycle is in progress.
• Ensure you are using the maximum number of parallel workers your system will support.

I have already run the finalize step in this patching cycle. Why is cutover running the finalize phase again?
You can run the finalize phase at any point during the Online Patching cycle. If you do not run the finalize phase just before cutover, cutover will run it again to process the latest changes (such as those in an apply command).

What is fs_clone and how is it used?
The command adop phase=fs_clone is a special command that is used to copy the run file system to the patch file system.

How do I apply or patch my customizations in Oracle E-Business Suite Release 12.2?
You should apply your customizations to the patch edition during the apply phase of an Online Patching cycle. The changes will subsequently be propagated to the new run edition, along with the fixes in the patches that were applied during the patching cycle. For more information, refer to Section 1.4.1: Recommended Approaches for Deploying Custom Code in My Oracle Support Knowledge Document 1577661.1, Developing and Deploying Customizations in Oracle E-Business Suite Release 12.2.

If custom code is installed on a separate database schema, do I have to edition-enable my custom database schema?
The coding standards in the Oracle E-Business Suite Developer's Guide state that the first step to any custom application development is to register the custom Oracle schema with the Oracle E-Business Suite applications. The Online Patching enablement patch enables editioning on all the schemas registered with the application. If you follow this process, your schema will be edition-enabled automatically.

SOME DEEP INSIGHTS ON THE PHASES OF ADOP PATCHING CYCLE

PREPAPE:

Some of the highlights of the PREPARE phase are listed below:

• Gets patching filesystem ready to apply a patch.
• Checks that ETCC has been run against the database Oracle Home and that all required patches are applied.
• synchronize filesystems - If any configuration changes have been made, this will cause the prepare phase to initiate a fs_clone of the current run filesystem to the patch filesystem.
• run autoconfig if necessary.
• check tablespaces for sufficient free space before starting.
• check APPS_FNDFS listener to make sure concurrent requests will launch.
• launch concurrent request - "Online Patching In Progress" (ADZDPATCH).
• synchronize snapshots between run and patch filesystems.

APPLY:

Some of the highlights of the APPLY phase are listed below:

FINALIZE / CUTOVER / CLEANUP:

Some of the highlights of these phases are listed below:

FINALIZE:

• compiles invalids objects in the patch edition.
• determines DDL statements to execute at cutover.
• validates that the system is ready for a cutover.
• gathers statistics for performance.

CUTOVER:

• once started, the abort phase cannot be issued.
• runs the finalize phase if it was missed.
• cancels concurrent request - "Online Patching In Progress" (ADZDPATCH).
• shuts down the internal concurrent manager.
• stops any processes that may have been started on the patch filesystem.
• stops the RUNNING applications tiers.
• executes the DDL statements prepared during the finalize phase.
• loads jar files into the database.
• synchronizes snapshots between run and patch filesystems.
• broadcasts a message to local users that a cutover is taking place and to re-source their working environment.

CLEANUP:

• cleans up old editions
• explains the three methods of cleanup.
• QUICK cleanup performs removal of obsolete crossedition triggers and seed data.
• STANDARD cleanup performs removal of obsolete crossedition triggers and seed data, drops (removes) obsolete editioned code objects (covered objects). (DEFAULT MODE)
• FULL cleanup performs maximum cleanup, which drops all obsolete code and data from earlier editions

 

FS_CLONE:

Some of the highlights of the FS_CLONE phase are listed below:

• verifies cleanup was run and runs it if it has not.
• performs the clone stage
• performs the clone to patch filesystem
• synchronizes snapshots between run and patch filesystems.
• stops any patch filesystem processes that were started.

BEHIND THE SCENES FOR ADOP

adop will perform the tasks required to apply the patch as mentioned below:

• Reads patch metadata to determine patch dependencies and requirements

• Uploads patch information from a prior patch session to the database (if applicable)

• Reads and validate the patch driver file and reads the product driver files

• Compares version numbers of object modules from the product libraries and

  version numbers of the existing files against the patch files

• Backs up all existing files that will be changed by the patch

• Copies files

• Archive files in libraries

• Relinks executables

• Generates forms, reports, messages, graphics, and Java archive (JAR) files

• Compiles JSP files and invalid database objects

• Updates database objects

• Runs AutoConfig to update configuration files if any template files are introduced

  or updated by the patch

• Saves patch information to the database

è Be aware of the following important points about adop:

• The adop utility always runs from the run edition file system. It automatically sets its

  environment correctly, regardless of the edition it is run from.

• If a patch contains no new updates to files or database objects in your system, adop

  takes no action.

• If adop detects a previously failed patching session, it will attempt to recover that

  session.

ADOP PARAMETERS

ADOP PATCHING MODES

The adop utility is normally used to apply patches in an online patching cycle. It can also be used:

• To run a patching cycle, and test patch application without actually taking any apply actions, in test mode

• To apply patches outside a patching cycle in downtime mode

• To apply patches without connecting to the database in preinstall mode

 

Each of these is described further below.

Test Mode:

In test mode, adop does not apply the patch. Instead, it lists each file it would have copied, relinked, executed, or generated, and shows exactly what actions it would have performed had it applied the patch. It also runs AutoConfig in test mode to determine any impending changes to the configuration files. This allows you to see the effects of a patch on your system before you apply it.

To run adop in test mode, add the apply=no parameter to the adop command you would use if you were actually going to apply the patch. In test mode, adop will go through the process of applying the patch but will not perform any of the following

actions:

• Copy files from the patch directory to the Oracle E-Business Suite file system

• Archive object modules into the product libraries

• Relink executables

• Generate forms, reports, PL/SQL libraries, or menu files

• Run SQL or EXEC commands (commands that change the database)

• Instantiate new configuration files

• Update the patch information files

• Update patch information and release version in the database

 

Downtime Mode:

To optimize the process of upgrading to Oracle E-Business Suite Release 12.2, support is provided for the capability to apply Oracle E-Business Suite patches in downtime mode. When applying patches in this mode, adop will first confirm that the application tier services are down, and will then proceed to apply the patch to the run edition of the Oracle E-Business Suite database and file system. Downtime mode patching does not use an online patching cycle. The process of applying a patch in downtime mode completes more quickly than in online mode, but at the cost of increased system downtime.

To run adop in downtime mode, you use the following command line options. In this

example, patch 123456 is applied in downtime mode:

$ adop phase=apply patches=123456 apply_mode=downtime

Important: Be aware that:

• Release 12.2 patches are not normally tested in downtime mode.

• Downtime mode is only supported for production use where

  explicitly documented, or when directed by Oracle Support or

  Development.

 

Preinstall Mode:

Preinstall mode is generally used during the upgrade process to update AD utilities, apply pre-upgrade patches, or work around other patching issues. adop asks all startup questions except those relating to the database. Important: Run adop in preinstall mode only if the patch readme instructs you to do so.

To run adop in preinstall mode, include preinstall=y on the adop command line. It performs the following actions:

• Compares version numbers

• Copies files 

• Relinks FND and AD executables

• Saves patch information to the file system

Because adop does not read driver files in preinstall mode, it copies all product files in the patch to the APPL_TOP directory. Additionally, even if a file in the patch should be both in the APPL_TOP and in another directory (such as in $OA_HTML), adop copies the file only to the APPL_TOP. In preinstall mode, adop validates codelevels against the files Preinstall_Codelevel_AD. txt and Preinstall_Codelevel_MP.txt. These files are located in the $APPL_TOP/admin directory, and contain codelevel information about AD and other products registered in the database tables. Since no database connection is available in preinstall mode, adop tries to validate whether the current patch should be applied based on the codelevel information in these two files, as follows:

• If Preinstall_Codelevel_AD.txt is missing from the APPL_TOP, adop will apply the patch in preinstall mode without validating the patch for codelevel compatibility.

• If Preinstall_Codelevel_MP.txt is missing from the APPL_TOP, adop will proceed

  with patch application without validating the patch for codelevel compatibility of the entities.

• If both files are missing, adop will not validate codelevels in preinstall mode.

 

  Note the following restrictions when applying a patch in preinstall mode:

 

• NLS patches cannot be applied on the instance.

• Baseline or codelevel-introducing patches cannot be applied on the instance.

• adop will not check to see if the patch is already applied on the system.

 

ADOP OPTIONS

The "options" argument is used to pass options that control how the patch is applied. It takes the form of a comma-separated list. Enter a single option, or a comma-separated list of options such as adop options=nocopyportion,nogenerateportion.

Note: As with adop arguments, there must be no space after the comma.

RUNNING ADOP

Step 1: Set the environment

You must set the environment to apply the configuration parameters that define your system. This task is common to many AD utilities, and is performed using the following

command:

$ . <EBS_ROOT>/EBSapps.env run

<EBS_ROOT> represents the Oracle E-Business Suite base install directory, such as /u01/oracle. There is no associated environment variable.

Note: The EBSapps.env file is provided by the AD-TXK release update packs.

Step 2: Unzip the patch

Download and unzip the patch into the patch top directory, which is identified by the $PATCH_TOP environment variable.

Step 3: Review the information in the readme file

In the directory where you unzipped the patch, you will find a README.txt file and a README.html file. Review either readme file for information about the patch and for instructions on using Oracle Patch Application Assistant (PAA) to generate customized instructions for your system.

Step 4: Run Oracle Patch Application Assistant

Run PAA (admsi.pl) to generate customized instructions for your system. Follow the steps in the customized instructions to start the patching process.

Step 5: Run adop

The customized instructions generated by PAA describe how to run adop using the adop command.

A simple example of the commands to execute a complete online patching cycle for patch 123456 is as follows:

$ . <EBS_ROOT>/EBSapps.env run

$ adop phase=prepare

$ adop phase=apply patches=123456

$ adop phase=finalize

$ adop phase=cutover

$ . <EBS_ROOT>/EBSapps.env run

$ adop phase=cleanup

Monitoring Status

You can obtain a brief report for the current patching session by running the command:

$ adop -status

This will display information that includes phases completed and the time taken.

The output is also recorded in a log or output file.

Two additional options with this command are as follows.

• If you want information about a particular session, specify the relevant session ID:

$ adop -status <session ID>

• If you want additional details of operations performed:

$ adop -status -detail

This option will give a summary of last ten adop session IDs, and full details of the file system and database changes introduced by a patch. It also shows the log file location of the current patching cycle.

ABORTING AN ONLINE PATCHING CYCLE

If a patching cycle is failing and the issue cannot be resolved quickly, it is possible to abort the patching cycle and return to normal runtime operation. The patch edition will be dropped.

You can abandon a patching cycle (without applying any patches) by running the command:

$ adop phase=abort

Important:

This abort command can only be used before successful completion of the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle.

Aborting a patching cycle will drop the patch edition, but you must then run the cleanup and fs_clone phases before starting a new patching cycle. The cleanup must be a full cleanup.

For example:

$ adop phase=prepare

$ adop phase=apply patches=123456

[Patch application encounters problems and you want to abort]

$ adop phase=abort

$ adop phase=cleanup cleanup_mode=full

$ adop phase=fs_clone

Optionally, you can combine the abort and cleanup commands as follows:

$ adop phase=abort,cleanup cleanup_mode=full

Note: You cannot abort application of a patch applied in hotpatch mode or downtime mode.

RESTARTING ADOP

If you have shut down the workers, or if adop quits while performing processing actions, it saves all the actions completed up to that point in restart files. Investigate and resolve the problem that caused the failure, then restart adop. After you restart adop, it will ask if you want to continue with the previous session (at the point where the processing stopped), or start a new session.

Note: A difference from adpatch is that adop restart behavior is controlled by the abandon=yes/no and restart=yes/no options in the input_file that can be passed to the adop command in the apply phase.

You have several options when restarting (or abandoning) application of individual patches, as follows.

• If you want to restart a failed patch from where it left off, you only need to specify

restart=yes on the command line:

adop phase=apply patches=1234 restart=yes

If you want to restart a failed patch from the very beginning, you need to specify

abandon=yes on the command line:

adop phase=apply patches=1234 abandon=yes

 

• If you want to ignore a previously failed patch and apply a different one instead, you need to specify the new patch number and abandon=yes on the command line:

adop phase=apply patches=5678 abandon=yes

ONLINE PATCHING CYCLE

The online patching cycle consists of a number of high level phases:

1. prepare

2. apply

3. finalize

4. cutover

5. cleanup

A high level overview of an online patching cycle would, programmatically, look like

this:

# Prepare for patching:

$ adop phase=prepare

# Apply patches:

$ adop phase=apply patches=<patch number>

# Apply any customizations to patch edition (optional):

$ . <EBS_ROOT>/EBSapps.env patch

$ sqlplus apps/apps @my_custom_script_01

$ sqlplus apps/apps @my_custom_script_02

...

# Finalize patch application:

$ adop phase=finalize

# Perform cutover:

$ adop phase=cutover

$ . <EBS_ROOT>/EBSapps.env run

# Perform user acceptance testing via application UI

# Perform cleanup:

$ adop phase=cleanup

Important Additional Points

• After an online patching cycle is started, you should not perform any configuration changes in the run edition file system. Any that are made will not be propagated, and will therefore be lost after cutover is complete.

• The prepare, apply, and fs_clone phases all require at least 10GB of free disk space. All other phases require 1GB of free space. A warning message will be displayed if less than the needed amount is available.

• The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes. This is also a requirement for patches applied in hotpatch or downtime mode.

• Any customizations must be applied to the patch edition during the apply phase, normally after any Oracle E-Business Suite patches have been applied.

Special Phases

Two additional phases are provided for specialized use. Neither can be run in conjunction with any other phase. Further details of these phases are described in later sections.

• The abort phase is used to terminate a patching cycle before it is complete, and roll back any changes that have been made. It can also be run in conjunction with a full cleanup operation.

• The fs_clone phase is a command (not related to adcfgclone.pl) that is used to synchronize the patch file system with the run file system. Normally, the fs_clone phase should only be run when mentioned as part of a specific documented procedure.

Important: You may perform a procedure that as a final step instructs you to run fs_clone. You do not have to do this immediately: the key requirement is to run fs_clone before you start the next patching cycle. And if you are performing multiple procedures, each of which requires fs_clone to be run at the end, you only need to run it once before the start of the next patching cycle.

adop will automatically set its environment as required, but it is the user's responsibility to set the environment correctly for any other commands that may be run. Set the run edition environment whenever executing commands that you intend to affect the run edition.

For example:

$ . <EBS_ROOT>/EBSapps.env run

$ adstrtal.sh

Set the patch edition environment whenever you intend to execute commands that affect the patch edition.

For example:

$ . <EBS_ROOT>/EBSapps.env patch

$ sqlplus apps/apps @my_custom_patch_script.sql

The adop tool executes non-interactively, executing the specified phase or phases in order. In a multi-node deployment, adop is only executed by the user on the master node: internally, adop will use ssh remote execution to run required actions on all secondary nodes automatically. In addition, adop can be used to generate reports about patching operations in the environment.

adop is typically run as follows:

$ adop phase=<phase_name>

The phase parameter accepts the following phase names. These names can be specified individually, or (except where otherwise noted) with other phase names in a comma separated list:

• prepare - Prepares the environment for patching.

• apply - Applies the specified patches to the environment.

• finalize - Performs any final steps required to make the system ready for cutover.

• cutover - Shuts down application tier services, makes the patch edition the new run edition, and then restarts application tier services. This is the only phase the involves a brief downtime.

cleanup - Removes obsolete code and data from old editions.

• abort - Aborts the online patching cycle by dropping the database patch edition. This phase cannot be specified with any other phase.

• fs_clone - Recreates the patch edition file system as an exact copy of the run edition file system. This phase cannot be specified with any other phase. Use of fs_clone is normally not required. Situations that do require fs_clone are will explicitly document that requirement. If running this phase, ensure that your current working directory is not within the patch edition file system.

Online Patching Cycle Steps - Prepare Phase:

The principles of adop operation in the prepare phase, followed by the steps you take to run this phase are described below:

Note: The exact actions taken during the prepare phase are context dependent: for example, the first time it is ever run on a system; when it is run after an apply phase has been aborted; and when it has been run after cutover.

Principal adop Actions

During the prepare phase, adop performs the following steps.

1. Checks whether to perform a cleanup, which will be needed if the user failed to invoke cleanup after the cutover phase of a previous online patching cycle.

2. Validates system configuration to ensure that the system is ready to start an online patching cycle.

3. Checks to see if the database is prepared for online patching:

• Checks if the database user is edition-enabled. If not, adop immediately exits with an error.

• Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. This service is created automatically, but its continued existence is validated on each prepare.

• Checks to see if logon trigger exists and is enabled. If the logon trigger is missing or the patch service has not been created, adop will automatically try to fix the issue so that it can proceed. If it cannot do so, it will exit with an error message.

• Checks the integrity of the database data dictionary. If any corruption is found, adop will exit with an error. For information on how to resolve data dictionary corruptions, refer to My Oracle Knowledge Document 1531121.1, Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2.

• Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.

4. Checks system configuration on each application tier node. A number of critical settings are validated to ensure that each application tier node is correctly registered, configured, and ready for patching.

5. Checks for the existence of the "Online Patching In Progress" (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists). The flow of control is as follows.

1. If the ADZDPATCH program has not yet been requested to run, a request is submitted.

2. The status of ADZDPATCH is determined. If it is pending, it may be waiting for an incompatible program to finish. At that point, its status will change to running, and it will allow the prepare phase to proceed. A message to this effect is displayed to the user.

The next stage depends on whether the concurrent managers are running:

1. If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.

2. If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.

3. If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues.

Note: ADZDPATCH is cancelled when the cutover phase is complete.

Invokes the TXK script

$AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.

7. Checks the database for the existence of a patch edition, and creates one if it does not find one.

8. Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working. If any of these checks fail, adop will exit with an error.

Optional User Checks

Before you run the prepare phase to start a new patching cycle, you may wish to perform a couple of optional checks.

The first check is to validate your system for patching, by running the command:

$ adop -validate

Note: If you run this command while a patching cycle is in progress, validation will take place for the cutover phase.

If you run this command from the primary (master) node, adop will perform the validation checks on all the available nodes available in the system. In contrast, if you run it from a secondary (slave) node, it will run only on that node.

The second check is to confirm there is adequate free space on your system to support a patching cycle:

• SYSTEM tablespace - minimum of 25 GB free

• APPS_TS_SEED tablespace - minimum of 5 GB free

You can do this by running the $AD_TOP/sql/ADZDSHOWTS.sql script:

Required User Actions

You perform the following steps in the prepare phase.

1. Set the environment by executing (sourcing) the run file system environment file:

$ source <EBS install base>/EBSapps.env run

You can confirm that the environment is properly set by examining the relevant environment variables:

$ echo $FILE_EDITION

run

$ echo $TWO_TASK

dbSID

If you had sourced the incorrect environment file (i.e. from the patch file system), the environment variables would show as:

$ echo $FILE_EDITION

patch

$ echo $TWO_TASK

dbSID_patch

2. Download patches to be applied and place them in the $PATCH_TOP directory of your system. This directory is pre-created by the install in the non-editioned file system (fs_ne) and should not be changed.

Important: On a multi-node system with non-shared file systems, you must copy the patch files to each separate $PATCH_TOP directory, so that the patch files are available from the same location on all nodes.

3. Unzip the patch:

$ unzip p99999999.zip

4. Prepare the system for patching by running the following command to start a new patching cycle:

$ adop phase=prepare

Synchronization Options

In this step, the patch file system application tier is synchronized with the run file system application tier. This can be done by any one of three methods, the first of which is the default and therefore does not need to be specified in the adop command:

• Method 1 - Identify the patches that were applied to the run APPL_TOP and apply them to the patch APPL_TOP. The following steps are performed

automatically:

1. The patches that need to be applied to the patch APPL_TOP are identified from the database.

2. The merged patches are applied by the adop utility. The adop utility identifies the patches to be applied, and applies them silently to the current patch APPL_TOP. As this procedure only requires the application of previously unapplied patches, it requires less time and disk space compared to Method 2 (below).

Using the skipsyncerror parameter

In some circumstances, this synchronization method may fail when applying a series of patches to the patch edition. This can happen if the previous patching cycle included patches that failed to apply correctly, and was followed by subsequent patches that corrected the issue.

The skipsyncerror parameter enables you to specify that you expect any synchronization errors in the prepare phase to be fixed automatically in the

synchronization that takes place with subsequent patches. If the value of the parameter is passed as 'yes', the first patch to be synchronized will be done with the 'autoskip' flag set.

Important: It is your responsibility to check the log files and correct any errors in the subsequent apply phase, or to confirm that synchronization with subsequent patches resolved the issue.

An example of using this parameter would be as follows.

1. You run adop phase=prepare.

2. The phase fails with an error when trying to synchronize the run and patch file systems. That is, the attempt to synchronize a patch fails, but it is known that a subsequent patch will correct the problem.

3. You examine the log files and conclude that the synchronization errors will be fixed automatically in the synchronization that takes place with subsequent patches.

4. You run the command adop phase=prepare skipsyncerror=yes to restart the prepare phase. This time, application of the patch that failed inthe previous prepare will be retried with the 'autoskip' flag set.

Alternatively, if you are not confident that the error will be fixed (for example, you cannot identify the cause from examination of the log files), you should:

1. Run the command adop phase=abort

2. Run the command adop phase=fs_clone

3. Rerun the command adop phase=prepare

Method 2 - Create a new patch file system by cloning the run file system using the fs_clone command.

This method is useful if the APPL_TOPs have become very unsynchronized (meaning that there would be a large number of delta patches to apply). It is a heavyweight process, taking a backup of the entire current patch APPL_TOP and then cloning the run APPL_TOP to create a new patch APPL_TOP. As this method requires more time and disk space, it should only be used when the state of the patch file system is unknown.

The fs_clone command is run as a special adop phase:

$ adop phase=fs_clone

This command must be invoked from the run file system, before the next prepare phase is run.

Note: The patch file system requires at least 25 GB of free disk space to be available for adop operations, including fs_clone. If there is insufficient free space, the adop operation will fail.

If desired, you can change the temporary file system location used by fs_clone, by setting the T2P_JAVA_OPTIONS environment variable to point to a temporary location of your choice:

$ T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/home/t2p/temp"

$ export T2P_JAVA_OPTIONS

Note: You cannot change the fs_clone temporary location by changing the value of the $TMP environment variable.

If an fs_clone operation fails, you can rerun it with the option force=yes to restart it from the beginning (with the same session ID), or force=no to restart it from the point where it failed.

Method 3 - Much faster than the other two methods, this delta synchronization method uses your choice of third-party utility to synchronize the file systems by copying files as applicable from the source directory to the destination directory, optionally ignoring any files and directories you may decide to specify in an exclusion file.

To use this method, specify the parameter/value pair sync_mode=delta on the adop command line:

$ adop phase=prepare sync_mode=delta

Example implementations are provided using rsync on UNIX and RoboCopy on Windows, but you are free to use another utility of your choice.

The delta_sync_drv.txt file includes examples for setting up synchronization using rsync on UNIX or RoboCopy on Windows.

Synchronizing Customizations

The default method of file system synchronization handles official patches but will not synchronize any manually applied customizations.

Additional Information: The synchronization modes and their associated actions are as follows:

• Patch synchronization - apply missing patches

• Delta (file) synchronization - copy file changes

• fs_clone synchronization - clone entire file system

 

 

Examples of patching actions that are not synchronized by default include:

• Compiling user-defined JSPs

• Copying some third-party libraries

• Copying and compiling user-defined concurrent programs

• Copying and generating user-defined forms

To include custom patching actions in the default file system synchronization, you must include the required commands in the Custom Synchronization Driver, $APPL_TOP_NE/ad/custom/adop_sync.drv. You will add your customizations to the following section of the file:

#Begin Customization

...

#End Customization

All the actions defined in this file will be performed by adop automatically during the prepare phase. Be aware that there are two categories of custom command in adop_sync.drv: those that are run one time only, and those that are run at each file system synchronization (during the adop prepare phase).

Important: The adop_sync.drv file is not currently reset to its template file at any point. Consequently, after cutover (and before the next prepare phase), you should review the contents of adop_sync.drv and ensure the requirements for your custom commands continue to be met.

This is only an outline of the steps you need to take to preserve customizations.

Prepare Phase in Multi-Node Environments

In a multi-node environment, one application tier node will be designated as the primary node. This is the node where the Admin Server is located, and will usually also be the node that runs Oracle HTTP Server. All other application tier nodes are designated as secondary nodes.

In a multi-node environment, you must enable ssh from the primary node to all secondary nodes to permit adop remote invocation. In a multi-node environment, adop commands are always run from the primary node only. adop executes required patching actions on the secondary nodes automatically via remote invocation.

If a node unexpectedly becomes inaccessible via ssh, it will be abandoned by adop, and the appropriate further actions taken. Consider a scenario where the adop phase=prepare command is run in a system with ten application tier nodes. The command is successful on nine nodes, but fails on the tenth. In such a case, adop will identify the services enabled on nodes 1-9. If they are sufficient for Oracle EBusiness Suite to continue to run normally, adop will mark node 10 as abandoned and then proceed with its patching actions. If they are not sufficient, adop will proceed no further.

Online Patching Cycle Steps - Apply Phase:

This section describes the principles of adop operation in the apply phase, followed by the steps you take to run this phase.

Principles

In the apply phase, adop applies the specified patches to the system. In an online patching cycle, patches are applied to the patch edition of the database and file system.

Steps

In this phase, you will apply the patches that you designated for inclusion in this patching cycle. You can apply as many patches as you want per patching cycle. By default, a list of patches is applied one at a time, in the specified order. If you specify the merge option "merge=yes", the listed patches will automatically be merged and the resulting merged patch will be applied.

The following example will illustrate the options.

$ adop phase=apply input_file=<inputfile.txt>

This uses the input_file that was mentioned earlier in this section.

An example input_filemight liook like this:

workers=<number of workers>

patches=<patch number 1>:<driver file 1>.drv, <patch number 2>:<driver

file 2>.drv ...

Reports under the $APPL_TOP/admin/<SID>/out directory can help you identify and diagnose problems that may occur in the online patching cycle. These reports list the proposed changes to database objects, both new and modified.

The key files to examine are:

• $APPL_TOP/admin/<SID>/out/adzdcmped.out

• $APPL_TOP/admin/<SID>/log/u<patch_number>.log

Note: For merged patches, the log file name will be derived from the timestamp when merging was performed.

Using the analytics parameter in apply

If you want to use the analytics parameter (see adop Parameters, page 2-2) with the apply phase, enter the command:

$ adop phase=apply analytics=yes

Specifying this option will cause adop to run the following scripts and generate the

associated output files (reports):

• ADZDCMPED.sql - This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_d irectory>/<context_name>/adzdcmped.out.

• ADZDSHOWED.sql - This script is used to display the editions in the system. The output file location is:

/u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_d irectory>/<context_name>adzdshowed.out.

• ADZDSHOWOBJS.sql - This script is used to display the summary of editioned objects per edition. The output file location is:

/u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_d irectory>/<context_name>adzdshowobjs.out

• ADZDSHOWSM.sql - This script is used to display the status report for the seed data manager. The output file location is:

/u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_d irectory>/<context_name>adzdshowsm.out

Note: The analytics parameter should only be used when required, because of the extra processing needed.

Online Patching Cycle Steps - Finalize Phase:

The finalize phase is used to perform any remaining processing that is needed to ensure the system is ready for the fastest possible cutover.

The key actions of the finalize phase are:

1. Pre-compute DDL that needs to be run at cutover.

2. Compile all invalid objects.

3. Validate that the system is ready for cutover.

4. If finalize_mode=full, compute statistics for key data dictionary tables for improved performance.

Run the finalize phase as follows:

$ adop phase=finalize

Online Patching Cycle Steps - Cutover Phase:

This section describes the principles of adop operation in the cutover phase, followed by the manual steps you can optionally execute to run this phase.

Important: No users should remain on the system during cutover, as there will be a short downtime period while the application tier services are restarted. Also, any third-party processes connected to the old run edition of the database should be shut down, or they will be terminated automatically. If desired, you can defer running cutover until a time which will cause minimal disruption to users.

Principles

The key actions performed in the cutover phase are:

1. Shut down internal concurrent manager: The adop utility signals the internal concurrent manager to shut down, but will wait for any existing concurrent requests to finish before it proceeds with cutover actions. The system is still available to users during this waiting period.

If you do not wish to wait indefinitely for concurrent requests to finish, specify the option cm_wait=<maximum_minutes_to_wait> with a number of minutes that reflects your operational needs.

When deciding whether to use this option, Oracle recommends:

• On production systems, do not specify cm_wait, but monitor progress of concurrent tasks and take manual action on them if needed.

• On non-production systems, specify cm_wait to limit the waiting time before cutover proceeds.

2. Shut down application tier services: All application tier services are brought down. During this period, the system is unavailable to users.

3. Cutover database: Promote patch database edition to become the new run database edition, using adzdpmgr.pl script.

4. Cutover file system: Promote patch file system to become the new run file system, switching the $FILE_EDITION values in the patch and run enviroments. The current patch APPL_TOP becomes the new run APPL_TOP, and the current run APPL_TOP becomes the new patch APPL_TOP.

5. Terminate old database sessions: Terminate any database connections to the old run edition of the database.

6. Start application tier services: Application tier services are restarted, on the new run edition. The system is now available again to users.

Note: The adop utility invokes the TXK script txkADOPCutOverPhaseCtrlScript.pl to perform tasks 1, 2, 3, 5, and 6. Task 4 is performed by AutoConfig.

1. Before running the cutover command, ensure you are ready to commit to application of the selected patches. Once cutover is complete, it is not possible to revert to the previous edition.

Note: Cutover will take longer if it has to wait for long-running concurrent processes to complete. In such a case, you can expect to

see an informational message of the form:

[STATEMENT] [END 2013/10/28 23:47:16] Waiting for ICM to go down

If you do not want to wait for in-progress concurrent requests to finish normally, you can terminate the internal concurrent manager by executing the adcmctl.sh abort command from a different shell.

2. In most cases (but see below for the important exception of analytics), you then proceed to execute cutover with the command:

$ adop phase=cutover

This will promote the patch edition to be the new run edition, as well as switching the patch and run labels on the file systems (and thereby, as noted above, changing the patch file system to be the new run file system and the run file system to be the new patch file system).

Important: In the event of problems with the cutover phase, refer to My Oracle Support Knowledge Document 1584097.1, Oracle EBusiness

Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover.

 

 

Deferring Application Tier Restart at Cutover

In some cases, you may need to perform additional manual steps after cutover but before restarting the application tier services. If this is the case, you can supply an additional parameter to the cutover command that causes the application services to remain shut down:

$ adop phase=cutover mtrestart=no

With this parameter, cutover will complete without restarting the application tier services. You can perform any additional steps that require the services to be shut down, and then start the application tier services manually using the adstrtal.sh script.

You must then also run the steps in the following section, Patching the Database Tier:

Patching the Database Tier

These steps are performed post-cutover.

1. On the application tier, as the applmgr user:

1. Change directory to the run file system $APPL_TOP and source your environment file.

2. Run the following command:

$ perl <AD_TOP>/bin/admkappsutil.pl

This will create the appsutil.zip file in <INST_TOP>/admin/out.

2. On the database tier, as the oracle user: Copy or ftp the appsutil.zip file to the RDBMS_ORACLE_HOME, then run the

following commands:

$ cd <RDBMS_ORACLE_HOME>

$ unzip -o appsutil.zip

3. Run AutoConfig on the database tier.

4. Run AutoConfig on the run file system of each application tier node.

5. Start the application tier services.

JAR Files and Cutover

In an online patching cycle, the requisite JAR files are initially stored in the $APPL_TOP/admin/<SID>/out directory, and then uploaded into the database during the cutover phase. Therefore, the out directory must not be deleted at least until cutover is complete.

Online Patching Cycle Steps - Cleanup Phase:

This section describes the principles of adop operation in the cleanup phase, followed by the steps performed in this phase.

Important: If you fail to run the cleanup phase explicitly, it will be run automatically on the next prepare cycle, but this will cause a delay in starting your next online patching cycle.

Principles

Various actions are performed during cleanup, including dropping (removing)obsolete:

• Crossedition triggers

• Seed data

• Editioned code objects (covered objects)

• Indexes

• Columns

• Editions

Steps

1. Cleanup is performed with the command:

$ adop phase=cleanup

The adop parameter cleanup_mode provides control of cleanup processing:

• cleanup_mode=quick - Performs minimum cleanup, including removal of obsolete crossedition triggers and seed data.

• cleanup_mode=standard - Does the same as quick mode, and also drops (removes) obsolete editioned code objects (covered objects). This is the default mode, so does not need to be specified.

• cleanup_mode=full - Performs maximum cleanup, which drops all obsolete code and data from earlier editions

Choosing the Cleanup Mode

Generally, you can use standard mode (the default). You might want to use the other modes in the following circumstances:

• Use quick cleanup when you need to start the next patching cycle as soon as possible. For example, if you want to start a new patching cycle right away, but have not yet run cleanup from the previous patching cycle, you can use quick cleanup mode to complete the essential cleanup tasks as fast as possible.

• Use full cleanup when you want to recover the maximum amount of space in the database. If you have run a large number of patching cycles, or applied a very large patch such as a rollup, significant space may be consumed by obsolete table columns and recovered by running a full cleanup. A full cleanup should only be performed when there is no immediate need to start a new patching cycle.

Note: Prior to AD-TXK Delta 8, if a table is patched and the definition of an existing column changed, the original column is marked as unused on a subsequent full cleanup. From AD-TXK Delta 8, a lower-versioned column is not marked as unused. If an abort is carried out, a higher-versioned column is marked as unused, as such a newly-added column is not in use.

Terminating Sessions Connected to Old Editions

During a full cleanup operation, the adop utility can only drop old editions if those editions are not in use. An SQL script, ADZDKILLOLDSESSIONS.sql, provides a solution to let you terminate sessions that are connected to old editions. If you receive an error notification indicating that an edition could not be dropped because it is in use, you can run this script to terminate any sessions using the edition, and then proceed with the cleanup operation.

Using the analytics Parameter in Cleanup

If you want to use the analytics parameter (see adop Parameters, page 2-2) with the cleanup phase, enter the command:

$ adop phase=cleanup analytics=yes

Specifying this option will cause adop to run the following script and generate the associated output file (report):

• ADZDCLEANUPRP.sql - This script is used to display the display the cleanup status. The output file location is: $NE_BASE/EBSapps/log/adop/<adop_sessionID>/<cleanup_directory>/<context_name>/adzdcleanuprp.out.

Note: The analytics parameter should only be used when required, because of the extra processing needed.

Online Patching Cycle Steps - Abort Phase:

If for some reason either the prepare or apply phase failed or gave problems, you can abort the patching cycle at either of these points by running a special phase with the command:

$ adop phase=abort

In the abort phase, adop does the following:

1. Confirms that there is an in-progress online patching cycle, so the abort call is therefore valid.

2. Checks for the existence of a patch edition and drops one if it exists.

3. Cancels the ADZDPATCH concurrent program, if it is running.

4. Deletes the rows inserted for the pending session ID from the ad_adop_sessions and ad_adop_session_patches tables.

Be aware of the following important points:

• After running abort, a full cleanup must be performed. The cleanup command is: adop phase=cleanup cleanup_mode=full). This will remove any columns that were added by the patch but are no longer needed because of the abort. If they are not removed, they may cause problems in a later patching cycle.

• Alternatively, you can run a combined command to abort the patching cycle and perform a full cleanup:

$ adop phase=abort,cleanup cleanup_mode=full

• If any attempt was made to apply patches to the patch edition, after abort you must run the fs_clone phase (adop phase=fs_clone) to recreate the patch file system.

Manual Post-Patch Installation Tasks

 

Traditionally, some patches have associated post-patch installation tasks, including recompilation of invalid packages, regenerating JAR files, and running AutoConfig. In an online patching environment such as Release 12.2 such tasks will normally be performed automatically in the apply phase. If a post-installation patch step mentions any tasks that need to be performed explicitly, where they are run from depends on the type of patching:

• In a normal online patching cycle, the steps should be executed from the patch file system after the apply phase.

• If the patch is being applied in hotpatch mode or downtime mode, the steps should be executed from the run file system after the apply phase.

Dropping Old Editions With the actualize_all Phase

As each online patching cycle is completed, the database will accumulate an additional old database edition. If the number of these grows too large, system performance will start to be affected. When the number of old database editions reaches 25 or more, you should consider dropping all old database editions by running the adop actualize_all phase and then performing a full cleanup.

Important: This procedure will take a large amount of time (significantly longer than a normal patching cycle), and should only be performed when there is no immediate need to start a new patching cycle. Before starting, you should ensure that the system has the recommended database patches and latest AD-TXK code level installed. To proceed, run the following commands in the order shown:

 

$ adop phase=prepare

$ adop phase=actualize_all

$ adop phase=finalize finalize_mode=full

$ adop phase=cutover

$ adop phase=cleanup cleanup_mode=full

 

You have now completed removal of the old database editions.

Configuration Management and Patching

 

The following guidelines apply to making configuration changes to Oracle E-Business Suite in the context of online patching. They particularly apply to the technology stack and application components that reside in the file system.

Note: For specific instructions on how to patch technology stack components, refer to My Oracle Support Knowledge Document 1355068.1, Oracle E-Business Suite 12.2 Patching Technology Components Guide.

The two basic scenarios are online and offline configuration changes. Each will be considered in turn.

ONLINE:

Online configuration changes are performed within the context of an online patching cycle. This is the recommended strategy. First, you prepare your system by running the adop phase=prepare command. You then make the desired configuration changes to the patch file system. They may

include:

• Oracle WebLogic Server configuration changes

• HTTP Server configuration changes

• File system changes performed by the AD utilities

After making the configuration changes, you must run the command adop phase=cutover to promote them.

You must also run the command adop phase=fs_clone to propagate the configuration changes to the secondary file system.

OFFLINE:

Offline configuration changes are applied directly to the run file system, outside an online patching cycle. You can use the adop -status command to verify that no patching cycle is currently active. After making the desired configuration changes, you must explicitly run the adop phase=fs_clone command to propagate the changes to the patch file system.

Important: This offline scenario will require a period of downtime for users.

Considerations When Re-Applying Patches

If you try to apply a patch that has already been applied, and you do not specify options=forceapply, adop will display an error such as:

[WARNING] Skipping the application of patch 14125999_AR since it has been already applied

[WARNING] Hint: Patches can be applied again by specifying options=forceapply when invoking adop

There are two more scenarios that may occur in this type of situation:

• When a failed patch session is restarted with abandon=no, restart=yes, the patches applied in current adop session will not be applied even if options=forceapply is specified. For example, you run the command adop phase=apply options=forceapply patches=1111, 2222 , and application of patch 1111 is successful but patch 2222 fails. After correcting the problem, you try to rerun adop with the command adop phase=apply options=forceapply patches=1111,2222 abandon=no, restart=yes. In this example, patch 1111 would be skipped as it had successfully been applied, and application of patch 2222 would resume. If you wanted to apply patch 1111 again, you would need to specify abandon=yes, restart=no.

• If you apply multiple patches with merge=yes, and you do not specify options=forceapply, the patches will be applied only if at least one of the patches has not been successfully applied before.

Note: This check will be performed for AD and non-AD patches separately, as adop applies these two categories of patch in different sessions.

Customizing Patch File System Backup Count

A new AutoConfig context variable, s_fs_backup_count, is used to specify the 'Patch File System Backup Count'. This variable denotes the number of backups of the patch file system that are to be preserved by adop. The variable is used during the fs_clone phase, where the existing patch file system is backed up before it is recreated from the run file system. Valid values for the s_fs_backup_count variables are 0-9. A value of 0 (the default) will not preserve any patch file system backups. A value of 1 will preserve the latest patch file system backup, a value of 2 will preserve the latest two backups, and so on. You can normally leave the setting at the default value of 0 (no backup), as the patch edition file system can be fully recreated from the run edition file system by fs_clone.

Monitoring, Reporting and Troubleshooting

 

This section is in three parts, that describe

• Reporting and monitoring

• Analyzing log files

• Common issues

REPORTING AND MONITORING:

ADOPMON:

You can run the Online Patching Monitoring utility (adopmon) to provide a continuously refreshed view of key adop actions, in a coceptually similar way to the tail -f UNIX command. The adopmon utility is useful both in following the overall progress of a patching cycle and identifying the various individual actions that are being taken.

$ adopmon

Running script. Press Ctrl-C to quit.

Enter the APPS password:

ADOPREPORTS:

A related tool, the Online Patching Diagnostic Reports utility, $AD_TOP/bin/adopreports, can be used to help diagnose issues or simply gather

information about the status of your system.

The adopreports utility is invoked by entering the command:

$ adopreports <APPS username> <APPS password>

This displays the adopreports Main Menu:

Online Patching Diagnostic Reports Main Menu

--------------------------------------------

1. Run edition reports

2. Patch edition reports

3. Other generic reports

4. Exit

Choosing option 1 from the Main Menu displays the Run Edition Reports Sub Menu:

Run Edition Reports Sub Menu

----------------------------

1. All

2. Count of uncovered objects per edition

3. List of uncovered objects per edition

4. Cleanup status - summary of covered objects per edition,etc.

5. Show covered objects count per edition.

6. Show list of covered objects per edition.

7. Back to main menu

Choosing option 2 from the Main Menu displays the Patch Edition Reports Sub Menu:

Patch Edition Reports Sub Menu

------------------------------

1. All

2. Patch status - new/changed objects

3. Objects patch in the current edition

4. Table manager status

5. Back to main menu

Choosing option 3 from the Main Menu displays the Other Generic Reports Sub Menu:

Other Generic Reports Sub Menu

------------------------------

1. Editions summary

2. Editioned objects summary

3. Free space in important tablespaces

4. Status of critical AD_ZD objects

5. Actual objects in current edition

6. Objects dependencies

7. Objects dependency tree

8. Editioning views column mappings

9. Index details for a table

10. Inherited objects in the current edition

11. All log messages

12. Materialized view details

13. Database sessions by edition

14. Table details (Synonyms, EV, etc.)

15. Count and status of DDL execution by phase

16. Back to main menu

ANALYZING LOG FILES:

ADOPSCANLOG:

The Online Patching Log Analyzer Utility analyzes adop log directories for errors and warnings, and displays messages to help the user quickly identify any problems that may have occurred during an adop run. It thereby offers an alternative to reviewing log files manually.

The Log Analyzer utility can be run without options, to scan all log directories of the latest adop session for errors:

$ adopscanlog

The utility can also be run with various options, as shown in the following list.

• To scan log directories relating to the latest run of adop in the latest session:

$ adopscanlog -latest=yes

• To scan log directories relating to the latest run of the specified phase, in the latest session:

$ adopscanlog -latest=yes -phase=<phase_name>

• To scan all log directories of a given session (represented by a session_id) for errors:

$ adopscanlog -session_id=<number>

• To scan a specific log directory for errors:

$ adopscanlog ?scan_dir=<full path of directory to scan>

• To see a complete list of supported parameters:

$ adopscanlog –help

COMMON ISSUES:

This section describes known issues and common problems with online patch application.

• Tables need Defragmenting

Sometimes, tables used by online patching will benefit from defragmenting. You can identify tables that need defragmenting by running the following

commands and queries:

-- Gather table statistics

SQL>exec dbms_stats.gather_table_stats ('APPLSYS','AD_ZD_DDL_HANDLER');

-- Query 1: Returns apparent table size

SQL>select table_name,round((blocks*8),2)||'kb' "size" from dbs_tables where table_name = 'AD_ZD_DDL_HANDLER' and owner = 'APPLSYS';

-- Query 2: Returns actual table size

SQL>select table_name,round((num_rows*avg_row_len/1024),2)||'kb' "size" from DBA_tables where table_name = '<table-name>' and owner = 'table-owner>';

If the "size" of the table reported by Query 1 is greater than the "size" reported by Query 2, then the table is fragmented.

To defragment the table, perform the following steps.

1. Run the prepare phase.

2. Run the command:

SQL>ADZDTABRDF.sql <owner> <table> [interim table name]

Note: Supplying the interim table name is only needed if you want to redefine a non-partitioned table to a partitioned table. In such a case, you must create the partitioned table before running this script.

3. Run the cutover phase.

The fragmentation will now have been eliminated.

• Defaults File Becomes Corrupt

The system-generated defaults file is located here:

$APPL_TOP/admin/<SID>/adalldefaults.txt

If this file becomes corrupt, running AutoConfig will automatically instantiate a new copy.

• Cutover Fails

If you need general assistance with recovering from a cutover failure, refer to My Oracle Support Knowledge Document 1584097.1, Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover

If you attempt to resume a failed session after cutover exits with cutover_status=3, you may receive an 'Invalid Credentials' error. This will be because the database patch edition has already been promoted to be the new run edition. To resume and complete cutover successfully, run the command:

$ adop phase=cutover action=nodb

The cutover operation is the most critical phase of an online patching cycle. Although other adop operations can be left to run unattended, you should carefully monitor the progress of cutover, so that you can respond quickly in case of any issues. If cutover fails to complete, check log messages for any problems that may require correction. Then try executing the cutover command again. When cutover is re-executed after a previous failure, adop will restart cutover processing at the failure point for any nodes that did not complete, and the processing may be

successful this time.

• Nodes Are Abandoned

In a multi-node environment, the user executes adop commands only on the primary node. The primary instance of adop then uses ssh remote invocation to execute patching actions on all secondary nodes. At the end of each primary node adop command, a summary report is produced showing the status of the command on each secondary node.

For example:

Format of Summary Report

------------------------

Node ebsint1 : COMPLETED

- Prepare status: COMPLETED

Node ebsint2 : COMPLETED

- Prepare status: COMPLETED

If a secondary node fails to complete a phase, you should investigate the related log messages. You may be able to retry running the phase by invoking adop again, this time from the primary node. When you retry a partially failed command, adop will determine which secondary nodes have not yet completed the phase, and only retry the command on the failed nodes.

If a secondary node fails and the issue cannot be easily resolved, you may be able to continue adop processing by invoking adop from the primary node in the next phase. When this happens, adop will display a confirmation message. 

There are two cases:

Case 1 - If the prepare phase fails on some secondary nodes, and you try to run the apply or cutover phases,

adop will display the following message:

Prepare phase failed on node <node_name>. If you choose to proceed with cutover, node <node_name> will be marked as abandoned. Do you

want adop to continue processing on completed nodes [y/n]?

Case 2 - If application of some patches fails on any secondary node, and you try to run the cutover phase, the following message will be displayed:

Applying patch(es): `Failed patch list> on node <node_name> failed.

If you choose to proceed with cutover, node <node_name> will be marked as abandoned. Do you want adop to continue processing on

completed nodes [y/n]?

If you respond 'y' to either of the above prompts, and all the essential services are available on completed nodes, adop will ignore any failed nodes and continue processing with the remaining available nodes. If an essential service is not available on any of the remaining available nodes, adop cannot continue and will display the following error:

Unable to continue with other available nodes: <comma-separated list of available nodes>

Once the cutover phase completes on the primary node, adop will mark all failed nodes as 'ABANDONED'. A primary node can never be an abandoned node.

Warning: Abandoned nodes must be deleted and recreated before they can be used again. Therefore, do not allow a node to be abandoned unless you are sure this is appropriate.

Abandoned nodes must either be removed from the system, or recloned (copied) from an available node.

If you start a new adop session (prepare, apply, fs_clone) while abandoned nodes are present in the system, adop will display an error:

Node(s) <abandoned_node_list> were abandoned in a previous patching cycle. To recover these node(s), follow the instructions in My Oracle Support Knowledge Document 1677498.1.

For further information, refer to the above-mentioned My Oracle Support Knowledge Document 1677498.1, How to Restore An Abandoned Node in Oracle EBusiness Suite Release 12.2.

• Old Data in adop Depository Tables Needs Removing

Over time, the adop repository tables will accumulate data that may no longer be required. To clear such obsolete data, you can run the following API:

SQL>exec ad_zd_fixer.clear_adop_repo_tables;

If there is no active patching cycle, and you want to clear all data unconditionally, use this command:

SQL>exec ad_zd_fixer.clear_adop_repo_tables(true);

Warning: Before you run this API, be sure that the repository table data is definitely not needed.

***********************************************************************************************

$ adop -examples

Applications DBA Online Patching Tool (adop)

See Oracle E-Business Suite Maintenance Guide for a full description of

adop features, operation, and usage.

***********************************************************************************************

Standard Online Patching Cycle:

    In Oracle E-Business Suite Release 12.2, patches are normally applied

    using an online patching cycle.  The online patching cycle consists of

    five phases which are executed in order.

    Example - Typical online patching cycle:

        source <ebs_root>/EBSapps.env run

        adop phase=prepare

        adop phase=apply patches=123456

        adop phase=finalize

        adop phase=cutover

        source <ebs_root>/EBSapps.env run

        adop phase=cleanup

    Note that after cutover the command line environment should be re-loaded

    as the run edition file system has changed.

    In a multi-node deployment, adop commands are only executed from the

    primary node.  The primary adop session uses remote execution to

    automatically perform required actions on any secondary node.

    Multiple phases can be executed in a single adop command.

    Example - combined finalize/cutover/cleanup:

        adop phase=finalize,cutover,cleanup

    Prior to cutover, it is possible to execute additional

    "apply" and "finalize" phases as needed.

    Example - applying multiple patches using separate apply commands:

        source <ebs_root>/EBSapps.env run

        adop phase=prepare

        adop phase=apply patches=123456

        adop phase=apply patches=223456

        adop phase=finalize

        adop phase=apply patches=323456

        adop phase=finalize

        adop phase=cutover

        source <ebs_root>/EBSapps.env run

        adop phase=cleanup

    Note that it is possible to apply additional patches after running the

    finalize phase, but if you do so then you will need to run the finalize

    phase again.  Finalize must always be run immediately prior to cutover.

***********************************************************************************************

General parameters applicable to all phases:

    workers=<number>  [default: computed]

        Number of parallel workers used to execute tasks.  Default value is

        computed principally according to number of available CPU cores.

        Example:

            adop phase=prepare workers=8

        The example command executes the prepare phase with 8 parallel workers.

        Not all processing can be executed with parallel workers, and in such

        cases the workers parameter is ignored.  In multi-node systems the

        workers parameter indicates workers per node.

***********************************************************************************************

    input_file=<file_name>

        As well as being entered directly on the command line, adop parameters

        can be specified in a text file, with one

            <parameter>=<value>

        on each line of the file.  Command line parameters override input file

        parameters.

        Example - Input File text "empty_cycle.txt" for empty patching cycle:

            phase=prepare,finalize,cutover,cleanup

            workers=4

            loglevel=statement

        Example - Use input file to execute empty patching cycle:

            adop input_file=empty_cycle.txt workers=1

        The example command will execute an empty patching cycle with a

        single worker and maximum log detail.

***********************************************************************************************

    loglevel=(statement|procedure|event|warning|error|unexpected)

        [default: event]

        Controls the level of diagnostic log detail displayed on the

        console output.  Each log message is tagged with a level:

            1) statement  - internal debugging messages

            2) procedure  - internal procedure begin/end messages

            3) event      - significant processing action

            4) warning    - important user message or possible problem

            5) error      - a patching action has failed, processing continues

            6) unexpected - a critical action has failed, processing halted

        Setting loglevel will display messages at that level and higher.

        Example - Display full log details:

            adop phase=prepare loglevel=statement

        Example - Display only warnings and errors:

            adop phase=prepare loglevel=warning

***********************************************************************************************

    prompt=(yes|no)  [default: yes]

        Specifies whether adop should prompt for user input on warnings.

        By default adop will ask user whether to continue or exit on

        some warning messages.  If this parameter is set to "no" adop

        will remain fully non-interactive, and will continue past any

        warning messages without user confirmation.

        Example - Disable user prompt for warnings:

            adop phase=cutover prompt=no

***********************************************************************************************

Diagnostic Parameters, normally not used unless directed by Support:

    allowcoredump=(yes|no)  [default: no]

        Specifies whether adop should create a core dump if it crashes.  This

        option should only be used if directed by support.

        Example - Enable writing of core dump files:

            adop phase=cutover allowcoredump=yes

***********************************************************************************************

    analytics=(yes|no)  [default: no]

        Controls whether adop writes additional reports with information

        that might be helpful in some diagnostic situations.  This option

        should not be used unless directed by Support.

        Example - Enable additional analytic reports

            adop phase=finalize analytics=yes

***********************************************************************************************

    defaultsfile=<file_name>  [default: adalldefaults.txt]

        Name of the response file providing default parameter values for

        non-interactive execution of adadmin and adop.  The file must be in

        the $APPL_TOP/admin/$TWO_TASK directory in both run edition and patch

        edition file systems.  The default file "adalldefaults.txt" is

        maintained by AutoConfig and normally you should not need to change

        any values.

        Example - Using a custom AD Defaults file:

            adop phase=apply defaultsfile=adcustomdefaults.txt

***********************************************************************************************

###################    Prepare Parameters:   ##################

    skipsyncerror=(yes|no)  [default: no]

        Specifies whether to ignore errors that may occur during incremental

        file system synchronization.  This might happen if you applied

        a patch in the previous patching cycle that had errors but decided

        to continue with the cutover.  When the patch is synchronized on

        the next patching cycle, the apply errors may occur again, but

        can be ignored.

    sync_mode=(delta|patch) [default: patch]

        Specifies the method used to synchronize the patch file system

        with the run file system.

        'delta' mode uses the file system synchronization command

        specified in $AD_TOP/patch/115/etc/delta_sync_drv.txt.

        'patch' mode reapplies the patches that were applied on

        the run file system.

        Example:

            adop phase=prepare skipsyncerror=yes sync_mode=delta

***********************************************************************************************

####################  Apply Parameters:   ##################

    apply=(yes|no)  [default: yes]

        Controls whether adop actually applies the patch.  You can specify

        "apply=no" to run adop in test mode, where the patch will not actually

        be applied, and adop will record what it would have done in the log.

        Example - Execute adop apply in test mode:

            adop phase=apply apply=no patches=1234

***********************************************************************************************

    patches=<patch#>[,<patch#>...]

    patches=<patch_directory>:<driver>[,<patch_directory>:<driver>...]

        This parameter specifies a comma-separated list of patches to be

        applied.  Patches can be specified either as the patch number or

        by the patch directory and driver file.  All patches are expected

        to be in the $PATCH_TOP directory on all tiers.  Patches are applied

        serially unless the merge=yes parameter is specified.

        Example - Apply two patches in series:

            adop phase=apply patches=1234,5678

        Example - Apply patch and its translation:

            adop phase=apply patches=1012464,10124646_AR:u10124646.drv

***********************************************************************************************

    patchtop=<directory_name>  [default: $PATCH_TOP]

        Path to a user-specified directory where patches are unzipped.

        The default and recommend location is the $PATCH_TOP directory

        automatically created by the install.  When using an alternate

        patchtop you must ensure that the location is not within the

        editioned file systems (fs1, fs2) and is accessible by the same

        path for all nodes of a multi-node deployment.

        Example - Apply patch from alternate patchtop directory:

            adop phase=apply patches=123456 patchtop=/install/patches

***********************************************************************************************

    apply_mode=(online|downtime|hotpatch)  [default: online]

        Use online mode to apply a patch to the patch edition during an

        online patching cycle; downtime mode to apply a patch to the

        run edition when application services are down; and hotpatch

        mode to apply a patch to the run edition when application

        services are up. Only use hotpatch mode when explicitly

        directed by documentation.

        Example - apply a patch in downtime mode:

            adop phase=apply patches=123456 apply_mode=downtime

        In downtime mode, adop will validate that application services

        are shutdown before apply the patch.  The patch will be applied

        to the run edition of the system.  Downtime mode cannot be used

        if there is an online patching cycle in progress.

        Example - apply a patch in hotpatch mode:

            adop phase=apply patches=123456 apply_mode=hotpatch

        In hotpatch mode, adop will apply the patch to the run edition

        of the system while application services are still running.  Patches

        that can be safely applied in hotpatch mode (such as NLS and Online

        Help patches) will document this in the patch readme.  Hotpatch mode

        cannot be used if there is an online patching cycle in progress.

***********************************************************************************************

    merge=(yes|no)  [default: no]

        Indicates whether adop should merge a list of patches before applying.

        By default, adop will apply a list of patches serially in the order

        specified.  You can also use AD Merge Patch to merge multiple patches

        ahead of the apply command.

        Example - Merge multiple patches before applying

            adop phase=apply patches=1234,5678 merge=yes

        In some cases it may be necessary to merge patches, such as when

        an additional patch corrects an installation issue in the main patch.

        If the merge parameter is required, this will be documented in the

        patch readme.

***********************************************************************************************

    restart=(yes|no)  [default: no]

        Use restart=yes to resume the previous failed apply command

        from where processing terminated.  If an apply command fails,

        check the log files for further information.  If the problem

        can be corrected, you can then restart the apply command where

        it left off using the restart parameter.

        Example - Restart a apply command

            adop phase=apply patches=123456 restart=yes

        When restarting a failed apply it is important to use the

        same parameters as the failed command, with only the addition

        of the restart=yes parameter.

***********************************************************************************************

    abandon=(yes|no)  [default: no]

        Use abandon=yes to abandon the previous failed apply command

        and start a new apply command.  Note that any changes made

        to the system by the failed command will remain in effect.

        The abandon flag is most useful when applying a replacement

        patch for the failing patch.

        Example - Abandon previous apply command and apply replacement

            adop phase=apply patches=223456 abandon=yes

        If a patch fails to apply and there is no replacement patch,

        you may also abort the online patching cycle.  See abort phase

        later in this text.

***********************************************************************************************

    options=<patch_option>[,<patch_option>...]

        Options can be specified in a comma-separated list to control

        advanced features when a patch is applied.  These options are normally

        not needed unless specified by documentation or support.  Note

        that these options can be prefixed with "no", e.g. "nocheckfile",

        to disable the behavior, and for some options "no" is the default.

            checkfile       - Skip running exec, SQL, and exectier

                              commands if they are recorded as already run.

                              [default: checkfile]

            compiledb       - Compile invalid objects in the database

                              after running actions in the database driver.

                              [default: compiledb]

            compilejsp      - Compile out-of-date JSP files, if the patch

                              has copy actions for at least one JSP file.

                              [default: compilejsp]

            copyportion     - Run commands found in a copy driver.

                              [default: copyportion]

            databaseportion - Run commands found in a database driver.

                              [default: databaseportion]

            generateportion - Run commands found in a generate driver.

                              [default: generateportion]

            integrity       - Perform patch integrity checking

                              [default: nointegrity]

            autoconfig      - Run AutoConfig.

                              [default: autoconfig]

            actiondetails   - Turn off display of action details.

                              [default: actiondetails]

            parallel        - Run actions that update the database or

                              actions that generate files in parallel.

                              [default: parallel]

            prereq          - Perform prerequisite patch checking prior to

                              running patch driver files.

                              [default: noprereq]

            validate        - Connect to all registered Oracle E-Business

                              Suite schemas at the start of patch application.

                              [default: novalidate]

            phtofile        - Save patch history to file

                              [default: nophtofile]

            forceapply      - Reapply a patch that has already been applied.

                              Useful in combination with "nocheckfile" option

                              to rerun files that have already been executed.

                              [default: noforceapply]

        Example - Apply patch again, force re-execution of driver commands:

            adop phase=apply patches=123456 options=forceapply,nocheckfile

        This command will apply the patch again, and will execute SQL, exec

        and exectier actions even if they have already been executed.

***********************************************************************************************

    flags=<patch_flag>[,<patch_flag>...]

        Flags can be specified in a comma-separated list to control advanced

        features when applying a patch.  Note that these flags can be prefixed

        with "no", e.g.  "nologging", to disable the behavior and for some

        flags "no" is the default.

            hidepw          - Omit the "HIDEPW:" comments in the log file.

                              [default: hidepw]

            trace           - Log all database operations to a trace file.

                              [default: notrace]

            logging         - Create indexes in LOGGING or NOLOGGING mode.

                              [default: nologging]

            autoskip        - To proceed with adpatch execution even if some

                              driver actions failed.  Failed actions are

                              recorded in a log file.

                              [default: noautoskip]

        Example - Skip apply failures:

            adop phase=apply patches=123456 flags=autoskip

***********************************************************************************************

    preinstall=(yes|no)  [default: no]

        Allows a patch to be applied to the file system without connecting

        to the database.  Do not use this parameter unless directed by Oracle.

***********************************************************************************************

    wait_on_failed_job=(yes|no)  [default: no]

        Controls whether adop apply command exits when all workers have

        failed.  Instead of exiting, you can force adop to wait, and use

        the "adctrl" to retry failed jobs.

***********************************************************************************************

    printdebug=(yes|no)  [default: no]

        Controls whether to display additional debugging information.

***********************************************************************************************

    uploadph=(yes|no)  [default: yes]

        Controls whether to upload patch history information to database after

        applying the patch.

***********************************************************************************************

#####################   Finalize Parameters:   ######################

    finalize_mode=(full|quick)  [default: quick]

        Quick mode will provide the shortest execution time, by

        skipping non-essential actions.

        Full mode performs additional actions such as gathering

        statistics that may improve performance after cutover.

        Example - Perform additional optimizations during Finalize:

            adop phase=finalize finalize_mode=full

        Using full mode after applying a major patch can help maintain

        optimal system performance.

***********************************************************************************************

####################   Cutover Parameters:   ###################

    mtrestart=(yes|no)  [default: yes]

        Specifies whether to restart application tier servers after cutover.

        Leave at default unless you need to perform any manual steps during

        downtime.

        Example - Perform downtime actions after cutover:

            adop phase=cutover mtrestart=no

            <Perform downtime steps>

            adstrtal.sh

***********************************************************************************************

    cm_wait=<minutes>  [default: forever]

        Specifies the number of minutes to wait for Concurrent Manager

        shutdown.  Adop cutover starts by requesting a concurrent manager

        shutdown and then waits for in-progress requests to complete.

        If Concurrent Manager does not shutdown within the specified

        time limit, remaining concurrent requests will be killed and

        cutover will proceed.

        Example - cutover with maximum wait of 15 minutes:

            adop phase=cutover cm_wait=15

        Note that any concurrent requests killed during forced shutdown

        may need to be manually re-submitted after cutover.  To avoid

        killing concurrent requests, schedule cutover at a time of

        minimal user activity or manually shutdown Concurrent Manager

        in advance of cutover.

***********************************************************************************************

##################    Cleanup Parameters:   ###################

    cleanup_mode=(full|standard|quick)  [default: standard]

        Quick mode provides the shortest execution time, by

        skipping non-essential actions.

        Standard mode performs additional processing to drop

        obsolete code objects from old editions.

        Full mode performs additional processing to drop empty

        database editions and unused table columns.

        Example - Run quick cleanup for minimum delay to next prepare:

            adop phase=cleanup cleanup_mode=quick

        Example - Run full cleanup for maximum space recovery:

            adop phase=cleanup cleanup_mode=full

***********************************************************************************************

##############    Special Operations:   #################

    Cloning the Patch Edition File System:

        The patch edition file system is normally synchronized with the

        run edition file system during the prepare phase.  There are some

        cases where it is helpful or required to manually re-clone the patch

        edition file system from the run edition.

            1) After aborting an online patching cycle.

            2) After manually changing the run edition file system.

            3) After patching middle-tier technology components.

            4) After applying an EBS RUP.

        By re-cloning the patch edition file system, you can be certain that

        it is correctly synchronized, and also minimize any synchronization

        delay that would normally occur on the next prepare command.

        Example - Re-clone the patch edition file system:

            adop phase=fs_clone

        If there is any error you must examine log files and correct the

        problem, then restart the fs_clone by running the command again.

        If fs_clone does not restart correctly you can force the process

        to restart from the beginning.

        Example - Restart failed fs_clone from the beginning:

            adop phase=fs_clone force=yes

***********************************************************************************************

    Aborting an online patching cycle:

        If an online patching cycle encounters problems that cannot be

        fixed immediately you can abort the patching cycle and return

        to normal runtime operation.

        Example - Aborting a failed online patching cycle:

            adop phase=prepare

            adop phase=apply patches=123456

            ### serious unfixable error reported

            adop phase=abort

            adop phase=cleanup cleanup_mode=full

            adop phase=fs_clone

        The abort command drops the database patch edition and

        returns the system to normal runtime state.  Immediately

        following abort, you must also run a full cleanup and

        fs_clone operation to fully remove effects of the failed

        online patching cycle.

***********************************************************************************************

    Dropping old database editions:

        As online patching cycles are completed, the system will build

        up a number of old database editions.  When the number of old

        database editions reaches about 25, you should consider running

        a special maintenance operation to drop old database editions.

        Example - Actualize_All and Full Cleanup:

            adop phase=prepare

            adop phase=actualize_all

            adop phase=finalize

            adop phase=cutover

            adop phase=cleanup cleanup_mode=full

        Warning: This maintenance operation will take much longer than a

        typical online patching cycle, and should only be performed when

        there is no immediate need to start a new online patching cycle.

        The actualize all and full cleanup can be done separately as shown

        above, or can be executed in conjunction with an online patching

        cycle.

        Example - Actualize_All and Full Cleanup combined with patching:

            adop phase=prepare

            adop phase=apply patches=123456

            adop phase=actualize_all

            adop phase=finalize

            adop phase=cutover

            adop phase=cleanup cleanup_mode=full

adop exiting with status = 0 (Success)

***********************************************************************************************

$ adop -help

Applications DBA Online Patching Tool (adop)

Usage: adop [phase=<phase,phase,...>] [patches=<patch#,patch#,...>]
            [<parameter>=<value> ...] [input_file=<filename>]
       adop -help
       adop -examples
       adop -status [<session_id>]
       adop -validate

See Oracle E-Business Suite Maintenance Guide for a full description of
adop features, operation, and usage.
The phase parameter specifies the parts (phases) of the online
patching cycle to be executed. The five standard phases are executed
in the order shown below.

Standard phases:

  prepare       - Prepare the instance for patch application.
  apply         - Apply patches (to the patch edition).
  finalize      - Ready the instance for cutover.
  cutover       - Make the patch edition the new run edition.
  cleanup       - Drop obsolete objects and data from old editions.

  There are also three special phases, for use when needed.

Special phases:

  abort             - Abort the current online patching cycle.
  actualize_all - Create actual copies of all code objects in the patch edition.
  fs_clone       - Copy the run file system to the patch file system.

General parameters applicable to all phases:

    workers=<number> [default: computed]
        Number of parallel workers used to execute tasks.  Default value is
        computed principally according to number of available CPU cores.
    input_file=<file_name>
        As well as being entered directly on the command line, adop parameters
        can be specified in a text file, with one
            <parameter>=<value>
        on each line of the file.  Command line parameters override input file
        parameters.
    loglevel=(statement|procedure|event|warning|error|unexpected)
        [default: event]
        Controls the level of diagnostic log detail displayed on the console.
    prompt=(yes|no)  [default: yes]
        Specifies whether adop should prompt for user input on warnings.
        By default adop will ask user whether to continue or exit on
        some warning messages.  If this parameter is set to "no" adop
        will remain fully non-interactive, and will continue past any
        warning messages without user confirmation.

Phase-specific parameters control operation of a particular phase:

  Prepare Parameters:

    skipsyncerror=(yes|no)  [default: no]
        Specifies whether to ignore errors that may occur during incremental
        file system synchronization.  This might happen if you applied
        a patch in the previous patching cycle that had errors but decided
        to continue with the cutover.  When the patch is synchronized on
        the next patching cycle, the apply errors may occur again, but
        can be ignored.
    sync_mode=(delta|patch) [default: patch]
        Specifies the method used to synchronize the patch file system
        with the run file system.
        'delta' mode uses the file system synchronization command specified
        in $AD_TOP/patch/115/etc/delta_sync_drv.txt.
        'patch' mode reapplies the patches that were applied on
        the run file system.
  

Apply parameters:

    patches=<patch#>[,<patch#>...]
    patches=<patch_directory>:<driver>[,<patch_directory>:<driver>...]
        This parameter specifies a comma-separated list of patches to be
        applied.  Patches can be specified either as the patch number or
        by the patch directory and driver file.  All patches are expected
        to be in the $PATCH_TOP directory on all tiers.  Patches are applied
        serially unless the merge=yes parameter is specified.
    restart=(yes|no)  [default: no]
        Use restart=yes to resume the previous failed apply command
        from where processing terminated.
    abandon=(yes|no)  [default: no]
        Use abandon=yes to abandon the previous failed apply command
        and start a new apply command.
    apply_mode=(online|downtime|hotpatch)  [default: online]
        Use online mode to apply a patch to the patch edition during an
        online patching cycle; downtime mode to apply a patch to the
        run edition when application services are down; and hotpatch
        mode to apply a patch to the run edition when application
        services are up. Only use hotpatch mode when explicitly
        directed by documentation.
    apply=(yes|no)  [default: yes]
        To run adop in test mode (without applying any patches), use apply=no
    options=<patch_option>[,<patch_option>...]
        Advanced options to control patching behavior.
        See adop -examples for details.
    flags=<patch_flag>[,<patch_flag>...]
        Advanced flags to control patching behavior.
        See adop -examples for details.

  Finalize parameters:

    finalize_mode=(full|quick)  [default: quick]
        Quick mode will provide the shortest execution time, by
        skipping non-essential actions.
        Full mode performs additional actions such as gathering
        statistics that may improve performance after cutover.
  

Cutover parameters:

    mtrestart=(yes|no)  [default: yes].
        Specifies whether to restart application tier servers after cutover.
        Leave at default unless you need to perform manual steps during the
        cutover downtime.
    cm_wait=<minutes>  [default: forever]
        Specifies the number of minutes to wait for Concurrent Manager
        shutdown.  Adop cutover starts by requesting a concurrent manager
        shutdown and then waits for in-progress requests to complete.
        If Concurrent Manager does not shutdown within the specified
        time limit, remaining concurrent requests will be killed and
        cutover will proceed.

  Cleanup parameters:

    cleanup_mode=(full|standard|quick)  [default: standard]
        Quick mode provides the shortest execution time, by
        skipping non-essential actions.
        Standard mode performs additional processing to drop
        obsolete code objects from old editions.
        Full mode performs additional processing to drop empty
        database editions and unused table columns.
  

Fs_clone parameters:

    force=yes/no [default: no]
        Use force=yes to restart a previous failed fs_clone command from
        the beginning.  By default fs_clone will restart where it left off.

Command flags:

    -status [<session_id>]
        Display status of the latest adop session, or a specified session.
    -validate
        Runs ADOP validations for verifying the current system state.
    -examples
        Display extended help information with common usage examples.
    -help
        This help screen.

adop exiting with status = 0 (Success)

REFERENCES:

https://docs.oracle.com/cd/E26401_01/doc.122/e22949/toc.htm

Oracle E-Business Suite 12.2 Patching Technology Components Guide (Doc ID 1355068.1)

Developing and Deploying Customizations in Oracle E-Business Suite Release 12.2 (Doc ID 1577661.1)

E-Business Suite - ADOP Basic Usage Training Videos [VIDEO] (Doc ID 2103131.1)

Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching (adop) Cutover (Doc ID 1584097.1)

Oracle E-Business Suite Concepts, Part No. E22949

Oracle E-Business Suite Installation Guide: Using Rapid Install, Part No. E22950

Oracle E-Business Suite Setup Guide, Part No. E22953

Oracle E-Business Suite Maintenance Guide, Part No. E22954

Oracle E-Business Suite Developer's Guide, Part No. E22961

***********************************************************************************************

How To Check If a Certain Patch Was Applied to Oracle E-Business Suite Instance (11i or R12) ? (Doc ID 443761.1)

***********************************************************************************************

In releases prior to 12.2, we used to query ad_bugs or ad_applied_patches to find out if a patch is applied or not.

In 12.2, this is no longer the right way to do it due to the implications of online patching.

For example, if you apply a patch in an online patching cycle and aborted the cycle before it is complete, the records inserted in the above two tables will not be updated or removed.

The following query helps in identifying the patches applied:

SELECT adop_session_id, bug_number, session_type, 

       DECODE(status,'N','Applied on other nodes',

                     'R','Running',

                     'H','Failed (Hard)',

                     'F','Failed (Jobs Skipped)',

                     'S','Success (Jobs Skipped)',

                     'Y','Success',

                     'C','Clone Complete') status,

       applied_file_system_base, patch_file_system_base,

       node_name, start_date, end_date, 

       ROUND((end_date - start_date) * 24*60,2) exec_time,

       adpatch_options, autoconfig_status, driver_file_name

FROM ad_adop_session_patches

WHERE session_type IN ('ADPATCH','HOTPATCH','DOWNTIME','ONLINE')

ORDER BY adop_session_id, start_date, end_date;

Session type column was restricted to the following values:

ADPATCH: Patches applied using the traditional adpatch tool. It is used right after installing 12.2 and before online patching is enabled by applying patch 13543062.

HOTPATCH: Patches applied in hotpatch mode (aka. no online patching cycle is involved). It should be used only when instructed by Oracle.

DOWNTIME: Patches applied when all application services are down. Only used when instructed by Oracle, such as when applying 12.2.6 RUP.

ONLINE: Patches applied in a normal online patching cycle. This is the recommended method.

 

If you want to find out which patch fixed a specific bug, you can use the following query:

SELECT b.bug_number, asp.adop_session_id, asp.bug_number patch#,

       asp.session_type, asp.applied_file_system_base,

       asp.node_name, asp.start_date, asp.end_date

FROM ad_bugs b, ad_patch_run_bugs prb, ad_patch_runs pr,

     ad_patch_drivers pd, ad_adop_session_patches asp

WHERE b.bug_id = prb.bug_id

  AND prb.patch_run_id = pr.patch_run_id

  AND pr.patch_driver_id = pd.patch_driver_id

  AND pr.patch_run_id = asp.patchrun_id

  AND prb.applied_flag = 'Y'

  AND b.bug_number = '&bug_num';