01 Patch GI and DB with PSU 12.1.0.2.5


Overview

Now that the base software for GI and RDBMS has been loaded, PSU 12.1.0.2.5 needs to be applied.

PSU 12.1.0.2.5 came out in October 2015.

References

Prepare for PSU

Download PSU

Download the PSU from COMBO OF OJVM COMPONENT 12.1.0.2.5 DB PSU + GI PSU 12.1.0.2.5 (OCT2015) .

Check OPATCH Version

Used the following command to get the version of opatch :

/opt/app/oracle/product/12.1.0/grid/OPatch/opatch version

The output was:

OPatch Version: 12.1.0.1.3

OPatch succeeded.

Since the version is less than 12.1.0.1.7, I need to get the latest OPatch from 6880880 .

Update OPatch

The latest OPatch is downloaded from 6880880 as /tmp/p6880880_121010_Linux-x86-64.zip .

The Readme says:

The new opatch utility should be updated in all the Oracle RAC database homes and the GI home that are being patched.

OPatch was updated as follows:

unzip -o /tmp/p6880880_121010_Linux-x86-64.zip -d /opt/app/oracle/product/12.1.0/grid
unzip -o /tmp/p6880880_121010_Linux-x86-64.zip -d /opt/app/oracle/product/12.1.0/dbhome_1

The OPatch version is now correct:

OPatch Version: 12.1.0.1.10

OPatch succeeded.

Create OCM Response File

The Readme says:

The OPatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. You should enter a complete path of OCM response file if you already have created this in your environment. OCM response file is required and is not optional.

Emphasis Mine

Following the instructions in How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1) , I ran the following commands for Option #1:

export ORACLE_HOME=/opt/app/oracle/product/12.1.0/grid
${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -no_banner -output /opt/app/oracle/software/ocm.rsp

The output was:

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/opt/app/oracle/software/ocm.rsp) was successfully created.

Download Patch

Downloaded patch from here into /tmp/p21523260_121020_Linux-x86-64.zip .

Unzipped the patch as follows:

cd /tmp
unzip -q p21523260_121020_Linux-x86-64.zip -d .

Patch Grid Infrastructure

GI Patch Conflict Detection

Ran the following command to detect any patch conflicts:

sudo /opt/app/oracle/product/12.1.0/grid/OPatch/opatchauto apply /tmp/21523260 -analyze -ocmrf /opt/app/oracle/software/ocm.rsp

The result was:

OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /opt/app/oracle/product/12.1.0/grid

opatchauto log file: /opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-28_13-30-19_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.


OCM RSP file has been ignored in analyze mode. 

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /tmp/21523260
Grid Infrastructure Patch(es): 21359755 21359758 21359761 21436941 
DB Patch(es): 21359755 21359758 21555660 

Patch Validation: Successful
Grid Infrastructure home:
/opt/app/oracle/product/12.1.0/grid


Analyzing patch(es) on "/opt/app/oracle/product/12.1.0/grid" ...
Patch "/tmp/21523260/21359755" successfully analyzed on "/opt/app/oracle/product/12.1.0/grid" for apply.
Patch "/tmp/21523260/21359758" successfully analyzed on "/opt/app/oracle/product/12.1.0/grid" for apply.
Patch "/tmp/21523260/21359761" successfully analyzed on "/opt/app/oracle/product/12.1.0/grid" for apply.
Patch "/tmp/21523260/21436941" successfully analyzed on "/opt/app/oracle/product/12.1.0/grid" for apply.

Apply Summary:
Following patch(es) are successfully analyzed:
GI Home: /opt/app/oracle/product/12.1.0/grid: 21359755,21359758,21359761,21436941

opatchauto succeeded.

Apply GI Patch

Run the following commands to apply the GI patch:

sudo /opt/app/oracle/product/12.1.0/grid/OPatch/opatchauto apply /tmp/21523260 -ocmrf /opt/app/oracle/software/ocm.rsp

The output was:

OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /opt/app/oracle/product/12.1.0/grid

opatchauto log file: /opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-28_13-36-23_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /tmp/21523260
Grid Infrastructure Patch(es): 21359755 21359758 21359761 21436941 
DB Patch(es): 21359755 21359758 21555660 

Patch Validation: Successful
Grid Infrastructure home:
/opt/app/oracle/product/12.1.0/grid


Performing prepatch operations on SIHA Home... Successful

Applying patch(es) to "/opt/app/oracle/product/12.1.0/grid" ...
Patch "/tmp/21523260/21359755" successfully applied to "/opt/app/oracle/product/12.1.0/grid".
Patch "/tmp/21523260/21359758" successfully applied to "/opt/app/oracle/product/12.1.0/grid".
Patch "/tmp/21523260/21359761" successfully applied to "/opt/app/oracle/product/12.1.0/grid".
Patch "/tmp/21523260/21436941" successfully applied to "/opt/app/oracle/product/12.1.0/grid".

Performing postpatch operations on SIHA Home... Successful

Apply Summary:
Following patch(es) are successfully installed:
GI Home: /opt/app/oracle/product/12.1.0/grid: 21359755,21359758,21359761,21436941

opatchauto succeeded.

Patch Singleton DB Home

Set Up Patching Environment for Singleton DB Home

Since the RDBMS home is not a RAC home, the PSU has to be applied separately. The following commands are used to set up the environment:

sudo su --login
export ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_1
export PATH=${ORACLE_HOME}/OPatch:${PATH}

Validate PSU for Singleton DB Home

Make sure the patch is valid:

opatchauto apply /tmp/21523260 -oh ${ORACLE_HOME} -ocmrf /opt/app/oracle/software/ocm.rsp -analyze

The output was:

OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /opt/app/oracle/product/12.1.0/dbhome_1

opatchauto log file: /opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-28_19-56-16_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.


OCM RSP file has been ignored in analyze mode. 

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /tmp/21523260
Grid Infrastructure Patch(es): 21359755 21359758 21359761 21436941 
DB Patch(es): 21359755 21359758 21555660 

Patch Validation: Successful
User specified the following DB home(s) for this session:
/opt/app/oracle/product/12.1.0/dbhome_1


Analyzing patch(es) on "/opt/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/tmp/21523260/21359755" successfully analyzed on "/opt/app/oracle/product/12.1.0/dbhome_1" for apply.
Patch "/tmp/21523260/21359758" successfully analyzed on "/opt/app/oracle/product/12.1.0/dbhome_1" for apply.
Patch "/tmp/21523260/21555660" successfully analyzed on "/opt/app/oracle/product/12.1.0/dbhome_1" for apply.

[WARNING] The local database(s) on "/opt/app/oracle/product/12.1.0/dbhome_1" is not running. SQL changes, if any, cannot be analyzed.

Apply Summary:
Following patch(es) are successfully analyzed:
DB Home: /opt/app/oracle/product/12.1.0/dbhome_1: 21359755,21359758,21555660

opatchauto succeeded.

Attempt to Apply PSU to Singleton DB Home

Apply the patch as follows:

sudo su --login
export ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_1
export PATH=${ORACLE_HOME}/OPatch:${PATH}
opatchauto apply /tmp/21523260 -oh ${ORACLE_HOME} -ocmrf /opt/app/oracle/software/ocm.rsp

The output is:

OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /opt/app/oracle/product/12.1.0/dbhome_1

opatchauto log file: /opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-28_20-01-07_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /tmp/21523260
Grid Infrastructure Patch(es): 21359755 21359758 21359761 21436941 
DB Patch(es): 21359755 21359758 21555660 

Patch Validation: Successful
User specified the following DB home(s) for this session:
/opt/app/oracle/product/12.1.0/dbhome_1


Performing prepatch operations on RAC Home (/opt/app/oracle/product/12.1.0/dbhome_1) ... Successful

Applying patch(es) to "/opt/app/oracle/product/12.1.0/dbhome_1" ...
Command "/opt/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch napply -phBaseFile /tmp/OraDB12Home1_oracle_patchList -local  -invPtrLoc /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc -oh /opt/app/oracle/product/12.1.0/dbhome_1 -silent -ocmrf /opt/app/oracle/software/ocm.rsp" execution failed: 
OPatch failed to restore OH '/opt/app/oracle/product/12.1.0/dbhome_1'. Consult OPatch document to restore the home manually before proceeding.
UtilSession failed: Re-link fails on target "javavm_refresh ioracle".

Log file Location for the failed command: /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-28_20-01-34PM_1.log

For more details, please refer to the log file "/opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-28_20-01-07_deploy.debug.log".

Apply Summary:
Following patch(es) are successfully installed:
DB Home: /opt/app/oracle/product/12.1.0/dbhome_1:

Following patch(es) failed to be installed:
DB Home: /opt/app/oracle/product/12.1.0/dbhome_1: 21359755,21359758,21555660
Command failure exception

opatchauto failed with error code 2.

Encountered Bug in OPATCHAUTO

It would appear that I had hit the bug described in MOS Doc ID 2002334.1 .

The solution is to set up the environment as follows, and reapply the PSU:

sudo su --login
export ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_1
export PATH=${ORACLE_HOME}/OPatch:${PATH}
export PERL5LIB=${ORACLE_HOME}/perl/lib # MOS Doc ID 2002334.1
opatchauto apply /tmp/21523260 -oh ${ORACLE_HOME} -ocmrf /opt/app/oracle/software/ocm.rsp

This also failed:

OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /opt/app/oracle/product/12.1.0/dbhome_1

opatchauto log file: /opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-29_13-05-54_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /tmp/21523260
Grid Infrastructure Patch(es): 21359755 21359758 21359761 21436941 
DB Patch(es): 21359755 21359758 21555660 

Patch Validation: Successful
User specified the following DB home(s) for this session:
/opt/app/oracle/product/12.1.0/dbhome_1


Performing prepatch operations on RAC Home (/opt/app/oracle/product/12.1.0/dbhome_1) ... Successful

Applying patch(es) to "/opt/app/oracle/product/12.1.0/dbhome_1" ...
Command "/opt/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch napply -phBaseFile /tmp/OraDB12Home1_oracle_patchList -local  -invPtrLoc /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc -oh /opt/app/oracle/product/12.1.0/dbhome_1 -silent -ocmrf /opt/app/oracle/software/ocm.rsp" execution failed: 
OPatch failed to restore OH '/opt/app/oracle/product/12.1.0/dbhome_1'. Consult OPatch document to restore the home manually before proceeding.
UtilSession failed: Re-link fails on target "javavm_refresh ioracle".

Log file Location for the failed command: /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_13-06-28PM_1.log

For more details, please refer to the log file "/opt/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/21523260/opatch_gi_2015-12-29_13-05-54_deploy.debug.log".

Apply Summary:
Following patch(es) are successfully installed:
DB Home: /opt/app/oracle/product/12.1.0/dbhome_1:

Following patch(es) failed to be installed:
DB Home: /opt/app/oracle/product/12.1.0/dbhome_1: 21359755,21359758,21555660
Command failure exception

opatchauto failed with error code 2.

Further Investigation

MOS Doc ID 1933203.1 suggests the cause could be:

A relink error relating to target " jox_refresh_knlopt " typically implies a mismatch of patch levels involving an OJVM PSU patch:

  • OJVM PSU patches require that the database home is already patched to at least October 2014 Database PSU ( or equivalent Security Patch Update (SPU) or Database Patch for Exadata for installations using these other patching models ).

Emphasis in original

The recommended solution is to apply:

  1. Patch 21359755: DATABASE PATCH SET UPDATE 12.1.0.2.5
  2. Patch 21555660: ORACLE JAVAVM COMPONENT 12.1.0.2.5 DATABASE PSU (OCT2015)
  3. Post-installation for DB Patch

Clean Up Patches

As I now have three (3) patches to manage, I decided to create a special patch directory, /opt/app/oracle/software/patches , and move the patches there:

mkdir -p /opt/app/oracle/software/patches/P21523260
mv /tmp/21523260 /opt/app/oracle/software/patches/P21523260
mv /tmp/PatchSearch.xml /opt/app/oracle/software/patches/P21523260
mkdir -p /opt/app/oracle/software/patches/P21359755 /opt/app/oracle/software/patches/P21555660

The patches were downloaded in /tmp , and then unpacked into the appropriate patch directory as follows:

unzip -q /tmp/p21359755_121020_Linux-x86-64.zip -d /opt/app/oracle/software/patches/P21359755
unzip -q /tmp/p21555660_121020_Linux-x86-64.zip -d /opt/app/oracle/software/patches/P21555660

Apply DB PSU

Followed the procedure in Patch 21359755 - Database Patch Set Update 12.1.0.2.5 (Includes CPUOct2015) .

Check for Patch Conflicts

Ran the following command as the oracle user in order to detect any patch conflicts:

opatch prereq CheckConflictAgainstOHWithDetail -ph /opt/app/oracle/software/patches/P21359755/21359755/

The output was:

Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2015, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /opt/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_19-47-07PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

Apply DB Patch

Ran the following command as the oracle user to apply the DB PSU:

opatch apply -ocmrf /opt/app/oracle/software/ocm.rsp /opt/app/oracle/software/patches/P21359755/21359755/

The output was:

Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_19-54-09PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   19769480  20299023  20831110  21359755  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/app/oracle/product/12.1.0/dbhome_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '19769480' to OH '/opt/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.rdbms.deconfig, 12.1.0.2.0...

Patching component oracle.xdk, 12.1.0.2.0...

Patching component oracle.tfa, 12.1.0.2.0...

Patching component oracle.rdbms.util, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.xdk.parser.java, 12.1.0.2.0...

Patching component oracle.oraolap, 12.1.0.2.0...

Patching component oracle.xdk.rsf, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...

Patching component oracle.rdbms.rman, 12.1.0.2.0...

Patching component oracle.ldap.rsf, 12.1.0.2.0...

Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
Applying sub-patch '20299023' to OH '/opt/app/oracle/product/12.1.0/dbhome_1'
ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.tfa, 12.1.0.2.0...

Patching component oracle.rdbms.deconfig, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.rdbms.rsf.ic, 12.1.0.2.0...

Patching component oracle.ldap.rsf, 12.1.0.2.0...

Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
Applying sub-patch '20831110' to OH '/opt/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.oraolap.dbscripts, 12.1.0.2.0...

Patching component oracle.ldap.rsf, 12.1.0.2.0...

Patching component oracle.tfa, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Applying sub-patch '21359755' to OH '/opt/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.assistants.server, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.tfa, 12.1.0.2.0...
Composite patch 21359755 successfully applied.
Log file location: /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_19-54-09PM_1.log

OPatch succeeded.

Apply OJVM PSU

Followed the procedure in Patch 21555660: ORACLE JAVAVM COMPONENT 12.1.0.2.5 DATABASE PSU (OCT2015) .

Check for Conflicts with OJVM PSU

Ran the following command as the oracle user in order to detect any patch conflicts:

opatch prereq CheckConflictAgainstOHWithDetail -ph /opt/app/oracle/software/patches/P21555660/21555660/

The output was:

Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2015, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /opt/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_20-32-50PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

Apply OJVM PSU - Attempt #1

Ran the following command as the oracle user in order to apply the OJVM PSU:

opatch apply -ocmrf /opt/app/oracle/software/ocm.rsp -silent /opt/app/oracle/software/patches/P21555660/21555660/

This failed as can be seen in the following output:

Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_20-33-44PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   21555660  

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/app/oracle/product/12.1.0/dbhome_1')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '21555660' to OH '/opt/app/oracle/product/12.1.0/dbhome_1'
ApplySession: Optional component(s) [ oracle.sqlj, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.javavm.client, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.javavm.server.core, 12.1.0.2.0...

Patching component oracle.javavm.server, 12.1.0.2.0...

--------------------------------------------------------------------------------
***************************************************************************************************
***************************************************************************************************
**                                ATTENTION                                                      **
**                                                                                               **
** It is critical and essential to complete the post installation (if you applying the patch) or **
** post deinstallation (if you are doing rollback of the patch) as per Readme to have a working  **
** and functional Oracle JavaVM Component.                                                       **
** Failure to follow the instructions could lead to inconsistencies in the database.             **
**                                                                                               **
***************************************************************************************************
***************************************************************************************************

--------------------------------------------------------------------------------

Make failed to invoke "/usr/bin/make -f ins_rdbms.mk javavm_refresh ioracle ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_1"....'make: perl: Command not found
make: *** [javavm_refresh] Error 127
'

The following make actions have failed :

Re-link fails on target "javavm_refresh ioracle".


Do you want to proceed? [y|n]
N (auto-answered by -silent)
User Responded with: N

Restoring "/opt/app/oracle/product/12.1.0/dbhome_1" to the state prior to running NApply...
Checking if OPatch needs to invoke 'make' to restore some binaries...
Make failed to invoke "/usr/bin/make -f ins_rdbms.mk javavm_refresh ioracle ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_1"....'make: perl: Command not found
make: *** [javavm_refresh] Error 127
'

--------------------------------------------------------------------------------
Failed to run make commands. Please contact Oracle Support.
OPatch failed to restore OH '/opt/app/oracle/product/12.1.0/dbhome_1'. Consult OPatch document to restore the home manually before proceeding.

NApply was not able to restore the home.  Please invoke the following scripts:
  - restore.[sh,bat]
  - make.txt (Unix only)
to restore the ORACLE_HOME.  They are located under 
"/opt/app/oracle/product/12.1.0/dbhome_1/.patch_storage/NApply/2015-12-29_20-33-44PM"

UtilSession failed: Re-link fails on target "javavm_refresh ioracle".

Log file location: /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_20-33-44PM_1.log

OPatch failed with error code 73

Recover from Failed PSU Apply of OJVM

As instructed by the previous output, I ran the following commands:

cd /opt/app/oracle/product/12.1.0/dbhome_1/.patch_storage/NApply/2015-12-29_20-33-44PM
./restore.sh

The output was:

This script is going to restore the Oracle Home to the previous state.
It does not perform any of the following:
- Running init/pre/post scripts
- Oracle binary re-link
- Customized steps performed manually by user
Please use this script with supervision from Oracle Technical Support.
About to modify Oracle Home( /opt/app/oracle/product/12.1.0/dbhome_1 )
Do you want to proceed? [Y/N]
y
User responded with : Y
Restore script completed.

After some trial and error, I successfully used the following commands to re-link the Oracle kernel:

export PERL5LIB=${ORACLE_HOME}/perl/lib
export PATH=${ORACLE_HOME}/perl/bin:${PATH}
sh make.txt 

The output was:

perl /opt/app/oracle/product/12.1.0/dbhome_1/javavm/install/update_javavm_binaries.pl -refresh
chmod 755 /opt/app/oracle/product/12.1.0/dbhome_1/bin

 - Linking Oracle 
rm -f /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle
/opt/app/oracle/product/12.1.0/dbhome_1/bin/orald  -o /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/ -L/opt/app/oracle/product/12.1.0/dbhome_1/lib/ -L/opt/app/oracle/product/12.1.0/dbhome_1/lib/stubs/   -Wl,-E /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/opimai.o /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/ssoraed.o /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv12 -Wl,--no-whole-archive /opt/app/oracle/product/12.1.0/dbhome_1/lib/nautab.o /opt/app/oracle/product/12.1.0/dbhome_1/lib/naeet.o /opt/app/oracle/product/12.1.0/dbhome_1/lib/naect.o /opt/app/oracle/product/12.1.0/dbhome_1/lib/naedhs.o /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/config.o  -lserver12 -lodm12 -lcell12 -lnnet12 -lskgxp12 -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lclient12  -lvsn12 -lcommon12 -lgeneric12 -lknlopt `if /usr/bin/ar tv /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap12" ; fi` -lskjcx12 -lslax12 -lpls12  -lrt -lplp12 -lserver12 -lclient12  -lvsn12 -lcommon12 -lgeneric12 `if [ -f /opt/app/oracle/product/12.1.0/dbhome_1/lib/libavserver12.a ] ; then echo "-lavserver12" ; else echo "-lavstub12"; fi` `if [ -f /opt/app/oracle/product/12.1.0/dbhome_1/lib/libavclient12.a ] ; then echo "-lavclient12" ; fi` -lknlopt -lslax12 -lpls12  -lrt -lplp12 -ljavavm12 -lserver12  -lwwg  `cat /opt/app/oracle/product/12.1.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnro12 `cat /opt/app/oracle/product/12.1.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnnzst12 -lzt12 -lztkg12 -lmm -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lztkg12 `cat /opt/app/oracle/product/12.1.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnro12 `cat /opt/app/oracle/product/12.1.0/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnnzst12 -lzt12 -lztkg12   -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 `if /usr/bin/ar tv /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo12 -lserver12"; fi` -L/opt/app/oracle/product/12.1.0/dbhome_1/ctx/lib/ -lctxc12 -lctx12 -lzx12 -lgx12 -lctx12 -lzx12 -lgx12 -lordimt12 -lclsra12 -ldbcfg12 -lhasgen12 -lskgxn2 -lnnzst12 -lzt12 -lxml12 -locr12 -locrb12 -locrutl12 -lhasgen12 -lskgxn2 -lnnzst12 -lzt12 -lxml12  -lgeneric12 -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lsnls12 -lunls12  -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lasmclnt12 -lcommon12 -lcore12  -laio -lons    `cat /opt/app/oracle/product/12.1.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/opt/app/oracle/product/12.1.0/dbhome_1/lib -lm    `cat /opt/app/oracle/product/12.1.0/dbhome_1/lib/sysliblist` -ldl -lm   -L/opt/app/oracle/product/12.1.0/dbhome_1/lib
test ! -f /opt/app/oracle/product/12.1.0/dbhome_1/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/dbhome_1/bin/oracle /opt/app/oracle/product/12.1.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/dbhome_1/bin/oracle

Apply OJVM PSU - Attempt #2

Ran the following commands as the oracle user in order to apply the OJVM PSU:

export ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_1
export PERL5LIB=${ORACLE_HOME}/perl/lib
export PATH=${ORACLE_HOME}/perl/bin:${PATH}
opatch apply -ocmrf /opt/app/oracle/software/ocm.rsp -silent /opt/app/oracle/software/patches/P21555660/21555660/

This succeeded as can be seen in the following output:

Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_20-40-29PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   21555660  

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/app/oracle/product/12.1.0/dbhome_1')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '21555660' to OH '/opt/app/oracle/product/12.1.0/dbhome_1'
ApplySession: Optional component(s) [ oracle.sqlj, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.javavm.client, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.javavm.server.core, 12.1.0.2.0...

Patching component oracle.javavm.server, 12.1.0.2.0...

--------------------------------------------------------------------------------
***************************************************************************************************
***************************************************************************************************
**                                ATTENTION                                                      **
**                                                                                               **
** It is critical and essential to complete the post installation (if you applying the patch) or **
** post deinstallation (if you are doing rollback of the patch) as per Readme to have a working  **
** and functional Oracle JavaVM Component.                                                       **
** Failure to follow the instructions could lead to inconsistencies in the database.             **
**                                                                                               **
***************************************************************************************************
***************************************************************************************************

--------------------------------------------------------------------------------

Patch 21555660 successfully applied.
Log file location: /opt/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2015-12-29_20-40-29PM_1.log

OPatch succeeded.

Post-Installation Steps

Since no databases have been created yet, there is no need to run the data patches.