Overview
The OMS on CRONULLA fails to start with the following messages:
Oracle Management Server Could Not Be Started
Status
Resolved
Solution
Synchronise the clocks between VICTORIA and GRIDCTRL . This synchronises the clocks betweeen CRONULLA and GRIDCTRL as the former is a VM on and VICTORIA .
References
- EMDIAG Omsvfy 13c,12c Kit - Download and Install (Doc ID 1374450.1)
- EMDIAG Omsvfy 13c, 12c Kit - How to Use it (Doc ID 1374945.1)
- How to Disable or set SELinux to Permissive mode (Doc ID 457458.1)
- EMDIAG REPVFY Kit for Cloud Control 12c, 13c - Download, Install/De-Install and Upgrade (Doc ID 1426973.1)
Investigation
OMS Start Up Fails
The OMS on CRONULLA fails to start with the following messages:
Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Could Not Be Started Check EM Server log file for details: /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out Oracle Management Server is Down
The log ( /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out ) does not appear to contain any useful information:
oracle.jdbc.thinLogonCapability not set
Successfully read all properties from repos.
securePort = 4903
<ConfigManager.getOMSSecureStatus> isOMSSecure = 1 isOMSSecureLocked = 1
In ConfigManager.loadProperties. Loaded properties from property file </opt/app/oracle/gc_inst/em/EMGC_OMS1/sysman/config/emomslogging.properties>
Plug-in deployment status: SUCCESS
Fetched repository credentials from Credential Store
<Dec 25, 2016 9:10:12 AM> <FINEST> <NodeManager> <Waiting for the process to die: 2638>
<Dec 25, 2016 9:10:12 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<Dec 25, 2016 9:10:12 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
Install Diagnostic Tools
Following the instructions in EMDIAG Omsvfy 13c,12c Kit - Download and Install (Doc ID 1374450.1) , I installed the omsvfy tool as follows:
cd /opt/app/oracle/middleware/oms mkdir emdiag unzip -q /tmp/omsvfy12.zip -d /opt/app/oracle/middleware/oms/emdiag
The tool was verified as follows:
export ORACLE_HOME=/opt/app/oracle/middleware/oms
export EMDIAG_HOME=${ORACLE_HOME}/emdiag
cd ${EMDIAG_HOME}/bin
./omsvfy version
The result was:
Component Version
-------------------- -------------------------
OMSVFY 2016.0627
Linux EL6.7 3.8.13 (XEN)
x86_64-linux-thread-mult
-------------------- -------------------------
OMS 12.1.0.4.0
OMS DST Files 4
OMS JDK 1.6.0_43
OMS JRE 1.6.0_43
OMS JRE DST tzdata2012i
OMS JSch 0.1.44o
OMS OPatch 11.1.0.10.4
OMS OUI 11.1.0.12.0
OMS PERL 5.10.0
OMS RCU 12.1.2.0.0
OMS Zip 3.0
OMS Unzip 6.00
-------------------- -------------------------
JRF 11.1.1.2.0
JRF OPatch 11.1.0.9.9
JRF OUI 11.1.0.9.0
-------------------- -------------------------
BI Publisher 0.0.0.0.1
BI Publisher OPatch na
BI Publisher OUI na
-------------------- -------------------------
Webtier 11.1.1.7.0
Webtier DST Files 4
Webtier JDK 1.6.0_35
Webtier JRE 1.6.0_35
Webtier JRE DST tzdata2012c
Webtier OPatch 11.1.0.9.9
Webtier OUI 11.1.0.9.0
Webtier Zip 2.32
Webtier Unzip 5.52
-------------------- -------------------------
WebLogic Server 10.3.6.0
WebLogic JDK 1.6.0_43
WebLogic JRE 1.6.0_43
WebLogic JRE DST tzdata2012i
-------------------- -------------------------
Run Diagnostics Tool
Following the instructions in EMDIAG Omsvfy 13c, 12c Kit - How to Use it (Doc ID 1374945.1) ,
cd ${EMDIAG_HOME}/bin
./omsvfy verify all -level 2 -details
Four (4) violations were found:
-- --------------------------------------------------------------------- --
-- OMSVFY: 2016.0627 OMS: 12.1.0.4.0 25-Dec-2016 11:41:03 --
-- --------------------------------------------------------------------- --
verifyAdminServer
verifyJAVA
201. Java TZ information not up-to-date: 3
OMS JAVA TZ information is outdated (tzdata2012i)
WebTier JAVA TZ information is outdated (tzdata2012c)
WebLogic JAVA TZ information is outdated (tzdata2012i)
verifyNodeManager
verifyOMS
002. Invalid lok/pid file: 2
Lock file found for OMS in directory (/opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok), but app is not running
PID file found for OMS in directory (/opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/EMGC_OMS1.pid), but app is not running
verifyOracle
verifyOS
011. SELinux is not disabled: 1
SELinux is not disabled
verifyOUI
200. Patched OPatch version not registered: 1
OPatch version (11.1.0.10.4) is higher than software component in inventory (11.1.0.10.2)
verifyPERL
verifyWebLogic
4 violation(s) found
Clean Up OMSVFY Violations
Update Software
Used the following commands to update the operating system software:
sudo apt-get update
This did not fix the time-zone problem.
Disable SELINUX
Followed the procedure in How to Disable or set SELinux to Permissive mode (Doc ID 457458.1) (as root ):
- Edit /etc/selinux/config
- Change the SELINUX value to "SELINUX=disabled" .
- Reboot the server.
Remove Lock and PID Files
Remove the identified lock and PID files as follows:
rm -f /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok rm -f /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/EMGC_OMS1.pid
Unregistered OPatch Version
I could not find any hits on My Oracle Support for instructions on how to register OPatch in the Oracle Inventory. I left it as i.
Validate Repository
Make Tablespaces Read-Write
During my attempt to migrate the OMR from GRIDTCTRL to GORDON , I had made tablespaces Read-Only as described in 06 Move OMR . I used the following SQL*Plus commands to find what tablespaces were Read-Only in the REPOS database:
set pages 62 lines 204 select tablespace_name, status from dba_tablespaces;
The results were:
TABLESPACE_NAME STATUS -------------------------- --------- SYSTEM ONLINE SYSAUX ONLINE UNDOTBS1 ONLINE TEMP ONLINE USERS READ ONLY MGMT_ECM_DEPOT_TS READ ONLY MGMT_TABLESPACE READ ONLY MGMT_AD4J_TS READ ONLY RMAN_CATALOG READ ONLY 9 rows selected.
I used the following SQL statements to make these tablespaces Read-Write:
alter tablespace users read write; alter tablespace MGMT_ECM_DEPOT_TS read write; alter tablespace MGMT_TABLESPACE read write; alter tablespace MGMT_AD4J_TS read write; alter tablespace RMAN_CATALOG read write;
This did not fix the start-up problem.
Install Diagnostic Tools
Followed the instructions in EMDIAG REPVFY Kit for Cloud Control 12c, 13c - Download, Install/De-Install and Upgrade (Doc ID 1426973.1) to install repvfy on GRIDCTRL :
unzip -qo /home/oracle/repvfy12_2016.0909.zip -d /opt/oracle/app/OracleHomes/db11g/emdiag chmod u+x /opt/oracle/app/OracleHomes/db11g/emdiag/bin/repvfy cd /opt/oracle/app/OracleHomes/db11g/emdiag/bin ./repvfy install
The following summary is produced:
COMPONENT INFO -------------------- -------------------- EMDIAG Version 2016.0909 EMDIAG Edition 2 Repository Version 12.1.0.4.0 Database Version 11.2.0.4.0 Test Version 2016.1225 Repository Type CENTRAL Verify Tests 569 Object Tests 228 Deployment SMALL 9 rows selected.
Verify Repository
I ran the following command to verify the repository:
cd /opt/oracle/app/OracleHomes/db11g/emdiag/bin ./repvfy -level 2 -detail
The following summary is produced:
verifyAGENTS
======================================================================
1005. Active Agents with clock-skew problems
=--------- ---------- ---------- ---------- ---------- ---------- ---------=
= Action:
= - If the skew is only a few minutes, adjust the OS clock on the machine
= or use a NTP service to act as the 'master of time' for the machines.
= - If the skew is more than an hour, the timezone of the Agent will need
= to get changed, to have the correct time represented.
= To change the timezone of an Agent:
= * Update the property on the Agent side:
= In case the OS changed, and the OS settings are different:
= $ emctl config agent updateTZ
= In case the Agent has a bad setting, which is NOT in sync with the OS:
= $ emctl setproperty agent -name agentTZRegion -value <new value>
= * Update the timezone in the repository for this Agent, to sync up the
= internal parameters:
= SQL> exec mgmt_target.set_agent_tzrgn('<agent_name>','<correct TZ region>')
= SQL> commit
=--------- ---------- ---------- ---------- ---------- ---------- ---------=
AGENT_NAME TIMEZONE_REGION SECONDS CLOCK_SKEW
-------------------------------------------------- ------------------------------ ---------- -------------
redfern1.yaocm.id.au:3872 Australia/Sydney -154 -00h02m34s
cronulla.yaocm.id.au:3872 Australia/Sydney -147 -00h02m27s
redfern2.yaocm.id.au:3872 Australia/Sydney -146 -00h02m26s
3 rows selected.
verifyREPOSITORY
======================================================================
1. Insufficient RAW partitions for new data
=--------- ---------- ---------- ---------- ---------- ---------- ---------=
= Action:
= - Check the status of the repository DBMS_SCHEDULER jobs, and make sure
= all tasks are running on the appointed time
= (In particular the 'EM Daily Maintenance' job)
= - To do one-time manual cleanup, run this routine:
= SQL> CONNECT sysman
= SQL> exec gc_interval_partition_mgr.partition_maintenance
=--------- ---------- ---------- ---------- ---------- ---------- ---------=
MAX_PTIME
-------------------
2016/09/11 00:00:00
1 row selected.
I fixed the verifyREPOSITORY as follows:
[oracle@gridctrl bin]$ sqlplus sysman SQL*Plus: Release 11.2.0.4.0 Production on Sun Dec 25 18:54:42 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. Enter password: Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production With the Partitioning option SQL> exec gc_interval_partition_mgr.partition_maintenance PL/SQL procedure successfully completed. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production With the Partitioning option
This did not fix the start-up problem.
Install REPVFY on CRONULLA
Install Software
Followed the instructions in EMDIAG REPVFY Kit for Cloud Control 12c, 13c - Download, Install/De-Install and Upgrade (Doc ID 1426973.1) to install repvfy on CRONULLA :
unzip -qo /home/oracle/repvfy12_2016.0909.zip -d /opt/app/oracle/middleware/oms/emdiag/
Create TNSNAMES Alias for OMR
Created the following entry in the TNSNAMES.ORA on CRONULLA :
cat >>/opt/app/oracle/middleware/oms/network/admin/tnsnames.ora <<DONE
# -----------------------------------------------------------------------------
# TNS Names entries
# -----------------------------------------------------------------------------
repos = (DESCRIPTION =
(ADDRESS = (PROTOCOL=TCP)(HOST=gridctrl.yaocm.id.au)(PORT=1521))
(CONNECT_DATA = (SERVICE_NAME=repos.yaocm.id.au)(SERVER=DEDICATED))
)
DONE
Update ORATAB
Added the following entry to the /etc/oratab on CRONULLA :
cat >>/etc/oratab <<DONE oms:/opt/app/oracle/middleware/oms:N DONE
This allows me to crudely set up the Oracle environment for OMS as follows:
[oracle@cronulla admin]$ . oraenv ORACLE_SID = [oracle] ? oms /opt/app/oracle/middleware/oms/bin/orabase: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory The Oracle base for ORACLE_HOME=/opt/app/oracle/middleware/oms is
Create REPVFY Configuration File
The REPVFY configuration file was created as follows:
cd /opt/app/oracle/middleware/oms/emdiag/cfg/ cp repvfy.cfg.template repvfy.cfg vi repvfy.cfg
The following changes were made to repvfy.cfg :
ora_tns=repos
Review Test Results
I ran the following commands on CRONULLA :
cd /opt/app/oracle/middleware/oms/emdiag/bin ./repvfy -level 2 -detail
The test results were:
-- --------------------------------------------------------------------- -- -- REPVFY: 2016.0909 Repository: 12.1.0.4.0 25-Dec-2016 21:20:56 -- --------------------------------------------------------------------------- -- Module: all Test: 0, Level: 2 -- -- --------------------------------------------------------------------- -- -- Visit My Oracle Support for a newer version of repvfy -- -- --------------------------------------------------------------------- -- verifyAGENTS 1005. Active Agents with clock-skew problems: 3 verifyASLM verifyAVAILABILITY verifyBLACKOUTS verifyCAT verifyCORE verifyECM verifyEMDIAG verifyEVENTS verifyEXADATA verifyJOBS verifyJVMD verifyLOADERS verifyMETRICS verifyNOTIFICATIONS verifyOMS verifyPLUGINS verifyREPOSITORY verifyTARGETS verifyTEMPLATES verifyUSERS
Synchronise Clocks between VICTORIA and GRIDCTRL
On GRIDCTRL , I ran the following command:
ssh root@victoria date --utc $(date --utc +"%m%D%H%M%Y.%S")
I then rebooted VICTORIA , and the OMS server was then able to be started on CRONULLA .