EMCC: 13.3 always-on install

Although always-on monitoring cannot be downloaded from the self-update console as it is documented, it is still available from the software distribution.

To install and setup always-on 13.3, once the previous 13.2 version has been uninstalled, proceed as below.

In the this setup, EMS is being configured colocated with the 2 OMS VMs em01 and em02, and the OMS repository is running on a RAC database.

 

#1 Copy the emkey into the OMS directory

./emctl config emkey -copy_to_repos

 

#2 On the first EMS node, unzip /u01/app/oracle/gc133/sysman/ems/ems_13.3.0.0.0.zip into the /u01/app/oracle/ems directory

Note that the two following environment variables can be set for EMS to work. They must be setup on all EMS nodes.

export JAVA_HOME=$ORACLE_HOME/oracle_common/jdk/jre
export EMS_HOME=/u01/app/oracle/ems

 

#3 Create the responsfile

emsRepConnectString=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emsdb01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=emsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=ao_ems)))
emsRepUsername=ems
emsRepPassword=<pwd>
emRepConnectString=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb01-vip.)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=em_oms)))
emRepUsername=sysman
emRepPassword=<pwd>
emsPort=8081
http.protocol=https

Note that here the OMS is running on a RAC.

 

#4 Run emsca from the first node

./emsca -createEmsDbUser=true -responseFile=/u01/app/oracle/ems/scripts/responsefile.txt

Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
 Always-On Monitoring Repository SYSDBA Username : sys
 Always-On Monitoring Repository SYSDBA Password : 
Creating Always-On Monitoring repository user ems
 Agent Registration Password : 
 Keystore for host em01 created successfully.
 Connecting to Always-On Monitoring Repository.
 Creating Always-On Monitoring Repository schema
 Creating repository storage for Targets data.
 Creating repository storage for Alerts and Availability data.
 Creating repository storage for Notification Metadata data.
 Creating repository storage for Target Metric Metadata data.
 Registering Always-On Monitoring instance
 Always-On Monitoring Upload URL: https://em01:8081/upload

 

#5 Run the synchronization from the first node. Eliminate any issue before moving to the setup of the second node.

$ ./emsctl sync
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Connecting to Always-On Monitoring Repository.
Starting synchronization with Enterprise Manager.
Synchronizing with Enterprise Manager repository: sysman@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=em_oms)))
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Thu Aug 22 12:44:58 BST 2019

Note that, the alias referenced in the connect string *MUST* exist in the hosts where the databases instances are started, in addition to the VMs where the ems is running. Otherwise, an error “ora-12545” would be returned.

 

#6 Always on can now be started from the first ems node. And the status be checked

$ ./emsctl start
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Starting Always-On Monitoring.
Waiting for process to start
Retrying...
Notifications Enabled : false
Total Downtime Contacts Configured : 1
Always-On Monitoring is up.

 

$ ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.3.0.0.0
Always-On Monitoring Home : /u01/app/oracle/ems
Started At : August 22, 2019 12:48:17 PM BST
Last Repository Sync : August 22, 2019 1:50:07 PM BST
Upload URL : httpa://em01:8081/upload
Always-On Monitoring Process ID : 17830
Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emsdb01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=emsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=ao_ems)))
Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=em_oms)))
Notifications Enabled : false
Total Downtime Contacts Configured : 1

 

#7 Login to the second node and copy the distribution setup ont he first node

scp -r em01p:/u01/app/oracle/ems/* .

 

#8 Start the ems on this second node

$ cd /u01/ap/oracle/ems/scripts
$ ./emsctl start
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Starting Always-On Monitoring.
Waiting for process to start
Retrying...
Notifications Enabled : false
Total Downtime Contacts Configured : 1
Always-On Monitoring is up.
$ ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.3.0.0.0
Always-On Monitoring Home : /u01/app/oracle/ems
Started At : August 22, 2019 12:49:32 PM BST
Last Repository Sync : August 22, 2019 1:50:07 PM BST
Upload URL : https://em02:8081/upload
Always-On Monitoring Process ID : 10619
Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=emsdb01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=emsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=ao_ems)))
Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=omsdb02-vip)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=em_oms)))
Notifications Enabled : false
Total Downtime Contacts Configured : 1

 

#10 Remove the em key from the repository

$ emctl config emkey -remove_from_repos
Oracle Enterprise Manager Cloud Control 13c Release 3 
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : 
The EMKey has been removed from the Management Repository.

 

#11 Configure the load balancer with a ssl listener configured on the port 8080 (https://em:8080) with a server pool that handle the two EMS servers https://em01:8081 and https://em02:8081

 

#12 Configure the EMS URL in EMCC

$ORACLE_HOME/bin/emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value "https://em:8080/upload" -sysman_pwd $LSYSMAN_PWD


Oracle Enterprise Manager Cloud Control 13c Release 3 
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Property oracle.sysman.core.events.ems.emsURL has been set to value https://em:8081/upload for all Management Servers
OMS restart is not required to reflect the new property value

 

#13 Check that the agents are now aware of the EMS existency

$ cd /u01/app/oracle/ems/scripts
$ ./emsctl list_agents
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Connecting to Always-On Monitoring Repository.
Number of Agents that have communicated with Always-On Monitoring Service in the past hour: 41
URLs of these Agents are: 
https://exadata01:3872/emd/main/
...
https://exadata41:3872/emd/main/

 

#14 Set the EMS downtime contact

If not yet set (check with the first command), set if via the second emctl command:

${ORACLE_HOME}/bin/emctl get property -name "oracle.sysman.core.events.ems.downtimeContact"

${ORACLE_HOME}/bin/emctl set property -name "oracle.sysman.core.events.ems.downtimeContact" -sysman_pwd <pwd> -value name@domain.com

 

#15 Activate the EMS notification temporarily to test the functionality. It is better to otherwise activate it only when the OMS does down, to avoid being flooded by email notifications.

$ cd /u01/app/oracle/ems/scripts
$ ./emsctl enable_notification