Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-71-2053185.1
Update Date:2017-12-22
Keywords:

Solution Type  Technical Instruction Sure

Solution  2053185.1 :   SuperCluster: Best Practices for Using Exachk  


Related Items
  • Oracle SuperCluster M6-32 Hardware
  •  
  • Oracle SuperCluster T5-8 Hardware
  •  
  • Oracle Platinum Services
  •  
  • Oracle SuperCluster T5-8 Half Rack
  •  
  • Oracle SuperCluster Specific Software
  •  
Related Categories
  • PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
  •  
  • Tools>Primary Use>Data Collection
  •  




In this Document
Goal
Solution
 Setting up Exachk in your SuperCluster Environment
 Notes and considerations
 Deployment
 Merging collections
 Advanced Setup
References


Applies to:

Oracle SuperCluster T5-8 Half Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Platinum Services - Version N/A to N/A
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster Specific Software - Version 2.x to 2.x [Release 2.0]
Oracle Solaris on SPARC (64-bit)
SPARC

Goal

SuperCluster Best Practices for using Exachk

- Download and install Exachk

- Exachk in Global vs. Local Zones

- Exachk in Application vs. Database zones

- Merging Exachk data collections for upload

- Proactive monitoring with Exachk

Solution

Setting up Exachk in your SuperCluster Environment

Once installed and used correctly on a regular basis, Oracle's Engineered Systems Configuration Audit Tool (aka Exachk) utility can be an invaluable aid in keeping your system up to date and recording changes, etc. as well as providing baselines for support analysis in the event such data is required to resolve a Service Request. There are configuration management utilities available to manage collections from across all Engineered Systems in a given environment. 

When exachk is run from the primary ldom as user 'root' it will discover all RAC/database instances and compute nodes and all storage servers and InfiniBand switches in the fabric and run exachk utilities specific to each.

When it is run in a DB zone or virtualized environment it will collect data from one or all instances / RAC nodes.

Command line options are available to run exachk on subset of system components; e.g. -clusternodes to designate Database Servers, -cells to designate Storage Servers and -ibswitches to designate InfiniBand Switches, etc.

See Exachk User's Guide for Engineered Systems Configuration Audit Tool for complete command line details.

 

Notes and considerations

If installed and run as root It will collect connectivity data from the GZ to the storage cells and other nodes; however it will only collect DB/RAC data if run as user root from within nodes or zones in which RDMBS and / or RAC software is installed.

By default there should be no need to set environment variables or run as any other user than root however there may be exceptions when one is asked to run it in some debug or extra-ordinary fashion by an Oracle Support Analyst. 

There are projects underway to make Exachk run without requiring super user privileges and without having root ssh/RSH key equivalency setup; however, currently it is best practice to have things set up within a given SuperCluster to allow Exachk to run as currently implemented.

The output destination is /opt/oracle.SupportTools/exachk unless otherwise specified. Set environment variable RAT_OUTPUT to change this. E.g.

export RAT_OUTPUT=/opt/oracle.SupportTools/exachk/output


Multiple collections can be merged into one file for ease of uploading to MOS for analysis; e.g. data collections from multiple GZ and NGZ environments for a single case, see example.

  

NOTE In zones there is currently an issue with discovery, and so one must set the RAT_ORACLE_HOME and RAT_GRID_HOME environment variables in some cases. This note will be removed once it is resolved.

  

Deployment

1. Download the latest copy of Exachk from <Document 1070954.1> into /opt/oracle.SupportTools directory on the primary node/LDOM of the SuperCluster.

    There may already exist an installed version of exachk, in which case the existing version should be backed up if desired or removed before installing the new version.
    There is a script that can be used to deploy Exachk in SuperCluster zones from the primary LDOM attached to this document. It will backup the current version, if any, and install the new.
    The latest version must be downloaded from <Document 1070954.1> and placed into the /opt/oracle.SupportTools directory along with the attached deploy_exachk.sh script.


2. Use the attached deploy_exachk.sh
to deploy and install Exachk into /opt/oracle.SupportTools/exachk directory in the primary ldom and in each zone.

Note: You will have to do this again for each of the other LDOMs
2015-10-07 NOTE: Will automate that as well in next version of the script.

# cd /opt/oracle.SupportTools/
root@orlt5db01:/opt/oracle.SupportTools# ./deploy_exachk.sh exachk_121024_bundle.zip
 
Oct 14 21:16:52 orlt5db01 ./deploy_exachk.sh 12.1.0.2.4.0 executing as PID 11479
 
Deploying Exachk from exachk_121024_bundle.zip into all zones...
WARNING: exachk_121024_bundle.zip had better be an exachk zip bundle from MOS Note 1070954.1. If it's not, hit Ctrl-C now....
Hit Ctrl-c to abort, Enter to continue:


<snip>
All done........
root@orlt5db01:/opt/oracle.SupportTools#


Example of what a zone's /opt/oracle.SupportTools looks like when this is finished:

orlt5db01z1: total 68
orlt5db01z1: drwxr-xr-x   7 root     root          10 Oct 14 19:27 .
orlt5db01z1: drwxr-xr-x   8 root     sys            8 Oct  7 22:06 ..
orlt5db01z1: drwxr-xr-x   3 root     root          21 Oct 14 19:27 Exachk.old
orlt5db01z1: drwxr-xr-x   3 root     root          23 Oct 14 19:06 Exachk
orlt5db01z1: drwxr-xr-x   2 root     root           7 Oct  5 20:52 em
orlt5db01z1: lrwxrwxrwx   1 root     root          14 Oct 14 19:27 exachk -> Exachk
orlt5db01z1: drwxr-xr-x   2 root     root           6 Oct  7 20:43 onecommand
orlt5db01z1: -rw-r--r--   1 root     root       40830 Oct  7 20:30 resourcecontrol

  

3. Manual / Standalone installation

Manual installation can be accomplished as follows for each host / LDOM / zone in which exachk is to be installed

root@orlt5db01:/opt/oracle.SupportTools# mv exachk Exachk.old
root@orlt5db01:/opt/oracle.SupportTools# mkdir exachk
root@orlt5db01:/opt/oracle.SupportTools# unzip exachk_121024_bundle.zip -d exachk
root@orlt5db01:/opt/oracle.SupportTools# unzip exachk/exachk.zip -d exachk

  

4. Check Version

# /opt/oracle.SupportTools/exachk/exachk -v

Note if root RSA keys are set up for ssh, then this is an example of an easy way to check all zones in a given LDOM:

root@orlt5db01:/opt/oracle.SupportTools# zoneadm list | grep -v global > zone_list
root@orlt5db01:/opt/oracle.SupportTools# hostname >> zone_list
root@orlt5db01:/opt/oracle.SupportTools# /opt/oracle.supercluster/bin/dcli -g zone_list -l root /opt/*Tools/exachk/exachk -v
orlt5db01z1:
orlt5db01z1: EXACHK  VERSION: 12.1.0.2.4_20150723
orlt5db01z2:
orlt5db01z2: EXACHK  VERSION: 12.1.0.2.4_20150723
orlt5db01z3:
orlt5db01z3: EXACHK  VERSION: 12.1.0.2.4_20150723
orlt5db01z4:
orlt5db01z4: EXACHK  VERSION: 12.1.0.2.4_20150723
orlt5db01:
orlt5db01: EXACHK  VERSION: 12.1.0.2.4_20150723

 

5. Run it in GZ / Primary LDOM:

root@orlt5db01:/opt/oracle.SupportTools/exachk# ./exachk
exachk did not find the inventory location on orlt5db01 from environment.Does orlt5db01 have Oracle software installed [y/n][n]?n

Checking ssh user equivalency settings on all nodes in cluster

Node orlt5db02 is configured for ssh user equivalency for root user
 
Node orlt5db03 is configured for ssh user equivalency for root user
 
Node orlt5db04 is configured for ssh user equivalency for root user
<snip>


6. Run it in non-GZ local zone, aka NGZ:

root@orlt4db01:/opt/oracle.SupportTools/exachk# zlogin orlt4db01z2
[Connected to zone 'orlt4db01z2' pts/10]
Oracle Corporation      SunOS 5.11      11.1    April 2014
root@orlt4db01z2:~# cd /opt/*Tools
root@orlt4db01z2:/opt/oracle.SupportTools# cd exachk
root@orlt4db01z2:/opt/oracle.SupportTools/exachk# ./exachk


Checking ssh user equivalency settings on all nodes in cluster
<snip>

  

Merging collections

With version 12.1.0.2.4 come some new features. For example, the ability to merge collections from various places into one file that can be uploaded to an SR via 'curl'.

Note: The -force option on the end after the comma separated file list must be there or the command will fail.
root@orlt4db01:/opt/oracle.SupportTools/exachk# ./exachk -merge exachk_orlt4db01_090715_214648.zip,/zoneHome/orlt4db01z1/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z1_orlt4z1_090415_183027.zip,/zoneHome/orlt4db01z2/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z2_orlt4z2_090715_211212.zip,/zoneHome/orlt4db01z3/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z3_dbm01_031015_212416.zip -force

Merging following collections:

  exachk_orlt4db01_090715_214648.zip
  /zoneHome/orlt4db01z1/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z1_orlt4z1_090415_183027.zip
  /zoneHome/orlt4db01z2/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z2_orlt4z2_090715_211212.zip
  /zoneHome/orlt4db01z3/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z3_dbm01_031015_212416.zip


Started merging exachk_orlt4db01_090715_214648.zip
  ........................................................................................

Started merging /zoneHome/orlt4db01z1/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z1_orlt4z1_090415_183027.zip
  ............................................................................................................................................................................................

Started merging /zoneHome/orlt4db01z2/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z2_orlt4z2_090715_211212.zip
  ............................................................................................................................................................................................

Started merging /zoneHome/orlt4db01z3/root/opt/oracle.SupportTools/exachk/exachk_orlt4db01z3_dbm01_031015_212416.zip
  ............................................................................................................................................................................................
<snip>
Detailed report (html) - /opt/oracle.SupportTools/exachk/exachk_orlt4db01_090715_214648_merge/exachk_orlt4db01_090715_214648_merge.html

UPLOAD(if required) - /opt/oracle.SupportTools/exachk/exachk_orlt4db01_090715_214648_merge.zip


root@orlt4db01:/opt/oracle.SupportTools/exachk#

  

Advanced Setup

The following is all about how to set up Exachk to run on a periodic basis.  First we set the following environment variables.

export RAT_TIMEOUT=120
export RAT_ROOT_TIMEOUT=600
export RAT_PASSWORDCHECK_TIMEOUT=30
export RAT_OUTPUT=/opt/oracle.SupportTools/exachk/output
export RAT_COPY_EM_XML_FILES=1


Ideally, create a file with these commands because they will need to be run anytime you restart the daemon.

  

Set the autorun day of month and time of day based on customer input.

AUTORUN_SCHEDULE * * * * :- Automatic run at specific time

                 - - - -

                 ¦ ¦ ¦ ¦

                 ¦ ¦ ¦ +----- day of week (0 - 6) (0 to 6 are

                 ¦ ¦ ¦       Sunday to Saturday)

                 ¦ ¦ +---------- month (1 - 12)

                 ¦ +--------------- day of month (1 - 31)

                 +-------------------- hour (0 - 23)

For example:
to run on the 5th of every month at 2am

     ./exachk -set "AUTORUN_SCHEDULE=2 5 * *"

  

Check the results

./exachk -get all

  

Run the presetup tasks. This sets up the required ssh equivalences.

./exachk -initpresetup

  

Setup the daemon and inittab

./exachk -initsetup


The preceding step may produce an error. If so, simply wait 1-2 minutes before executing the next command to give init time to respawn the process.

  

Check that the daemon is running.

./exachk -d nextautorun

 

 Remember to keep Exachk up to date and to monitor root zpool file system usage on a regular basis!

 

References

<NOTE:1070954.1> - Oracle Exadata Database Machine exachk or HealthCheck

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback