SuperCluster- Troubleshooting exavm zone creation failures

Asset ID:	1-75-1556451.1
Update Date:	2015-03-23
Keywords:

Solution Type Troubleshooting Sure

Solution 1556451.1 : SuperCluster- Troubleshooting exavm zone creation failures

Applies to:

SPARC SuperCluster T4-4
SPARC SuperCluster T4-4 Full Rack
SPARC SuperCluster T4-4 Half Rack
Oracle SuperCluster T5-8 Full Rack
Oracle SuperCluster T5-8 Half Rack
Oracle Solaris on SPARC (64-bit)
Background: The supported mechanism to deploy zones within a Data Base LDOM on SuperCluster is with the tools provided from the system/platform/supercluster/ssc-exavm package. This package provides two main scripts that are use in the creation of the zone:

1) /opt/oracle.supercluster/zonetools/exavmgen/exavmgen is used to take a dat file combined with the output of Oracle Exadata Deployment Assistant (OEDA) and generate the required xml definitions for the creation of the zone.

For example something similar to the following syntax may have been used to generate the xml:
exavmgen -nodes dbnode1 dbnode2 -datfile dbnode1_zone1.dat dbnode2_zone1 -zipfile OEDA_out.zip

The output of the this script will place the xml file in the following location:
/zoneData/ExaVmConfigDir/{cluster_name}/vmgen/tmp/xmldata/{GLOBAL_ZONE_NAME}.{LOCAL_ZONE_NAME}.xml

2) /opt/oracle.supercluster/zonetools/ssc_exavm/ssc_exavm is the script used to actually create the zone on each host.

For example a command similar to the following may have been used to start the creation of the zone. Optionally the -createvm may have been passed to the proceeding exavmgen script to automatically call ssc_exavm
ssc_exavm -create -vmDataDir /zoneData/ExaVmConfigDir/{CLUSTER_NAME}/vmgen/tmp/xmldata/

3) /opt/oracle.supercluster/zonetools/ssc_exavm/ssc_exavm -template may have been run to create a template zone. This will be noted in `zoneadm list -cv` as a zone with "_T" proceeding the zone name. If a template zone was created, the ssc_exavm command will use `zoneadm -z zonename clone` as opposed to `zoneadm -z zonename install`

NOTE: With the introduction of JOC Ocmd-13.326 version of OneCommand, OneCommand itself will perform the function of exavmgen. In this case the xml file with reside in /opt/oracle.supercluster/zonetools/ssc_exavm/{ZONE_NAME}-zoneConfig.xml

Purpose

To assist in determining the cause of a failed zone creation using scripts provided by the system/platform/supercluster/ssc-exavm package

Troubleshooting Steps

Examine the following files in opt/oracle.supercluster/zonetools/ssc_exavm/ssc_exavm/tmp:

1) {GLOBAL_ZONE_NAME}-{DATE_TIME}.out example: orlscdb01-20131112155557.out This file captures standard out from ssc_exavm

2) {GLOBAL_ZONE_NAME}-{DATE_TIME}.log example: orlscdb01-20131112155557.log This file provides debug information and helpful when investigating variable issues or parsing errors from reading the .xml file

3) {GLOBAL_ZONE_NAME}-{DATE_TIME}.trc example: orlscdb01-20131112155557.trc This file if applicable from error generated during installation attempt

Examine the following files in opt/oracle.supercluster/zonetools/ssc_exavm/ssc_exavm/tmp/vmtrash:

1) {RUN_NUMBER}.SysCall.{SSC_EXAVM_STEP_NUM}.err Example: 10358.SysCall.20.err Contains error generated from command execution

2) {RUN_NUMBER}.SysCall.{SSC_EXAVM_STEP_NUM}.out Example 10358.SysCall.20.out Contains console output returned from command execution (depending on failure may also contain the error)

3) {RUN_NUMBER}.SysCall.{SSC_EXAVM_STEP_NUM}.cmd Example 10358.SysCall.20.cmd Contains the actual command passed from the script

Examine the following OS related zone logs:

1) /system/volatile/install.{ZONE_INSTALL_NUMBER}/install_log Contains log for OS installation in zone. Good place to look for manifest or IPS repository issues.

2) /var/log/zones/zoneadm.{TIMESTAMP).orlscdb01z1.{install | uninstall | clone} Contains command, and manifest location for the zone build.

Examine the following OS related command outputs:

- pkg publisher
- pkg list -fa entire idr*
- pkg list -av ssc-exavm
- dladm show-part
- dladm show-phys
- ipadm show-addr
- dladm show-vnic
- pooladm
- zpool list
- zfs list

References

<NOTE:888828.1> - Exadata Database Machine and Exadata Storage Server Supported Versions
<NOTE:1452277.1> - SuperCluster Critical Issues
<BUG:15813348> - SUNBT7195309 SMB PERFORMANCE TEST SHOWS SEVERE DEGRADATION
<BUG:12951619> - DATABASE TO USE CRITICAL THREADS FEATURE IN SOLARIS

Attachments

This solution has no attachment