SuperCluster - Solaris 11 Local zones running 11.2.0.4 stuck in shutting_down state and ps commands from global zone hang

Asset ID:	1-72-1938952.1
Update Date:	2014-10-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1938952.1 : SuperCluster - Solaris 11 Local zones running 11.2.0.4 stuck in shutting_down state and ps commands from global zone hang

Applies to:

SPARC SuperCluster T4-4 - Version All Versions to All Versions [Release All Releases]
Solaris SPARC Operating System - Version 11.1 to 11.2 [Release 11.0]
Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database - Enterprise Edition - Version 11.2.0.4 to 11.2.0.4 [Release 11.2]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Solaris 11 local zones stuck (wedged) in shutting_down state. Attempt at simple OS commands such as ps become wedged as well.

Changes

Customer is running SuperCluster with Solaris 11 local zones. These zones are running 11.2.0.4 GI and/or DB.

Cause

This is DB_15 in the critical issues note SuperCluster Critical Issues (Doc ID 1452277.1).
<Bug 17997507> - 11.2.0.4: xdmg process exits without closing skgxp context when ora-15311 is see

To verify cd to /zoneHome and run

for i in `find . -name \*xdmg\* |grep asm`
do
grep 15311 *
done

Look in the returned xdmg trace files for "error 15311"

/zoneHome/osc02r8z16db03cn3-client/root/u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_xdmg_2232.trc:error 15311 detected in background process

This background ora-15311 combined with Solaris 11 local zones stuck in shutting_down and ps commands from the global zone wedged is the signature of this bug.

This bug is fixed in 11.2.0.4.7 and beyond.

Solution

This bug is fixed in 11.2.0.4.7 and higher. It is also Documented in <Document 1452277.1 > SuperCluster Critical Issues item DB_15.

Get the system operational by cleanly rebooting the global zone (LDom) if this is not possible panic the global zone(LDom) and then bring it back up.

Make sure that you communicate all action plans clearly to the customer if from siupport as they may have other zones which are functioning fine. They should be given the opportiunity to cleanly shutdown any other items in other zones before the action plan.

References

<BUG:17997507> - 11.2.0.4: XDMG PROCESS EXITS WITHOUT CLOSING SKGXP CONTEXT WHEN ORA-15311 IS SEE
<NOTE:1452277.1> - SuperCluster Critical Issues

Attachments

This solution has no attachment