FS System: Content details of an IO Module (IOM) ddump File

Asset ID:	1-79-2188416.1
Update Date:	2017-05-04
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2188416.1 : FS System: Content details of an IO Module (IOM) ddump File

Applies to:

Oracle FS1-2 Flash Storage System - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Purpose

Gathering a ddump output from an FS1-2 Drive Enclosure creates a document that is often tens of thousands of lines in length. This document provides keywords to search for which brings you to specific areas of potential interest quickly. It will also provides information on the general layout of the information.

Scope

This document only discusses the contents of a ddump file. It does not discuss the various types of problems the contents can be used to identify.

Details

To actually create a diagnostic dump (ddump) from an FS1-2 Drive Enclosure, please refer to KM document 1954866.1 FS System: How to Collect the System-Wide Diagnostic Dump from Drive Enclosure (DE2-24C or DE2-24P) IO Module (IOM). The ddump log itself is a series of commands run in essentially alphabetical order. Each command output is separated by a line of 70 -'s. Please note that most of the examples below have trimmed output (...) to preserve space.

Note: When gathering a ddump output, be sure to gather this information from both Input/Output Modules (IOMs).

Initially this document will show a list those commands who's information will provide the information most likely to be needed for troubleshooting. As more information becomes available, that list will grow.

Commands:

ddump_drvmgr - provides information about individual drives, their status, WWNs, fault LEDs, port status etc.
ddump_envctrl - provides general information about the Enclosure from the view point of the Canister IOM ddump was gathered from. Information includes temperatures, fan status etc.
ddump_events - provides a list of events.
ddump_general - provides information about the Drive Enclosure, Midplane, Canisters, which Canister this ddump was gathered from (local/remote).
ddump_phycounters - provides Link Rate as well as individual PHY counts of Invalid DWORDs, Disparity errs, DWORD sync loss, PHY reset failed and PHY change count.
ddump_pwrmgr - provides details on the Power Cooling Modules (PCMs) including power used, part number and output voltages.
dumpdrives - provides a quick view of the drives location and status.
envctrl_fan - provides a quick view of the status of the fans in the Enclosure.
envctrl_zone - provides a quick view of the temperatures inside the Enclosure.
getenclosurewwn - provides the WWN of the Drive Enclosure.
logdump - provides a dump of logged messages as far back as the last power cycle. The timestamp at the beginning of each line is based on time since the IOM was last rebooted (search on POWERLOSS).
logdumpnv - provides a dump of logged messages that may go back as far as the original power on (up to the size of flash memory).
phydump - provides a count and status of the SAS PHYs in the IOM. PHYs 0-23 are the drive side connections in the IOM, 24-27 host side.
ver - provides IOM version (Package descriptor) and Vital Product Data (VPD).

Command Examples:

ddump_drvmgr example:

----------------------------------------------------------------------
ddump_drvmgr
Diagnostic dump for the Drive Manager service
**** Drive Manager diagnostic dump ****
HA mode: master
Drives spinning up: 0
Drive bays: 24
Drive Index Base: 0
Allowed drives: SAS OR SATA
Drive power control: supported
Enclosure power loss: no
Pending power loss update: no
...
**** Drive Bay 0 status ****
present       : yes
SES_info_bit : not set
RAID_info_byte: 0x0
spin up time : 65+21:11:29.915
drive_type    : SAS
WWN           : 5000CCA05408BAF5
faults        : none
fault LED     : OFF
array LED     : OFF
inject        : NONE
pending       : ONLINE
current       : ONLINE
SlotA
bypass       : 0x00
SlotB
bypass       : 0x00
force off     : no
...
**** Drive Bay 23 status ****
present       : yes
SES_info_bit : not set
RAID_info_byte: 0x0
spin up time : 0+00:00:28.197
drive_type    : SAS
WWN           : 5000CCA016290A89
faults        : none
fault LED     : OFF
array LED     : OFF
inject        : NONE
pending       : ONLINE
current       : ONLINE
SlotA
bypass       : 0x00
SlotB
bypass       : 0x00
force off     : no
ddump_envctrl example:

----------------------------------------------------------------------
ddump_envctrl
Diagnostic dump for the Environmental Control service
BEGIN RSync ddump for "env_control":
Device role:                        MASTER
Instance run state:                 RUNNING
Sync to Slave status:               None in progress
This instance's next UID will be:   0xEFA633 (slot=0 val=15705651)
Total expanded transactions:        0x0 (0)
Transaction pool capacity:          0x10 (16)
Transaction pool free count:        0x10 (16)
Num concurrent ACKS:                0x18 (24)
WI Store info - UIDs of stored transactions:
   -                      Not stored:
   -                           ERROR:
   -        Syncing M->S (new trans):
   -            Pending ack to slave:
   -          Awaiting worker thread:
   -    In pfnMaster_PerformAction():
   -     Awaiting M_ActionComplete():
   -       Syncing M->S (completion):
   - Retry Syncing M->S (completion):
   -        Syncing S->M (new trans):
   -   In pfnSlave_ActionCompleted():
   -    In RSync_SendTransToClient():
END RSync ddump for "env_control"

max num zones: 8

zone 0
name                              : Ambient
location                          : Mp0:0
currentTemperature                : 23.449
faultStates.generatedFault        : 0
faultStates.detectedFault         : 0
faultStates.generatedPredictedFail: 0
faultStates.detectedPredictedFail : 0
faultStates.elementSpecificFaults : 0x0
defaultCriticalColdTemperature    : 3
defaultCriticalHotTemperature     : 42
modifiedWarningColdTemperature    : 5
modifiedNormalTemperature         : 20
modifiedWarningHotTemperature     : 40

zone 1
name                              : Midplane
...
max num fans: 4

fan 0
name                              : PCM 0 Fan 0
currentFanSpeedRPM                : 3975
currentFanSpeedLevel              : 1
faultStates.generatedFault        : 0
faultStates.detectedFault         : 0
faultStates.generatedPredictedFail: 0
faultStates.detectedPredictedFail : 0
faultStates.elementSpecificFaults : 0x0
...
Summary:
--------

PCM 0 zones   : OK
PCM 0 fans    : OK

PCM 1 zones   : OK
PCM 1 fans    : OK

overall config: OK
overall zones : OK
overall fans : OK

lastFanSpeedPID: -128
extFanCtrl: DISABLED
CurrentFanSpeedOverride: INVALID

enableCoolingBoost   : FALSE
ddump_events example:

----------------------------------------------------------------------
ddump_events
Diagnostic dump for the Event service
Incoming events: enqueued 1054, retries 0; queue full: 0 dropped.
Invalid events: type mismatch 0, flag undeclared 0.

REGISTERED EVENT LIST
----------------------------------------------------------------
Name              TypeCode   Bitmask       Owner
TEMPERATURE       005        0x00000F0F
                            [0x00000F0F -> ENC_MGT:env_control]

POWER             006        0x00000003
                            [0x00000001 -> HAL:HalProduct]
                            [0x00000003 -> ENC_MGT:PwrMgrEventAC]

STATESAVE         010        0x00000007
                            [0x00000007 -> LOGGING:statesave]

STAT_UPDATE       011        0x00000001
                            [0x00000001 -> HAL:HalPhy]
                            [0x00000001 -> ENC_MGT:statistics]

STAT_RULE_TRIGGER   013        0x00000001
                            [0x00000001 -> ENC_MGT:statistics]

STAT_RESET        014        0x00000001
                            [0x00000001 -> ENC_MGT:statistics]

STAT_COUNTER_RESET   015        0x00000001
                            [0x00000001 -> ENC_MGT:statistics]
ddump_general example:

----------------------------------------------------------------------
ddump_general
Diagnostic dump for the General service.
General Status Service Diagnostic Data
--------------------------------------
Enclosure Info:
Enclosure WWN: 5080020001474E7B
Vendor:        ORACLE
Product Id:    DE2-24P
Part Number:   0977301-03
Fru Id:        0x31
Serial Number: SHM0977301R81PD

Midplane Info:
Product Id:    DE2-24P
Part Number:   0949144-07
Serial Number: BPS0949144G1H1G

SBB Canister Info: (2 canisters)
Canister 0: ( remote ) - Present
Vendor:         ORACLE
Product Id:
Part Number:    0976147-03
Fru Id:         0xCA
Serial Number: IMS0976147G020P
Firmware Ver:   03030020
Bootloader Ver: 0504
VPD Ver:        06
CPLD Ver:       13
Config Data:    01
SoD reset type: Power-on reset
Num Expanders: 1
Expander 0 WWN: 508002000141EB3F
Element index: Ses:0, Type:0, Logical Index:0
Fault Bitmask: 0x00000000
Identify LED : OFF
Fault LED    : OFF

Canister 1: ( local ) - Present
...
ddump_phycounters example:

----------------------------------------------------------------------
ddump_phycounters
Dumps the PHY Counters
PHY Counters
-------------------------------------

PHY 0
Valid            : 1
Link Rate        : 6.0Gbps
Invalid DWORDs   : 0
Disparity errs   : 0
DWORD sync loss : 0
PHY reset failed : 0
PHY change count : 4
...
PHY 23
Valid            : 1
Link Rate        : 6.0Gbps
Invalid DWORDs   : 0
Disparity errs   : 0
DWORD sync loss : 0
PHY reset failed : 0
PHY change count : 2

PHY 24
Valid            : 1
Link Rate        : 6.0Gbps
Invalid DWORDs   : 0
Disparity errs   : 0
DWORD sync loss : 0
PHY reset failed : 0
PHY change count : 142
...
ddump_pwrmgr example:

----------------------------------------------------------------------
ddump_pwrmgr
Diagnostic dump for the Power Manager service
**** Power Manager diagnostic dump ****
HA mode: master
PCM bays: 2
PCMs present: 2
PCMs present at boot: 2
PCM redundancy warning threshold: 2 PCMs
Enclosure drive config: SAS
System Power redundancy bit: set
Total power provided: 1160W
Total power provided (incl surge): 1240W
Current safety overhead: 60W
Enclosure power use: 308W (estimated)
                      187.77W (PCM-measured)
                      273.42W (PCM-measured high watermark)
                      73.31W (output 1 total)
                      114.46W (output 2 total)
Redundancy: Running in minimal redundant mode
Critical Power level: Safe
Full Power (local) : ENABLED
Full Power (remote): ENABLED

PCM index: 0
present: yes
type: PSMI
FRUid: 0x9C
product name: HB-PCM-01-580AC
manufacturer: POWERONE
part number: 7043627
part revision:
serial number: PMW8256200DC8B9
firmware version: 3.11
power state: on
fans self-powered: yes
nominal power rating: 580W
nominal power rating output bitmask: 0x7
surge power rating: 620W
surge power rating output bitmask: 0x3
surge hold time: 10s
AC dropout tolerance: 20ms
standby power: 10W
cooling power: 60W
swap: no
hotSwap(private): yes
hotSwap(public): yes
fault: no
DC output count: 3
output 1 (5.00V)
   voltage: 5.15V
   current: 7.26A
   power:   37.46W
   min voltage: 4.80V
   min surge voltage: 4.80V
   max voltage: 5.19V
   max current: 42.00A
   min current: 0.20A
   max surge current: 42.00A
...
output 3 (5.10V)
   voltage: monitoring unsupported
   current: monitoring unsupported
   power: monitoring unsupported
   min voltage: 4.75V
   min surge voltage: 4.75V
   max voltage: 5.39V
   max current: 2.00A
   min current: 0.00A
   max surge current: 2.00A
combined power: 101.45W
SES info bit: not set
dumpdrives example (slot 3 has a Fault status):

----------------------------------------------------------------------
dumpdrives
Quick view of all drives
Note: layout of table does not reflect physical drive bay locations
______________________________________________________________________
[ 0 online      ][ 1 online      ][ 2 online      ][ 3 online    F ]
______________________________________________________________________
[ 4 online      ][ 5 online      ][ 6 online      ][ 7 online      ]
______________________________________________________________________
[ 8 online      ][ 9 online      ][ 10 online      ][ 11 online      ]
______________________________________________________________________
[ 12 online      ][ 13 online      ][ 14 online      ][ 15 online      ]
______________________________________________________________________
[ 16 online      ][ 17 online      ][ 18 online      ][ 19 online      ]
______________________________________________________________________
[ 20 online      ][ 21 online      ][ 22 online      ][ 23 online      ]
envctrl_fan example:

----------------------------------------------------------------------
envctrl_fan
Environmental Control temperature zones
Fan   Zone    Name           Speed   State
0   2   PCM 0 Fan 0            3900   OK
1   2   PCM 0 Fan 1            3000   OK
2   4   PCM 1 Fan 0            3600   OK
3   4   PCM 1 Fan 1            3000   OK
envctrl_zone example:

----------------------------------------------------------------------
envctrl_zone
Environmental Control temperature zones
Zone   Card    Name            Location   Temperature          Threshold   State
0   Common   Ambient             Mp0:0          23.449       40   OK
1   Common   Midplane            Mp0:1          29.250       50   OK
2   Common   PCM 0 inlet         PCM0:0        29.984       50   OK
3   Common   PCM 0 hotspot       PCM0:1        38.484       65   OK
4   Common   PCM 1 inlet         PCM1:0        29.234       50   OK
5   Common   PCM 1 hotspot       PCM1:1        39.234       65   OK
6   Remote   SBB Canister 0      SBB0:0        39.437       57   OK
7   Local   SBB Canister 1      SBB1:0        45.437       57   OK
getenclosurewwn example:

----------------------------------------------------------------------
getenclosurewwn
Get the enclosure WWN
50 05 0C C1 02 00 00 30
logdump example:

----------------------------------------------------------------------
logdump
Display logged msgs
    0+00:04:18.808; ENC_MGT; power_manager; 02; Clearing PSU Fail (redundant) alarm
    0+00:04:18.808; ENC_MGT; power_manager; 04; PCM 0 fault cleared, assume power restored (580W)
    0+00:04:17.873; ENC_MGT; power_manager; 04; PCM 0 faults indicate loss of power (580W)
    0+00:04:17.873; ENC_MGT; power_manager; 02; Setting FAIL REDUNDANT alarm for PCM 0
    0+00:04:17.872; ENC_MGT; power_manager; 02; PCM 0 DC FAILURE Fault Detected
    0+00:04:17.787; ENC_MGT; env_control; 02; Fan 1 Fault OFF, LED still in use
    0+00:04:17.787; ENC_MGT; env_control; 02; Fan 0 Fault ON
...
    0+00:00:07.008; HAL; hal; 02; EBOD FW: V3.3.0.22
    0+00:00:07.000; HAL; hal; 02; Canister VPD Structure: 0x06
    0+00:00:06.431; HAL; hal; 02; Midplane VPD Structure: 0x0C
    0+00:00:00.409; HAL; hal; 02; Canister CPLD: V13
    0+00:00:00.408; HAL; hal; 02; Canister CPLD POST passed
    0+00:00:00.219; HAL; hal; 02; POWERLOSS:     0+00:16:16.447
logdumpnv example:

----------------------------------------------------------------------
logdumpnv
Display Non-Vol logged msgs
193+00:23:27.271; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x8a, index 0, data - 0x80 0x0 0x10 0x0
193+00:21:17.839; HAL; hal; 04; PHY 27 Link up
193+00:21:17.839; HAL; hal; 04; PHY 26 Link up
193+00:21:17.839; HAL; hal; 04; PHY 25 Link up
193+00:21:17.838; HAL; hal; 04; PHY 24 Link up
193+00:07:38.459; HAL; hal; 04; PHY 27 Link down
193+00:07:38.459; HAL; hal; 04; PHY 26 Link down
...
   45+23:27:03.230; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x19, index 25, data - 0x81 0x4 0xff 0x0
   45+23:27:03.230; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x19, index 24, data - 0x81 0x4 0xff 0x0
   45+23:27:03.179; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0xe, index 0, data - 0x81 0x0 0x0 0x0
   45+23:27:03.140; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x7, index 1, data - 0x81 0x0 0x1 0x0
   45+23:27:03.140; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x7, index 0, data - 0x81 0x0 0x0 0x0
   45+23:27:03.100; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x2, index 1, data - 0x81 0x0 0x0 0x0
   45+23:27:03.100; ENC_MGT; ses; 02; Pg 2 Ctrl: element 0x2, index 0, data - 0x81 0x0 0x0 0x0
*** ALL OLDER MESSAGES HAVE BEEN LOST ***
phydump example:

----------------------------------------------------------------------
phydump
Reports status of SAS PHYs on the expander.
State: E(nabled)
       L(ink up)
       S(pin-up command sent)
Type : SAS/SATA
...
37 PHYs
PHY | Type | Index | Flags | State | Speed | Type | WWN
---------------------------------------------------------------------------
0 |Drive | 0 | | E L S |6.0Gbps| SAS |5000cca0224bb949
1 |Drive   | 1 | | E L S |6.0Gbps| SAS |5000cca0224bc5a5
2 |Drive |    2 | | E L S |6.0Gbps| SAS |5000cca0224b8595
3 |Drive |    3 | | E L S |6.0Gbps| SAS |5000cca0224bba1d
4 |Drive   |    4 | | E L S |6.0Gbps| SAS |5000cca0224b88c9
...
22 |Drive |   22 |       | E L S |6.0Gbps| SAS |5000cca0224b85e1
23 |Drive   |   23 | | E L S |6.0Gbps| SAS |5000cca0166e2231
24 |Port    |    0 |       | E L |6.0Gbps| SAS |528002000158ba50
25 |Port    |    0 | | E L   |6.0Gbps| SAS |528002000158ba50
26 |Port    |    0 | | E L |6.0Gbps| SAS |528002000158ba50
27 |Port    |    0 | | E L |6.0Gbps| SAS |528002000158ba50
28 |Port    |    1 | | E     |       |    |
29 |Port    |    1 | | E     | | |
30 |Port    |    1 | | E | |    |
31 |Port    |    1 | | E | |    |
32 |Port    |    2 | | E | |    |
33 |Port    |    2 | | E | |    |
34 |Port    |    2 | | E | |    |
35 |Port    |    2 | | E     | |    |
36 |Virtual |    0 | | E L   | | SAS |5080020001451d7e
ver example:

----------------------------------------------------------------------
ver
Version information
Package descriptor        : 001e
Package descriptor CRC    : 0x97a7bbe9
Canister firmware         : 3.3.0.32
Canister firmware date    : Jun 5 2014 17:07:15
Canister bootloader       : 5.04
Canister config CRC       : 0x89A26C90
Canister VPD structure    : 0x06
Canister VPD CRC          : 0xD57398ED
Canister CPLD             : 0x13
Canister Power CPLD       : 0x00176CF8
Canister chip             : 0x80050002
Canister SDK              : 3.06.01-B028
Midplane VPD structure    : 0x0C
Midplane VPD CRC          : 0xFA5942E4
Midplane CPLD             : 0x18
PCM 0 firmware            : 3.11
PCM 1 firmware            : 3.11
PCM 0 VPD structure       : 0x03
PCM 1 VPD structure       : 0x03
PCM 0 VPD CRC             : 0x6B58AD13
PCM 1 VPD CRC             : 0x6B58AD13
Battery 0 firmware        : Not present
Battery 1 firmware        : Not present

Attachments

This solution has no attachment