FS System: How to Collect the System-Wide Diagnostic Dump From Drive Enclosure (DE2-24C or DE2-24P) IO Module (IOM)

Asset ID:	1-71-1954866.1
Update Date:	2018-02-17
Keywords:

Solution Type Technical Instruction Sure

Solution 1954866.1 : FS System: How to Collect the System-Wide Diagnostic Dump From Drive Enclosure (DE2-24C or DE2-24P) IO Module (IOM)

Applies to:

Oracle FS1-2 Flash Storage System - Version All Versions and later
Information in this document applies to any platform.

Goal

The intent of this document is to provide the instructions to collect logs from the Oracle Flash Storage (FS) Drive Enclosure (DE) IOM cards. The instructions in this document are specific to the DE2 model.

There are various methods to collect these logs:

Using the "fscli" command line.
Using the "pcli" command line.

Note: Whenever you collect logs from the IOM cards, it is important to collect from both IOMs 0 and 1 because we need to have a global view for the DE.

Please refer to Document 2188416.1 FS System: Content details of an IO Module (IOM) ddump File for content details of the ddump file.

The above methods are the preferred one. However, if the software stack does not recognize the DE2 ID/FQN due to the persistence not being loaded (refer to the unpublished Bug 20719463), you can use remotely the next method below.

Using the RAID console via ssh.
This method is useful in case the software stack does not recognize the DE ID/FQN due to the issue described above.
Using the serial port on the DE2.
This method requires a Field Engineer to go onsite.

If you use the serial port method, you can use the Axiom Brick console cable (Part Number 7064484) to connect to the DE2 serial port.

Note: The user should not deviate from the instructions in this document and/or execute other commands. The execution of any other commands, unless instructed by the Oracle engineering, can lead to serious production impact for the customers or even lead to data integrity issues.

Solution

Note: FS1-2 Systems running software versions before R6.1.5 will need to contact Oracle Support to gather this information.

Releases below 060105-033400, you may encounter the error "UNSATISFIED_REQUEST_PMI_COMMUNICATION_ERROR" with one of the IOM. In this case, the only option is via the serial port.

Note: FS1-2 on releases below 6.2.15 must collect ddumps manually following the instructions in this document.

FS1-2 on releases from 6.2.11 to 6.2.14 do include ddumps in the TDS brix folder but the ddump files are incomplete when collected with manual or event driven logs, Bug 26249151 addresses this issue fixed in release 6.2.15.

A good ddump collection will have about 30,000 lines, but the automatic ones collected on releases below 6.2.15 only contain about 2000 lines.

Using the "fscli" command

Note: This procedure requires the use of the "pillar" account. Refer to Document 2046703.1 FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System for the default password.

Refer to Document 1991938.1 FS System: How to Obtain and Install the fscli Tool Software if you have not yet installed fscli.
Login with the user "pillar":

# fscli login -u pillar -oracleFs <FS1 IP address>
Collect the IOM dump:

# fscli enclosure -download -enclosure <enclosure id> -iom [0 | 1] -ddump <target filename> -o text

Example:

# ./fscli enclosure -download -enclosure /ENCLOSURE-09 -iom 0 -ddump dump_iom_0.txt -o text
Command Succeeded
#

Using the "pcli" command

Execute the following command with the user account "pillar" (the tool pcli is included in the bundle with fscli):

# pcli sub -u pillar -H <FS1 IP> DownloadEbodControllerDdump EnclosureIdentity.Fqn=<enclosure number> EBODControllerNumber=[0 | 1]

Note: This will generate a text file with the name reported in the result of the command.

Example with the ENCLOSURE-02 and IOM 1:

# ./pcli sub -u pillar -H 111.222.333.444 DownloadEbodControllerDdump EnclosureIdentity.Fqn=/ENCLOSURE-02 EBODControllerNumber=1
Password:
Message
Response
    CorrelationID: 1420798714
    BeginStreamResponse
      TaskGuid: 414B303031323639A13F84E50E5DE7D2
      TaskFqn: /DownloadEbodControllerDdump/86559/pillar
Message
Response
    CorrelationID: 1420798714
    DownloadEbodControllerDdumpResponse
      EnclosureIdentity
        Id: 5080020001475241
        Fqn: /ENCLOSURE-02
      EBODControllerNumber: 1
      DdumpFile: csik91OFj<=== Target file name "csik91OFj" located in the current directory
Message
Response
    CorrelationID: 1420798714
    EndStreamResponse
      TaskGuid: 414B303031323639A13F84E50E5DE7D2
      TaskFqn: /DownloadEbodControllerDdump/86559/pillar

#

NOTE: When executing the fscli/pcli commands on a Windows 7 or Windows 10 host the ddump file may not exist in the current directory where the command is being executed, instead it may be located under %User%\AppData\Local\VirtualStore\<path to exe> directory. This is part of Windows Security.

Using the RAID console via ssh

First, you will need to determine from which DE2 IOM you need to extract the ddump. The IOM 0 is connected to controller 1 and the IOM 1 is connected to controller 2. If for example you need to collect the ddump from the IOM 1, you will need to use the controller 2 in the following procedure.

Connect to the DE2 RAID console via the controller 1 or 2 as explained above. Refer to Document 1991213.1 FS System: How to SSH to Drive Enclosure RAID Console.
Start a capture session to log the outputs to a file.
Use the steps in the Document 1991213.1 to determine what is the target number for the DE2 from which you want to collect the ddump.
Enter the SES submenu:

FCRoot> SES
SES>
Execute the ddump command as below:

SES> ddump [target number extracted from the step 3]

Example:

SES> ddump 2
When finished, you can type CTRL-C to exit the raidconsole

Using the serial port on the DE2

Note: At the present time there is no part number for the DE2 IOM console cable on the FS1 component list. However you can use the Axiom Brick console cable or make your own cable following the wiring below.

Console cable with its wiring diagram:

Stereo to DB9 cable Stereo to DB9 wiring

Instructions to collect the logs:

Connect the console cable to the IOM serial port:

DE2-24P

DE2-24C
Use a serial port program (ie: putty) to open a session with the following parameters:

Note: Make sure to start a log of the session in order to capture all the output.
- Baud rate (bits per second): 115200
- Data bits: 8
- Parity: None
- Stop bits: 1
- Flow control: None
Press enter. You should get a prompt.
Example:

11+16:16:49.762 M1 GEM>
Execute the command "ddump" in order to collect all the diagnostic data from the IOM:
Example:

11+16:16:50.033 M1 GEM> ddump

----------------------------------------------------------------------
batt_dump_regs
Diagnostic dump of the battery registers
Battery 0 - Not Present
Battery 1 - Not Present

High Watermark counters:
AcquireLockAttempts        : 0
ReadRegisterAttempts       : 0
WriteRegisterAttempts      : 0
ReadRegisterBlockAttempts : 0
WriteRegisterBlockAttempts : 0
ResetDeviceAttempts        : 0
ReadPageDataAttempts       : 0
WritePageDataAttempts      : 0

----------------------------------------------------------------------
batt_voltage
Display voltages for each battery
Battery 0 - Not Present

Battery 1 - Not Present

----------------------------------------------------------------------
battstats
Report the battery statistics for a given battery

<this example is truncated here>
Close the serial port session.

Attachments

This solution has no attachment