Pillar Axiom: How to Interpret a Pitman Output on AxiomOne R5.X Systems

Asset ID:	1-71-1906880.1
Update Date:	2018-01-08
Keywords:

Solution Type Technical Instruction Sure

Solution 1906880.1 : Pillar Axiom: How to Interpret a Pitman Output on AxiomOne R5.X Systems

Applies to:

Pillar Axiom 600 Storage System - Version All Versions and later
Pillar Axiom 500 Storage System - Version All Versions and later
Information in this document applies to any platform.

Goal

The intent of this document is to provide some basic guidance to interpret a PITMan collection. For the instructions to run PITMan, please refer to <Document 1473515.1> Pillar Axiom: How to Run Pitman on AxiomOne R5.X Systems.

Should you still have questions after reading this document, do not hesitate to use My Oracle Support Communities. Communities put you in touch with industry professionals like yourself. They are monitored by Oracle support engineers, so you can expect reliable and correct answers. Ask questions and see what others are asking about in the Disk Storage Pillar Axiom System Community.

Solution

Each connection will be listed and with the number and kinds of errors listed below it. Here below is a sample output. The important values to note are the crcErrRate, crcCnt, ItwCnt, and dstbCnt fields. These would be indicative of errors on that path.

============[IN:FC0]=========
* (0x2108000b080466e2) SLAMMER-01:CU0
* PIM_SOC422
*   (perGB)crcErrRate=0
*            crcCnt=0
*            ItwCnt=0
*           dstbCnt=0
* rdBytes=8191934584
* wrBytes=8199160832
* (sample: cnt=27 totalSec=299)
============[OUT:FC0]========
|
============[IN:FC2]=========
* (0x2309000b080466ea) SLAMMER-01:CU1
* PIM_SOC422
* (perGB)crcErrRate=0
*            crcCnt=0
*            ItwCnt=0
*           dstbCnt=0
* rdBytes=8151931904
* wrBytes=8145507456
* (sample: cnt=27 totalSec=299)
============[OUT:FC2]========
|
============[IN:FC0]=========
* (0x2109000b080466ea) SLAMMER-01:CU1
* PIM_SOC422
* (perGB)crcErrRate=0
*            crcCnt=0
*            ItwCnt=0
*           dstbCnt=0
* rdBytes=8204316672
* wrBytes=8221593728
* (sample: cnt=27 totalSec=299)
============[OUT:FC0]========
|
============[IN:FC2]=========
* (0x2308000b080466e2) SLAMMER-01:CU0
* PIM_SOC422
* (perGB)crcErrRate=0
*            crcCnt=0
*            ItwCnt=0
*           dstbCnt=0
* rdBytes=7711291452
* wrBytes=7690093568
* (sample: cnt=27 totalSec=299)
============[OUT:FC2]========

As the file is quite lengthy, it is recommended to search the file for the respective fields to find errors quickly. Below is a sample command that can be used:

grep -i -n -e Ticket -e crc -e itw -e dstb <filename.txt> | grep -v "=0"

Example:

13803:Thu May 23 09:13:05 2013 (Ticket ffffffff) Diagnose Physical Error Detected
13820:                   crcCnt=1
13821:                   ItwCnt=2
13826:Thu May 23 09:13:27 2013 (Ticket ffffffff) Map Save
14163:              *               crcCnt=1
14164:              *               ItwCnt=1
14471:Thu May 23 09:13:27 2013 (Ticket ffffffff) Diagnose State Change

It is important to note the timestamp as the pitman result may contain older pitman session. If there was errors, make sure that they relates to the recently performed session.

A clear session would look like this:

3322: statsSessTicket : 11
23324:Fri May 24 11:28:31 2013 (Ticket ffffffff) CLI Command
23331:Fri May 24 11:28:35 2013 (Ticket ffffffff) Map Save
23976:Fri May 24 11:28:35 2013 (Ticket ffffffff) Map Save
24621:Fri May 24 11:28:35 2013 (Ticket ffffffff) Map Save
24702:Fri May 24 11:28:35 2013 (Ticket ffffffff) Push Request
24709:Fri May 24 11:32:57 2013 (Ticket ffffffff) Stats Save
24778:Fri May 24 11:32:57 2013 (Ticket ffffffff) Stats Save
24847:Fri May 24 11:32:57 2013 (Ticket ffffffff) Map Save
25492:Fri May 24 11:32:57 2013 (Ticket ffffffff) Map Save
26137:Fri May 24 11:32:57 2013 (Ticket ffffffff) Map Save
26218:Fri May 24 11:32:58 2013 (Ticket ffffffff) Push Request
26225:Fri May 24 11:37:20 2013 (Ticket ffffffff) Stats Save
26294:Fri May 24 11:37:20 2013 (Ticket ffffffff) Stats Save
26363:Fri May 24 11:37:20 2013 (Ticket ffffffff) Map Save
27008:Fri May 24 11:37:20 2013 (Ticket ffffffff) Map Save
27653:Fri May 24 11:37:20 2013 (Ticket ffffffff) Map Save
27734:Fri May 24 11:37:20 2013 (Ticket ffffffff) Push Request
27741:Fri May 24 11:41:42 2013 (Ticket ffffffff) Stats Save
27810:Fri May 24 11:41:42 2013 (Ticket ffffffff) Stats Save
27879:Fri May 24 11:41:42 2013 (Ticket ffffffff) Map Save
28524:Fri May 24 11:41:42 2013 (Ticket ffffffff) Map Save
29169:Fri May 24 11:41:42 2013 (Ticket ffffffff) Map Save
29250:Fri May 24 11:41:42 2013 (Ticket ffffffff) Push Request
29257:Fri May 24 11:46:05 2013 (Ticket ffffffff) Stats Save
29326:Fri May 24 11:46:05 2013 (Ticket ffffffff) Stats Save
29395:Fri May 24 11:46:05 2013 (Ticket ffffffff) Map Save
30040:Fri May 24 11:46:05 2013 (Ticket ffffffff) Map Save
30685:Fri May 24 11:46:05 2013 (Ticket ffffffff) Map Save
30766:Fri May 24 11:46:05 2013 (Ticket ffffffff) Push Request
30773:Fri May 24 11:50:26 2013 (Ticket ffffffff) Stats Save
30842:Fri May 24 11:50:26 2013 (Ticket ffffffff) Stats Save
30911:Fri May 24 11:50:26 2013 (Ticket ffffffff) Map Save
31556:Fri May 24 11:50:26 2013 (Ticket ffffffff) Map Save
32201:Fri May 24 11:50:26 2013 (Ticket ffffffff) Map Save
32282:Fri May 24 11:50:26 2013 (Ticket ffffffff) Push Request
32289:Fri May 24 11:50:27 2013 (Ticket ffffffff) Stats Stop
32295:Fri May 24 11:55:08 2013 (Ticket ffffffff) CLI Command

Troubleshooting tips for PI cabling

**If there is any other result then Zero Errors, reseat the suspect cabling and re-run the 5 minute test and review again, if errors are still present, contact Support for assistance in hardware isolation **
If the errors detected are at the head of a brick string, all cables within that string should be reseated according to the appropriate wiring guide and utilizing the current cabling best practices.
Be careful to always use the Physical Position of each Slammer and Brick and not the logical name from the GUI when installing or checking cables. The name as displayed in the GUI is a user settable logical name only - it has no mandatory relationship to the hardware name. The physical or hardware name is based entirely on the Physical position of the hardware in the cabinet, as shown in the Wiring Diagrams, and has no mandatory relationship with the name displayed in the GUI or system logs. You may be asked to create a mapping of the physical name to the GUI name.
Do not use Tie Wraps to secure the cables. Velcro straps are the only recommended cable management. Do not tighten the Velcro tightly.
Minimum Bend Radius is 2 inches (4 inches diameter) for the fibre channel cables. Be careful when working with the cables to never bend a cable to a radius smaller than 2 inches.
Do NOT pull the cables taut. Leave a large curve in all installed cables to prevent damage to the cable or connectors.
Do NOT route Power Cables over the Fibre Channel Cables. The power cables are heavy enough to cause damage to the Fibre Channel Cables if the power cables rest on the fibre cables. Do not rest any bundles of cables on any Fibre Channel cable.
Do NOT support loops of the Fibre Channel cables by their own weight. No more than half a meter of Fibre Channel cable should ever be supported by the weight of that cable.
Make sure there are no twists or knots in the Fibre Channel or Ethernet cables.
Do NOT support the Slammer to Brick or Brick to Brick Fibre Channel Cables with the Brick RAID Controller Crossover Cables. This will cause connection problems for the Crossover Cables.
Check the Cabling at least Twice. It is strongly recommended that at least two people perform a full cabling audit to make sure that no cabling errors are introduced. The best audit is to have one person provide the connections for one end of each cable, then have someone trace that cable and provide the connection for the other - to be checked against the wiring diagram.

References

<NOTE:1473492.1> - Pillar Axiom: How to interpret a PITMAN output from a PSG_PITMAN_EVENT_DIAGNOSTIC_RESULTS_AVAILABLE Callhome event
<NOTE:1473515.1> - Pillar Axiom: How to Run Pitman on AxiomOne R5.X Systems

Attachments

This solution has no attachment