Pillar Axiom: Multiple attempts to replace a Brick component fail using Guided Maintenance in R5

Asset ID:	1-72-1606390.1
Update Date:	2017-05-23
Keywords:

Solution Type Problem Resolution Sure

Solution 1606390.1 : Pillar Axiom: Multiple attempts to replace a Brick component fail using Guided Maintenance in R5

Applies to:

Pillar Axiom 500 Storage System - Version All Versions to All Versions [Release All Releases]
Pillar Axiom 600 Storage System - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Multiple attempts to replace a Brick component using Guided Maintenance fail with an error message FRU_GM_ALREADY_IN_PROGRESS

Below is what the failed task looks like seen in the events.xml file

        <SystemEventInformation>
          <EventType>TASK_FAILED</EventType>
          <Severity>WARNING</Severity>
          <Category>AUDIT</Category>
          <Time>2013-11-14T20:03:10.022</Time>
          <ComponentIdentity>
            <Guid>4130303431333042A13F11FA4A49DEE8</Guid>
          </ComponentIdentity>
          <AuthenticatedAccountIdentity>
            <Id>4130303431333042A14000008AFC79FD</Id>
            <Fqn>/pillar</Fqn>
          </AuthenticatedAccountIdentity>
          <AuthenticatedAccountName>pillar</AuthenticatedAccountName>
          <EventParameterList>
            <ParameterName>EventParameters.TaskFailed.pmiError.1</ParameterName>
            <ParameterValue>CM_FRU_GM_ALREADY_IN_PROGRESS</ParameterValue>
          </EventParameterList>
          <EventParameterList>
            <ParameterName>EventParameters.TaskFailed.taskName</ParameterName>
            <ParameterValue>PrepareBrickFruReplacement/1267075/pillar</ParameterValue>
          </EventParameterList>
          <EventParameterList>
            <ParameterName>EventParameters.TaskFailed.csiError.1</ParameterName>
            <ParameterValue>FRU_GM_ALREADY_IN_PROGRESS</ParameterValue>
          </EventParameterList>
          <EventGuid>00190F0B6A3B000611FA51664041587D</EventGuid>
        </SystemEventInformation>

Similar task failed message can be seen in the GUI event log or System Alerts screen.

Cause

The error occurrs because a previous Guided Maintenance task was started and never completed, the ConMan software component can only have one FRU stored as prepped for replacement. The software won't allow the user to run more than one Guided Maintenance task without successful completion.

If any Brick component is prepared for replacement the user may receive multiple System Alerts indicating a FRU should be replaced. This can be viewed in the GUI by navigating to "Monitor" tab and clicking on the "System Alerts" link. It can also be viewed by running the following axiomcli command:

axiomcli system_alert -list -details

Solution

A-Admin Alerts:

There may be existing System Alerts instructing the user to remove (or abort) a Brick component replacement. If the alerts exist, use the alerts to cancel (or abort) the replacement.

If another System Alert exists of a task failure aborting the FRU replacement, delete the new failed task alert and continue to process any alert telling you to replace a component.

If the user still cannot cancel or abort a FRU replacement started by Guided Maintenance then the user will have to cancel the replacement using the pcli utility.

B- CLI:

If the System Alerts have been deleted or you cannot cancel (or abort) a FRU replacement in the GUI, then you will need to use the pcli to find the outstanding task and replace it.

The pcli must be used as there is no obvious axiomcli request to show outstanding GM requests.

You can use the administrator or pillar user account to execute each of the commands

1- List all active alerts with

axiomcli login -u administrator -p <Password> <axiom IP>
axiomcli system_alert -list -details

2- Use pcli to check which FRU is the one that is currently prepared for replacement.

pcli sub -u administrator -H <axiom IP> GetFruPreparedForReplacement

3- Use pcli to cancel that replacement

pcli sub -u administrator -H <axiom IP> AbortBrickFruReplacement Identity.Fqn=<related FQN> FruType=<related FRU> FruNumber=<related FRU Number>

4- Use pcli to make sure there are no other GM requests

pcli sub -u administrator -H <axiom IP> GetFruPreparedForReplacement

5- Log back into the GUI and use Guided Maintenance to replace the desired Brick component.

Example:

Listing active GM FRU Replacement task:

pcli sub -u administrator -H <axiom IP> GetFruPreparedForReplacement

The output should look like the following:

Message
Response
   CorrelationID: 1386346047
   BeginStreamResponse
     TaskGuid: 4130303431333042A13FC5FF0A2256CC
     TaskFqn: /GetFruPreparedForReplacement/2614380/pillar
Message
Response
   CorrelationID: 1386346047
   GetFruPreparedForReplacementResponse
     PreparedFruInformation
       Identity
         Id: 200C000B083A7ECB
         Fqn: /BRICK-001
       BrickFruType: RAID_CONTROLLER
       SlammerFruType: NOT_VALID
       FruNumber: 1
Message
Response
   CorrelationID: 1386346047
   EndStreamResponse
     TaskGuid: 4130303431333042A13FC5FF0A2256CC
     TaskFqn: /GetFruPreparedForReplacement/2614380/pillar

Then use the pcli to cancel the replacement, substituting the values from the GetFruPreparedForReplacement request.

pcli sub -u administrator -H <axiom IP> AbortBrickFruReplacement Identity.Fqn=/BRICK-001 FruType=RAID_CONTROLLER FruNumber=1

The output should look like the following:

CorrelationID: 1386347614
   BeginStreamResponse
     TaskGuid: 4130303431333042A13FC76C0C96F5E4
     TaskFqn: /AbortBrickFruReplacement/2615772/pillar
Message
Response
   CorrelationID: 1386347614
   EndStreamResponse
     TaskGuid: 4130303431333042A13FC76C0C96F5E4
     TaskFqn: /AbortBrickFruReplacement/2615772/pillar

Once this is done the user should be able to use Guided Maintenance in the GUI to replace the desired Brick component.

If this issue is seen while replacing a Slammer Component using Guided Maintenance, please see KM Doc ID <Document 1529154.1>

References

<NOTE:1529154.1> - Pillar Axiom: Multiple attempts to Replace a Slammer component fails using Guided Maintenance in R5

Attachments

This solution has no attachment