FS System: How to Handle a "Save Trace Event" Auto-Service Request

Asset ID:	1-72-2126081.1
Update Date:	2016-09-22
Keywords:

Solution Type Problem Resolution Sure

Solution 2126081.1 : FS System: How to Handle a "Save Trace Event" Auto-Service Request

Applies to:

Oracle FS1-2 Flash Storage System - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Your FS1-2 created an Auto-Service Request (ASR) for "Save Trace Event" or "RAID_EVENT_SAVE_TRACE"

Cause

A Save Trace Event occurs when the FS1 detects that "something" that may be interesting happened on a Drive Enclosure that we may want logs for. The FS1 collects and uploads RAID_EVENT_SAVE_TRACE logs in response. This was instituted in order to have logs collected as close to an event as possible, because the RAID logs wrap very quickly.

The Save Trace Event in itself is not an indication of a problem, it is just a mechanism to collect logs when a Drive Enclosure event occurs, to collect the logs as close to the event as possible.

Solution

If the Save Trace Event logs were captured in relation to an actual problem, the FS1 should have created an additional ASR for something like ENCLOSURE_DRIVE_STATE_CHANGE and that Service Request would be worked appropriately. Normally in this case, we would not even see the Save Trace Event ASR being created.

In that case, you can safely ignore the Save Trace Event ASR if one is created.

In the case where a Save Trace Event ASR is created with no corresponding ENCLOSURE_DRIVE_STATE_CHANGE (or similar) ASR, the Service Request will be routed out of the automation system to a TSC Engineer to determine the reason why the ASR was created.

Previous Bugs have found that there are instances where a RAID_EVENT_SAVE_TRACE is generated without a corresponding Enclosure problem SR - when there really was a problem on an Enclosure. If one of those cases is encountered, please update that information into this KM Doc.

You might be able to find information about what caused the Save Trace Event by searching in the pilot server.log.YYMMDDHHMMSS for RAID_EVENT_SAVE_TRACE and reviewing the entries just prior to that one.

In this instance, the Save Trace Event was generated as a result of a user-initiated verifyDataConsistency job:

INFO 2016-04-04 15:10:54,962 com.pillardata.server.csi.frontcontrol.AuthenticationFilter.
populateAndSendAuthenticationResponse(AuthenticationFilter.java:701) ChannelMsgWorker-[id: 0xeb082f5e, /172.17.0.25:32100 => /172.19.102.103: 26012]-1 - Server event generated: SystemEventImpl[
m_type=LOGIN_SUCCEEDED
m_pmiEventType=-2147483648
m_category=AUDIT
m_severity=INFORMATIONAL
m_visibility=true
m_serverReceivedTime=1459782654962
m_componentSpecifiedTime=1459782654961
m_triggeredId=A140000000000000
m_triggeredFqn="/administrator" <<<<<<<<<<<<------------------- user "administrator" logged in
m_eventPmiXmlData="<EventParameters><IpAddress>/172.17.0.25</IpAddress></EventParameters>"
m_authenticatedAccountId=A140000000000000
m_authenticatedAccountFqn="/administrator"
m_sourceNodeId=<null>

INFO 2016-04-04 15:11:38,276 com.pillardata.server.domain.operation.AbstractOperation.run(AbstractOperation.java:139) StandardOperationPool-6 - Executing Operation: PerformVerifyDataConsistencyOperation[
Operation Class="com.pillardata.server.domain.model.hardware.enclosure.PerformVerifyDataConsistencyOperation@99f9080"
Operation Name="PerformVerifyDataConsistency"
Fqn=/PerformVerifyDataConsistency/306804/administrator <<<<<<<<<<-------------------- administrator requests the verifyDataConsistency job
Id=A13F4128EE422613
LockHolderIdentity=A13F4128EE422613
ExecutionClassification=USER_CREATE_MODIFY_DELETE
LockWatcher=<null>
LockHolderType=ADMIN

INFO 2016-04-04 15:11:38,278 com.pillardata.pmi.net.InfoLogger.logSend(InfoLogger.java:72) PMICommandProcessor - com.pillardata.pmi.net.Message@ 69942736[Header=MessageHeader[
revision=0
messageID=0x31d343b63a
transactionID=0
sourceNodeId=2008000101000000
sourceComponent="PDS_COMP_PACMAN(0x1f)"
destNodeId=2008000101000001
destComponent="PDS_COMP_CM(0x1c)"
flags=136
type="PDS_MSG_TYPE_SUPER_CMD(0x5)"
command="CM_MSG_ENCLOSURE_VERIFY_DATA_CONSISTENCY(0x1c01e6)"
result="PMI_EOK(0x0)"
...
driveGroupId=A2860CED0070EE94
priority="CM_VERIFY_PRIORITY_LOW(0x1)"

INFO 2016-04-04 15:11:38,283 com.pillardata.pmi.net.InfoLogger.logReceive(InfoLogger.java:96) JniActionReceiverSubsystem - com.pillardata.pmi.net. Message@565c2c31[Header=MessageHeader[
revision=131073
messageID=0x0
transactionID=1688
sourceNodeId=508002000158BDD0
sourceComponent="PDS_COMP_AGENT(0x3)"
destNodeId=2008000101000000
destComponent="PDS_COMP_PACMAN(0x1f)"
flags=0
type="PDS_MSG_TYPE_EVENT(0x3)"
command="RAID_EVENT_SAVE_TRACE(0x140029)"

References

<BUG:23052392> - RAID_EVENT_SAVE_TRACE

Attachments

This solution has no attachment