A number of Brick RAID controller reboots are due to Cache Parity Errors. What is a Cache Parity Error ?

The RAID Controller uses an Intel X-Scale microprocessor, which has internal cache memory. This error indicates that the parity error is in the cache memory inside the X-Scale processor. Extensive testing has indicated that these highly intermittent errors are not the result of heat or voltage conditions at the chip. The X-scale microprocessor, on encountering this error must reset itself, which causes the RAID Controller to fail over to the companion (which continues to provide data access), reboot itself, then fail back.  

The typical failure is that a RAID Controller Unit (CU) will experience no more than one of these events in its service life. There is no benefit to replacing a RAID CU with a single instance, as the new RC would have the same probability of failure.  
If a RAID CU experiences multiple failures of this type, it is replaced.

The Serial Number of the individual RAID CU is kept in an internal Oracle database, to track any repeat failures.

This section is Oracle Internal Only

Customer Support Facing Description (Internal Only)

Description:

The issue is that there is a product defect in an Intel chip within the RAID Controller such that when certain conditions are met, the RAID Controller faults and is reset by the companion Controller.  This is not a defect specific to the Axiom, but rather a defect in the OEM part (Intel 80321 X-Scale microprocessor) Over the course of product development with the Intel 80321 (X-Scale microprocessor) and now the Intel 80331, Pillar Data Systems has seen an occasional condition known as a “Cache Parity Error - FSR:0x408”.  Most recent conditions reports have also been “Data Abort: Data Cache Parity Error Exception - FS:0x408”.  Pillar Data Systems has been unable to reproduce this issue on the same boards during on-going FA (Failure Analysis).

Containment:

This error occurs inside the X-Scale microprocessor. There is a different error code for an External Data Abort, so this error is contained in the X-Scale. This corresponds with our experimental data, that we cannot induce a 408 type error by margining the X-Scale’s external PCI-X bus or the CPU voltage.  We have reported the problem to Intel, but they have, thus far, not been able to duplicate the problem.  Given the infrequent nature of these errors and the inability to generate a reoccurrence, we are working to generate a test code version that will loop and trap errors, in order to generate enough information for Intel to debug the problem at the chip level.

Solution:

There is currently no firmware work-around at the RAID controller level to address this problem. The CPU core inside the X-Scale has received bad data and must abort and reset. Until this problem is fixed at chip level, we must assume that we must accommodate an occasional failure by performing a fail-over and a fail-back and keep on running while logging that the controller has experienced a fault.

Recommendations:

We have entered the SSN of this RAID Controller in our tracking database to reflect this instance of the Cache Parity Error, should this Controller encounter this condition again Pillar Support will make arrangements for replacement of the Controller.