"ASR: HALRT-02024: Cache Vault degradation." logged by one or more Exadata X6 Compute (DB) nodes after Disk Expansion Kits are installed.

Asset ID:	1-72-2284109.1
Update Date:	2017-08-06
Keywords:

Solution Type Problem Resolution Sure

Solution 2284109.1 : "ASR: HALRT-02024: Cache Vault degradation." logged by one or more Exadata X6 Compute (DB) nodes after Disk Expansion Kits are installed.

Applies to:

Exadata X6-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

After installing a disk expansion kit on one or more Exadata X6 nodes you may see messages similar to the below logged in the alerthistory

2_1 2017-06-26T09:57:07-07:00 warning "A disk expansion kit was installed. The additional physical drives were automatically added to the existing RAID5 configuration, and reconst
ruction of the corresponding virtual drive was automatically started."
2_2 2017-06-26T16:36:38-07:00 clear "Virtual drive reconstruction due to disk expansion was completed."
3_1 2017-06-26T09:57:08-07:00 critical "All disk drives have been placed in WriteThrough caching mode. Disk write performance may be reduced. The flash drives are not affected. Ba
ttery Serial Number : 21974 Battery Type : cvpm02 Battery Temperature : 27 C Pack Energy : 243 Joule Ambient Temperature : 25 C"
3_2 2017-06-26T16:34:41-07:00 clear "All disk drives are in WriteBack caching mode. Battery Serial Number : 21974 Battery Type : cvpm02 Battery Temperature : 27 C
Pack Energy : 243 Joule Ambient Temperature : 25 C"

If ASR is enabled an "ASR: HALRT-02024: Cache Vault degradation." may be logged by one or more nodes at the same/roughly same time.

Changes

Installation of an Exadata DB Node disk expansion kit

Cause

Cache mode is temporarily changed to WriteThrough (WT) during the duration of the Virtual Drive (VD) reconstruction. If ASR is not disabled first this will cause one or more "Cache Vault Degraded "ASR's to be logged for each node that the expansion kit was installed on.

Solution

It is normal and expected for the Cache mode to be temporarily changed to WriteThrough (WT) caching mode while the Virtual Drive Reconstruction is performed which can take anywhere from a few hours to 6 or more depending on the configuration. During this time db node disk sub-system performance will be impacted, but is generally not serious. Once the RAID volume finishes reconstructing, the SAS controller should automatically change the cache mode back to WriteBack (WB). This will also be logged in the alerthistory.

ALWAYS DISABLE ASR, before having this or any other service/upgrade/installation/etc performed as follows:

1. Set the assets into "Maintenance Mode" (Ops Center) or create a "Blackout" (OEM Cloud Control).

2. Leave the asset in "Maintenance mode"/"Blackout" until the BBU charging is completed and reported normal by alerthistory or "MegaCLi -GetBbuStatus -a0"

This will prevent false ASR's from being generated due to the planned activities being performed.

Refer to the "Solution" Section of "ASR: HALRT-02013: HDD Disk Controller Battery Degraded." during Preventive Maintenance (Doc ID 2189092.1)" for specific instructions, or consult with the Oracle Global Support for more information and advice or help.

Legitimate Degraded Cache Vault/SuperCap faults are rare. ASR's are logged on a per node basis however multiple similar faults occurring on different DB nodes in the same rack will be identified by back-end processing and the additional SR's closed and collapsed into 1 SR for engineer review. Since legitimate faults of this type are rare, if you don't find an obvious fault, or see disk expansion upgrade messages logged, it will be a good idea to check the service history for the sibling assets in MOS to see any others having a problem.

If you have reviewed and confirmed no faulty hardware is the cause, but an upgrade was taking place, then advise the CU as noted above and request SR closure.

References

<NOTE:761868.1> - Oracle Exadata Diagnostic Information required for Disk Failures and some other Hardware issues
<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices
<NOTE:2189092.1> - "ASR: HALRT-02013: HDD Disk Controller Battery Degraded." during Preventive Maintenance

Attachments

This solution has no attachment