Exadata X3-8 Hardware
 
Exadata X4-8 Hardware
 
Oracle Exadata Storage Server Software
 
Exadata X3-2 Hardware
 
Exadata X4-2 Hardware
 
Oracle SuperCluster Specific Software
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

In this Document

Description

Occurrence

Pre-requisite Conditions for Bug 22909764

How to check if your storage servers are susceptible to bug 22909764

Symptoms

Assessing the Risk of Hitting Critical Issue EX28

Workaround

Patches

History

References

Applies to:

Description

Due to <bug 22909764>, on X4 and X3 storage servers (with X4-2, X4-8, X3-2, and X3-8 Exadata Database Machines) running Exadata 12.1.2.2.0, 12.1.2.2.1, or 12.1.2.3.0, when Exadata Smart Flash Cache Compression is enabled, one or more flash drives may fail on multiple storage servers, leading to a potential for data loss if flash cache is configured write-back, or reduced performance if flash cache is configured write-through.

Flash Cache Compression is disabled by default. Only storage servers where this feature has been explicitly enabled will be affected. An Oracle Advanced Compression option license is required in order to use Flash Cache Compression.

Occurrence

Pre-requisite Conditions for Bug 22909764

<Bug 22909764> may occur when all of the following conditions are met:

1. Exadata software version is 12.1.2.2.0, 12.1.2.2.1, or 12.1.2.3.0.
2. Exadata Smart Flash Cache Compression is enabled.  Note that Exadata Smart Flash Cache Compression is supported with X3 hardware (SUN FIRE X4270 M3) and X4 hardware (SUN SERVER X4-2L) only.
3. Flash controller firmware version is 13.05.10.00 or 13.05.11.00 (the default versions shipped with 12.1.2.2.0, 12.1.2.2.1, and 12.1.2.3.0).

How to check if your storage servers are susceptible to bug 22909764

Run the following checks on all storage servers.  If one or more storage servers meet all of the conditions then review the Workaround and Patches sections below.

1. Run the following imageinfo command to determine the current Exadata software version:

If current version is 12.1.2.2.0, 12.1.2.2.1, or 12.1.2.3.0, then proceed to the next check.

2. Run the following CellCLI command to determine if Flash Cache Compression is enabled:

A value of TRUE indicates Flash Cache Compression is enabled, then proceed to the next step.  A value of FALSE or no value at all indicates Flash Cache Compression is disabled.  Note that Flash cache compression requires licensing the Oracle Database Advanced Compression option.

3. As the root user, run the following command to check for flash controller firmware version:

If flash controller firmware version is 13.05.11.00 (for X4 systems) or 13.05.10.00 (for X3 systems), then proceed to the Workaround and Patches sections for further action.

Symptoms

Multiple flash drives on more than one flash card, potentially across multiple storage servers, fail at the same time with one of the following statuses:

• warning - peer failure
• warning - poor performance
• warning - poor performance, peer failure

Multiple failed flash drives may lead to data loss if flash cache is configured write-back, or reduced performance if flash cache is configured write-through.

If you believe that your system is currently exhibiting the symptoms described, please contact Oracle Support.  Note that installing the software fix as described in the Patches section, or disabling Flash Cache Compression as described in the Workaround section, does not correct a system that already has failed flash drives.

Assessing the Risk of Hitting Critical Issue EX28

X4 and X3 Storage servers running Exadata version 12.1.2.2.0, 12.1.2.2.1, or 12.1.2.3.0 with Flash Cache Compression enabled are exposed to bug 22909764. Storage servers with this configuration that then have flash devices reach too little free space may experience many flash cards failing in a short period of time across multiple cells.  If Flash Cache mode is set to write-back and multiple flash cards fail across multiple cells, then it is likely that ASM disk group(s) will dismount and data will be lost.  If Flash Cache mode is set to write-through, then there is no data loss, but there will likely be performance impact because flash cache content will have been lost.

This section applies to systems that currently have no flash disk failures due to this issue.

Flash failure due to this issue is caused by flash disks reaching too little free space. Because flash cache free space is affected by workload and working set size, some storage servers may hit the issue shortly after upgrade, while others may never hit the issue.  The following command may be used on each storage server to determine the amount of free space currently available on the 16 flash disks that are operating normally (i.e. ensure all 16 flash drives are reported - drives cannot already be failed):

Depending on workload and working set size the amount of free space on flash drives may continue to decrease over time (potentially over a short period of time).  As free space decreases, the likelihood of hitting this issue increases.  The free space threshold is dependent on hardware model (X4 or X3) according to the table below.

With the software fix in place and flash cache compression operating normally, the amount of free space should stabilize at ~68 GiB on X4 storage servers, and ~34 GiB on X3 storage servers.

Failure and Recovery Scenarios

When this issue occurs content on the failed flash drives will be lost.  Impact and recovery steps depend on flash cache mode (writethrough or writeback).

Flash cache mode writethrough

If FC mode is writethrough, then there will likely be impact to database read performance since read-only flash cache will be lost. There should be no availability impact to ASM disk groups or database.

1. Re-enable failed flash drives using the steps below.

Flash cache mode writeback

If FC mode is writeback, then there are writes acknowledged by the flash cache that are not persisted on disk, hence there is data loss. There are multiple possible scenarios.

Scenario 1 - One or more flash disks have failed but the database is still operational

1. To avoid the need to restore and recovery the database, workload must be immediately moved to a standby system or stopped to avoid additional flash failures that might cause disk group dismount.
2. Re-enable failed flash drives using the steps below.

Scenario 2 - One or more flash disks have failed and the database is not operational.  DATA disk group has dismounted and, possibly clusterware has crashed due to loss of OCR/voting files.

1. DATA disk group will have writes acknowledged by the flash cache that are not persisted on disk, hence there is data loss, will likely dismount, and cannot be mounted. Assume data files in DATA are lost and must be restored/recovered.
2. If there is a standby database
   1. Switch to the standby
   2. Re-enable failed flash drives using the steps below.
   3. Restore the primary from backup.
3. If there is no standby database, then restore/recover from backup.
   1. Re-enable failed flash drives using the steps below.
   2. Restore the primary from backup.
      1. If OCR and voting files are lost, then recreate them.
      2. RECO disk group, by default, is not cached in flash cache. Backups and archived logs stored in RECO should be accessible.
      3. Online redo logs, by default, are not cached in flash cache, hence online logs in DATA are intact, but DATA cannot be mounted. Online logs can be extracted from the offline DATA disk group and used for recovery using the process in <document 1968607.1> in section "The Recovery Procedure example with the Diskgroup dismounted."

Steps to re-enable flash drives that failed due to bug 22909764 (bug 22848220)

To re-enable failed flash cards where any flash drive on a flash card has failed, follow these steps:

1. If flash mode is writeback, and the database is still operational, then workload must be immediately moved or stopped.  See Scenario #1 under Flash cache mode writeback section above.
2. Stop cell services: CellCLI> alter cell shutdown services all
   1. If this procedure needs to be run on more than one cell, then do so in a rolling manner inactivating grid disks cell-by-cell while the work is performed, as you would for any rolling cell maintenance.
3. Add this parameter to cellinit.ora on the cell: _cell_allow_reenable_predfail=true
4. Install the software fix using details from the Patches section of this document.  Patch installation reboots the cell.
5. Erase the flash card(s) where any flash disk has failed
   1. Determine PCI slot / address mapping for the flash card containing the failed flash disk.  This is required for the --pci option of the set_flash_compression.sh command run below.
      CellCLI> list physicaldisk attributes name,slotNumber,status where diskType=FlashDisk
      

      slotNumber from list physicaldisk	Address to use in --pci option in set_flash_compression.sh command
      PCI Slot: 1	90
      PCI Slot: 2	b0
      PCI Slot: 4	30
      PCI Slot: 5	20

   2. # cd $OSS_SCRIPTS_HOME/unix/hwadapter/diskadp/flash/lsi/
   3. # ./set_flash_compression.sh --erase --pci '<one of 90,b0.30,20>' --slot '0,1,2,3'
      e.g. # ./set_flash_compression.sh --erase --pci '90' --slot '0,1,2,3'
6. Re-enable flash disks on the erased flash card(s) that are not NORMAL, one at a time.
   CellCLI> alter physicaldisk <name> reenable force
   e.g. 
   CellCLI> alter physicaldisk FLASH_1_0 reenable force
   CellCLI> alter physicaldisk FLASH_1_1 reenable force
   CellCLI> alter physicaldisk FLASH_1_2 reenable force
   CellCLI> alter physicaldisk FLASH_1_3 reenable force
7. Remove parameter _cell_allow_reenable_predfail from cellinit.ora on the cell
8. Restart cell services: CellCLI> alter cell restart services all

Steps to handle upgrade failure due to latent effects of critical issue EX17

An X3 cell may be exposed to a similar flash disk failure issue previously published as Exadata critical issue EX17 in <Document 1968234.1>.  EX17 summary is an X3 cell upgraded from 11.2.3.3.0/12.1.1.1.0 to 11.2.3.3.1/12.1.1.1.1/12.1.2.1.0 while flash cache compression is enabled will have a mismatch on each flash disk between the compression setting and the flash disk size.  The result is that Exadata software behaves as if compression is enabled, when, in fact, it is not, which can lead to flash disk failure if free space is exhausted.

However, in cases where workload and working set size is small enough not to exhaust free space, there will be no failure after upgrade even though the mismatch is still present.

A new check was added in Exadata 12.1.2.2.0 to validate each flash disk has correctly matching compression setting and size.  If there is a mismatch and the cell is upgraded from a release affected by EX17 to 12.1.2.2.0 or later, then cellsrv will fail to start with the following error:

ORA-00600: internal error code, arguments: [ossflc_create - DFF and Falcon out of sync regarding DLC state.]

The scenario can occur when upgrading to 12.1.2.2.0 or later using one of the full release patches listed in the Patches section below.

To resolve this condition, perform the internal-only resolution steps in <Document 1968234.1>.

Workaround

If you believe that your system is currently exhibiting the Symptoms described above, please contact Oracle Support.

Recommended Action

Apply the software fix as described in the Patches section.

If your system is running an earlier Exadata version (12.1.2.1.3 or lower), and Flash Cache Compression is enabled, and you will upgrade to 12.1.2.3.0, 12.1.2.2.1 or 12.1.2.2.0, then upgrade using the full release component of the target release as described in the Patches section.

Alternate Action

A workaround to avoid this issue is to disable Flash Cache Compression by following the steps in the Oracle Exadata Database Machine Maintenance Guide.  Disable Flash Cache Compression as a workaround only if there are no failed flash drives.  If there are failed flash drives, contact Oracle Support before attempting to disable Flash Cache Compression.  Grid disks created on flash drives must be moved to hard drives before disabling Flash Cache Compression.

The workaround may be performed on systems already running 12.1.2.2.0, 12.1.2.2.1, or 12.1.2.3.0, or on systems running an earlier version prior to upgrade.  Re-enable Flash Cache Compression only after the software fix is installed.

Patches

Recommended Action

The recommended action is update to Exadata 12.1.2.2.2, 12.1.2.2.3, 12.1.2.3.1, or higher release.  

INTERNAL: Due to EX29, these patches have been revoked.  Recommended action for all customers who are still exposed to this issue is update to revised 12.1.2.2.2/12.1.2.3.1.  The revoked patches will not be re-released.  Customers who previously updated to one of the revoked patches are considered safe - no further action is required.

Alternatively, patches are available for the following releases:

• 12.1.2.3.0 - <Patch 22934283>
• 12.1.2.2.1 - <Patch 22917774>¹
• 12.1.2.2.0 - <Patch 22928622>¹

The patch contains the following two components:

1. Interim fix
2. Full release plus interim fix

The following table summarizes the recommended actions based on the currently installed Exadata version:

Footnotes

¹ Patches 22917774 and 22928622 for Exadata versions 12.1.2.2.1 and 12.1.2.2.0, respectively, were re-released on 01-Apr-2016.  No action is required for storage servers that were updated using the previously released patches.

History

29-Jul-2016 - Add 12.1.2.2.3 as recommended release
25-Jul-2016 - Due to EX29, 12.1.2.2.2/12.1.2.3.1 re-release, and revoked earlier version patches, remove earlier version patches from Patches section.  No action required for systems already updated using an earlier version patch.
21-Apr-2016 - 12.1.2.2.2 and 12.1.2.3.1 released, both which contain the fixes.  Update to either of these versions becomes the recommended action.
01-Apr-2016 - 12.1.2.3.0 patch released; 12.1.2.2.1/12.1.2.2.0 patches revised; externalize section "Assessing the Risk of Hitting Critical Issue EX28"
22-Mar-2016 - Add reference to check for EX17 first if upgrading; address open comment - empty flashCacheCompress setting means disabled
19-Mar-2016 - 12.1.2.2.1 and 12.1.2.2.0 patches released
16-Mar-2016 - Published

This solution has no attachment