LUN not created automatically after Hard disk replacement. Alerthistory shows warning "Failed to create LUN on Hard disk"

Asset ID:	1-72-2188340.1
Update Date:	2016-10-03
Keywords:

Solution Type Problem Resolution Sure

Solution 2188340.1 : LUN not created automatically after Hard disk replacement. Alerthistory shows warning "Failed to create LUN on Hard disk"

Applies to:

Exadata Database Machine X2-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

LUN was not created automatically after a hard disk was replaced. As a result, the celldisks/griddisks were not created as well.

Alert history shows:

7_2 2016-08-15T14:15:41-05:00 warning "System hard disk can be replaced now. Status : WARNING - PREDICTIVE FAILURE Manufacturer : SEAGATE Model Number : ST360057SSUN600G Size : 600GB Serial Number : 1108E0BMBT Firmware : 0B25 Slot Number : 0 Cell Disk : CD_00_dm01cel02 Grid Disk : DATA_DM01_CD_00_dm01cel02, RECO_DM01_CD_00_dm01cel02 "
7_3 2016-08-29T14:32:59-05:00 warning "System hard disk was removed. Status : NOT PRESENT Manufacturer : SEAGATE Model Number : ST360057SSUN600G Size : 600GB Serial Number : 1108E0BMBT Firmware : 0B25 Slot Number : 0 Cell Disk : CD_00_dm01cel02 Grid Disk : DATA_DM01_CD_00_dm01cel02, RECO_DM01_CD_00_dm01cel02"
7_4 2016-08-29T14:33:42-05:00 warning "Failed to create LUN on Hard disk. Status : HITACHI Manufacturer : HUS1560SCSUN600G Model Number : SAS Size : 600GB Serial Number : 1624KY1WUL Firmware : A820 Slot Number : 0"

MS log shows the following error continuously, from the time the new disk was inserted.

[2016-08-29T14:33:42.301-05:00] [ossmgmt] [WARNING] [] [oracle.ossmgmt.common.util.CommandExecutionUtil] [tid: 64] Exec Command Error for cmd: /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -R0 [20:0] WB NORA Direct NoCachedBadBBU -strpsz1024

Adapter 0: Configure Adapter Failed

FW error description:
The current operation is not allowed because the controller has data in cache for offline or missing virtual disks.

Exit Code: 0x54
errorString:
]]

Changes

Hard disk replacement

Cause

Disk array controller has preserved cache for the old disk in slot 0, which is preventing the creation of lun

Solution

We are using HDD in slot 0 as an example below. All the commands listed needs to be run as root user

1. Drop the preserved cache for the disk in slot 0.

#/opt/MegaRAID/MegaCli/MegaCli64 -DiscardPreservedCache -L0 -force -a0

-L0 in the above command indicates slot #0.
It is safe to do this while cluster/databases are up, as there are no griddisk created on the disk because of LUN creation failure.

2. Wait a few minutes and see if the lun gets autocreated.

cellcli -e list lun 0_0

if the lun is auto created, then check to see if the celldisks/griddisks are created and the rebalance has kicked off on ASM while the disks are being added to the diskgroup.

3. If the luns are not autocreated, please try restarting the cell services.

--make sure the griddisks can be taken offline
#cellcli -e list griddisk attributes name,status,asmmodestatus,asmdeactivationoutcome

**** Please proceed with cell services restart _ONLY_ if asmdeactivationoutcome= Yes for _ALL_ griddisks. If you see any other status other than "Yes", Please stop and resolve the condition.
-- if asmdeactivationoutcome is "Yes" for all the disks, restart cell processes.
#cellcli -e alter cell restart services all

--This should only take less than a minute to restart. check the status of services and make sure all 3 processes are running
eg:
# service celld status
rsStatus: running
msStatus: running
cellsrvStatus: running

-- follow step 2 above and see if the lun/celldisk/griddisks are created.

4. If all the above steps fail, please plan on rebooting the cell. The lun will be created on boot.

Note:1188080.1 Steps to power down or reboot an Exadata storage cell without affecting ASM

Attachments

This solution has no attachment