Oracle Exadata Storage Server Software
 
Exadata Database Machine X2-2 Hardware
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

Applies to:

Symptoms

This issue could happen at the time of rolling upgrade from say 11.2.2.4.2 to 11.2.3.1.1 or after replacing a bad disk. After reboot of cell the error could be :-

"FAILED: Wait for cell to reboot and come online."

Cellcli commands give error below :-

#cellcli -e 'list griddisk attributes name,status,asmmodestatus,asmdeactivationoutcome'

CELL-04582: An error was encountered while populating the disk.

List lun disk, list celldisk also give the error :-

CELL-04582: An error was encountered while populating the disk.

Cause

 When one FDOM is having problem ,cellsrv or MS should mark it as not present and should proceed further with all the available disks, instead of preventing any command to execute and giving no access to any disk at all in the cell

From /var/log/messages we can see:
 
Sep 25 18:34:47 x02pcel01 kernel: scsi 11:0:0:0: mptscsih: ioc3: qdepth=64,
  tagged=1, simple=1, ordered=0, scsi_level=6, cmd_que=1
Sep 25 18:34:47 x02pcel01 kernel: sdev dma_alignment 511
Sep 25 18:34:47 x02pcel01 kernel: SCSI device sdz: 47999744 512-byte hdwr
sectors (24576 MB)
..

..
Sep 25 18:34:47 x02pcel01 kernel:  sdz:<6>sd 11:0:0:0: SCSI error: return  <<< Error
code = 0x08000002
Sep 25 18:34:47 x02pcel01 kernel: sdz: Current: sense key: Aborted Command
Sep 25 18:34:47 x02pcel01 kernel:     Add. Sense: No additional sense
information
Sep 25 18:34:47 x02pcel01 kernel:
Sep 25 18:34:47 x02pcel01 kernel: end_request: I/O error, dev sdz, sector 0 <
Sep 25 18:34:47 x02pcel01 kernel: Buffer I/O error on device sdz, logical

MS was running, recreating flashcache did not fix the problem.

cellcli>drop flashcache;
cellcli>create flashcache......;

See the note Exadata Write-Back Flash Cache - FAQ <Document 1500257.1> and the Documentation for more details about the syntax.

system could have a bad flash card with IO errors, and it needs replacement as it has disappeared from the OS.

Solution

Replace the failing flash card.

Once the bad flash card was replaced with a good one, all cellcli commands started working & flashcache size was also showing up correctly  

References

This solution has no attachment