Oracle Exadata Storage Server Software
 
Exadata X3-2 Hardware
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

In this Document

Symptoms

Cause

Solution

References

Created from <SR 3-8773210011>

Applies to:

Symptoms

The battery was replaced for the LSI disk controller on one of the storage cells. After the replacement, the storage cells is continuously rebooting. The same symptoms may also occur without any hardware maintenance activity having previously been performed.

Cause

From the ILOM snapshot or via the /SP/console or ILOM Remote Console, we see the following messages before the reboot of the server, which indicates a kernel panic :

md: ... autorun DONE.
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
EXT3-fs (md4): error: unable to read superblock   
EXT3-fs (md6): error: unable to read superblock
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.39-400.126.1.el5uek #1
Call Trace:
 [] panic+0xbf/0x1f0
 [] ? free_vfsmnt+0x3a/0x50
 [] ? account_entity_enqueue+0x8f/0xa0
 [] find_new_reaper+0xc9/0xd0
 [] forget_original_parent+0x45/0x1b0
 [] ? sched_move_task+0xb2/0x150
 [] exit_notify+0x16/0x170
 [] do_exit+0x283/0x460
 [] do_group_exit+0x41/0xb0
 [] sys_exit_group+0x17/0x20
 [] system_call_fastpath+0x16/0x1b
Rebooting in 60 seconds..
ACPI MEMORY or I/O RESET_REG.

This may suggest that the RAID controller is either not present or has not been able to find any disk drives. The system is booting from the internal USB drive, but is failing to mount the root partition located on the physical disks in slots 0 and 1. As it is unable to mount the root partition, /sbin/init is not found and the kernel panics.

The ilom/@persist@hostconsole.log as collected via the ILOM snapshot will contain lines similar to the following when the LSI HBA controller is properly detected, along with it's 12 physical disks. Note that the disk manufacturer may differ from below:

If entries similar to above do show up, we are likely dealing with a corruption of the root filesystem. This generally requires that the cell be reimaged from the internal USB. 

Solution

This may be due to improper battery replacement or improper card seating or a card damaged during battery replacement.  It may also be due to a corruption of the root filesystem.

1: An ILOM snapshot should be gathered to assist support.
2: Open a SR with Oracle Support to determine the cause and assist with resolving the issue. If there is an existing SR, then please re-enagage the owner to assist.

If it's deemed that the storage cell needs to be reimaged, the following steps can be followed via the java based ILOM Remote Console. They assume that the internal USB is healthy.

1. Select the last line from the grub menu. It reads: CELL_USB_BOOT_CELLBOOT_usb_in_rescue_mode

2. When prompted, select (r)einstall or try to recover damaged system. Confirm your decision when prompted "Are you sure?"

3. When prompted whether to erase data partition and disks, choose "no"

4. Follow the remaining prompts

5. Once cell is up, the celldisks will need to be imported. Run:

5. Check if flashlog and flashcache are created:

6. Run the following and check if flashCacheMode matches that of a healthy cell:

7. Manually add the griddisks to ASM:

a. Check the status of the griddisks

- The griddisks belonging to the reimaged cell should show up with a header_status of CANDIDATE

b. For each diskgroup (DATA, RECO, DBFS_DG, etc), run:

 e.g. for DATA diskgroup, given a cell whose name is chsmchs00203 and whose Infiniband is configured for active/active, hence two IPs:

References

This solution has no attachment