Exalogic Compute Nodes going into Readonly state with "sd 0:2:0:0: rejecting I/O to offline device" errors

Asset ID:	1-72-1952405.1
Update Date:	2016-09-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1952405.1 : Exalogic Compute Nodes going into Readonly state with "sd 0:2:0:0: rejecting I/O to offline device" errors

Applies to:

Oracle Exalogic Elastic Cloud X2-2 Hardware - Version X2 and later
Information in this document applies to any platform.

Symptoms

In Exalogic racks following issue is observed

Issue of Compute Nodes file system going to read only state is observed. At the time of the issue root file system freezes and access to some commands get restricted (ls and other commands not working) on the Compute Nodes. After bouncing the server it comes back normally.

MegaCLI commands also do not work at the time of the issue.

Also OS level commands fail with Input/Output error as shown below.

[root@testCN ~]# sosreport
-bash: /usr/sbin/sosreport: Input/Output error

[root@testCN redis]# touch /etc/20160825
touch: can not touch “/etc/20160825”: Read only file system

Following are messages shown in the logs.

sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
EXT3-fs error (device sda5): ext3_find_entry: reading directory #346372 offset 0
sd 0:2:0:0: rejecting I/O to offline device
EXT3-fs error (device sda5): ext3_find_entry: reading directory #346372 offset 0
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device

Cause

This issue is likely caused due to failure of the RAID Controller. RAID controller needs to replaced on this issue.

Solution

In case you face this issue please collect following data from the CN where we see the issue and contact support.

ILOM snapshot
sundiag output. Refer to <Note 1965892.1> which has information on collecting sundiag output.

As immediate workaround for this issue reboot the CN which has the issues.

Attachments

This solution has no attachment