Exadata Database Machine X2-2 Hardware
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

Applies to:

Symptoms

Observed following messages in the /var/log/messages file before the reboot:

Oct 28 23:58:27 myexa104 kernel: ib1: Found gid with mismach lids.(cache:54,from arp: 52)
Oct 28 23:58:27 myexa104 MRdiagd: MR Controller event (seq 27870) tracer=Controller_500605b004920b80 ctrlId=500605b004920b80 code=113 (PD:Info)
Oct 28 23:58:28 myexa104 MRdiagd: MR Controller event (seq 27871) tracer=Controller_500605b004920b80 ctrlId=500605b004920b80 code=113 (PD:Info)
Oct 28 23:58:29 myexa104 MRdiagd: MR Controller event (seq 27872) tracer=Controller_500605b004920b80 ctrlId=500605b004920b80 code=113 (PD:Info)
.
.
.
Oct 29 00:08:55 myexa104 kernel: imklog 5.8.10, log source = /proc/kmsg started.

Cause

System disk utilization was 100% for more than 60 Sec
And during this time, because of no network communication, node was evicted by CRS. (CRS miscount time is 60sec)

This is believed to be related to "Bug 21549999 : ENHANCE SCSI ERROR HANDLING TO NOT BLOCK IOS TO ALL DISKS , JUST THE BAD DISK". When one disk has issues like this, the whole system will hang.
 
10/29/16 00:05:34 (100% disk utilization for 10/29/16 00:04:39 to 10/29/16 00:05:34 and continue)
avg-cpu: %user %nice %system %iowait %steal %idle
  0.83 0.20 0.92 12.18 0.00 85.87

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 30.80 0.00 4.00 0.00 278.40 69.60 1.02 5.25 0.00 5.25 250.00 100.00 <<<<<<<
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 30.80 0.00 4.00 0.00 278.40 69.60 1.02 5.25 0.00 5.25 250.00 100.00
dm-0 0.00 0.00 0.00 34.80 0.00 278.40 8.00 0.10 2.83 0.00 2.83 0.30 1.06
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

CRS alert.log
===========
2016-10-29 00:05:08.279 [OCSSD(75476)]CRS-1612: Network communication with node myexa104 (4) missing for 50% of timeout interval. Removal of this node from cluster in 29.330 seconds <<< Attempt to evict node
2016-10-29 00:05:23.281 [OCSSD(75476)]CRS-1611: Network communication with node myexa104 (4) missing for 75% of timeout interval. Removal of this node from cluster in 14.330 seconds
2016-10-29 00:05:32.282 [OCSSD(75476)]CRS-1610: Network communication with node myexa104 (4) missing for 90% of timeout interval. Removal of this node from cluster in 5.340 seconds
2016-10-29 00:05:36.178 [OHASD(70682)]CRS-8011: reboot advisory message from host: myexa104, component: cssagent, with time stamp: L-2016-10-29-00:05:36.143
2016-10-29 00:05:36.178 [OHASD(70682)]CRS-8013: reboot advisory message text: oracssdagent is rebooting this node due to vote disk I/O timeout (no disk activity for 58490 milliseconds).
2016-10-29 00:05:37.416 [OCSSD(75476)]CRS-1632: Node myexa104 is being removed from the cluster in cluster incarnation 371410709
2016-10-29 00:05:37.606 [OCSSD(75476)]CRS-1601: CSSD Reconfiguration complete. Active nodes are myexa101 myexa102 myexa103 .
2016-10-29 00:05:37.627 [CRSD(90438)]CRS-5504: Node down event reported for node 'myexa104'.
 

Solution

The fix for Bug 21549999  is in UEK4 where one bad disk should not block the IOs from all disks.
But this kernel is not supported by Exadata yet.

Workaround
==========
1] Decrease scsi IO timeout value to let the I/O is able to recovery in short time
"echo 10 > /sys/block/sda/device/timeout

The command "echo 10 > /sys/block/sda/device/timeout" do not persistent after server reboot.
To make it persist, use the command:

# echo "echo 10 > /sys/block/sda/device/timeout" >> /etc/rc.local

2] Check for system disk and get it replaced.

This solution has no attachment