How to prevent Buffer I/O Error during the boot time in Linux 6

Asset ID:	1-72-1515713.1
Update Date:	2017-12-13
Keywords:

Solution Type Problem Resolution Sure

Solution 1515713.1 : How to prevent Buffer I/O Error during the boot time in Linux 6

Applies to:

Linux OS - Version Oracle Linux 6.1 to Oracle Linux 6.3 [Release OL6U1 to OL6U3]
Exalogic Elastic Cloud X4-2 Hardware - Version X4 to X4 [Release X4]
Information in this document applies to any platform.

Symptoms

The server gets below Buffer I/O error during boot time under kernel 2.6.32-220.el6.x86_64 in Linux 6, and the storage is Sun Storagetek 6140 operating in active/passive mode.

Oct 15 13:07:01 png1bkme4 kernel: sd 1:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 15 13:07:01 png1bkme4 kernel: sd 1:0:0:1: [sdc] Sense Key : Illegal Request [current]
Oct 15 13:07:01 png1bkme4 kernel: sd 1:0:0:1: [sdc] <> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Oct 15 13:07:01 png1bkme4 kernel: sd 1:0:0:1: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Oct 15 13:07:01 png1bkme4 kernel: end_request: I/O error, dev sdc, sector 0
Oct 15 13:07:01 png1bkme4 kernel: Buffer I/O error on device sdc, logical block 0
Oct 15 13:07:02 png1bkme4 kernel: sd 1:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 15 13:07:02 png1bkme4 kernel: sd 1:0:0:1: [sdc] Sense Key : Illegal Request [current]
Oct 15 13:07:02 png1bkme4 kernel: sd 1:0:0:1: [sdc] <> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Oct 15 13:07:02 png1bkme4 kernel: sd 1:0:0:1: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Oct 15 13:07:02 png1bkme4 kernel: end_request: I/O error, dev sdc, sector 0
Oct 15 13:07:02 png1bkme4 kernel: Buffer I/O error on device sdc, logical block 0

Cause

The main reason of the errors come out during boot time is because the HBA driver gets loaded before the RDAC driver and the ghost paths get probed by the HBA driver.

The error is only seen for the ghost devices, however, in fact, the ghost paths should not be probed.

multipath -ll output

===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
  1:0:0:1 sdc 8:32 -1 undef ghost SUN,CSM200_R running
  1:0:1:0 sde 8:64 -1 undef ghost SUN,CSM200_R running
  1:0:1:2 sdg 8:96 -1 undef ghost SUN,CSM200_R running
  1:0:2:0 sdh 8:112 -1 undef ghost SUN,CSM200_R running
  1:0:2:1 sdi 8:128 -1 undef ghost SUN,CSM200_R running
  1:0:2:2 sdj 8:144 -1 undef ghost SUN,CSM200_R running
  1:0:2:3 sdk 8:160 -1 undef ghost SUN,CSM200_R running
  1:0:3:5 sdq 65:0 -1 undef ghost SUN,CSM200_R running
  2:0:0:1 sds 65:32 -1 undef ghost SUN,CSM200_R running
  2:0:1:0 sdu 65:64 -1 undef ghost SUN,CSM200_R running
  2:0:1:2 sdw 65:96 -1 undef ghost SUN,CSM200_R running
  2:0:2:0 sdx 65:112 -1 undef ghost SUN,CSM200_R running
  2:0:2:1 sdy 65:128 -1 undef ghost SUN,CSM200_R running
  2:0:2:2 sdz 65:144 -1 undef ghost SUN,CSM200_R running
  2:0:2:3 sdaa 65:160 -1 undef ghost SUN,CSM200_R running
  2:0:3:5 sdag 66:0 -1 undef ghost SUN,CSM200_R running

multipath configuration

device {
  vendor "SUN"
  product "CSM200_R"
  getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
  features "0"
  hardware_handler "1 rdac"
  path_selector "round-robin 0"
  path_grouping_policy group_by_prio
  failback immediate
  rr_weight uniform
  no_path_retry queue
  rr_min_io 1000
  path_checker rdac
  prio rdac
  }

Solution

Note that this kind of Buffer I/o error messages are harmless and can be ignored.

To solve this issue, please add parameter "rdloaddriver=scsi_dh_rdac" to the kernel command line in file "/etc/grub/grub.conf" in order to load the RDAC driver before the HBA driver and skip the unmeaningful probing of ghost paths. Then reboot.

title Red Hat Enterprise Linux (2.6.32-220.el6.x86_64)
  root (hd0,0)
  kernel /vmlinuz-2.6.32-220.el6.x86_64 ro root=UUID=bf52627d-407d-4a25-b828-f530a4d6fba5 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rdloaddriver=scsi_dh_rdac
  initrd /initramfs-2.6.32-220.el6.x86_64.img

References

<BUG:14285401> - PAGES OF ERROR MESSAGES END_REQUEST: I/O ERROR, DEV SDX SLOWING DOWN BOOTUP
<NOTE:790716.1> - The Error "Buffer I/O error on device XXX, logical block XXX" on Reboot
<NOTE:1493675.1> - Sun StorageTek[TM] 6140 Multipath Software Compatibility Matrix
<BUG:14784404> - LINUX 6.2 BUFFER I/O ERROR

Attachments

This solution has no attachment