Oracle VM: Buffer I/O Error on Device, FSCK Unable to Fix the Issue

Asset ID:	1-72-1963849.1
Update Date:	2018-01-08
Keywords:

Solution Type Problem Resolution Sure

Solution 1963849.1 : Oracle VM: Buffer I/O Error on Device, FSCK Unable to Fix the Issue

Applies to:

Oracle VM - Version 3.0.2 to 3.3.1 [Release OVM30 to OVM33]
Exalogic Elastic Cloud X4-2 Hardware - Version X4 to X4 [Release X4]
x86 64 bit

Symptoms

I/O Errors like below in /var/log/messages file of the Guest VM:

Jan 2 14:06:25 kernel: EXT4-fs warning (device dm-3): ext4_end_bio:258: I/O error writing to inode 62 (offset 1048576 size 8192 starting block 820226)
Jan 2 16:48:11 kernel: Buffer I/O error on device dm-3, logical block 820226
Jan 2 16:48:11 kernel: Buffer I/O error on device dm-3, logical block 820227
Jan 2 16:48:11 kernel: EXT4-fs warning (device dm-3): ext4_end_bio:258: I/O error writing to inode 62 (offset 1048576 size 8192 starting block 820226)

Jan 7 08:39:08 kernel: EXT4-fs (dm-0): error count: 8
Jan 7 08:39:08 kernel: EXT4-fs (dm-0): initial error at 1420187288: ext4_mb_generate_buddy:736
Jan 7 08:39:08 kernel: EXT4-fs (dm-0): last error at 1420578279: ext4_lookup:1044: inode 393219
Jan 7 08:39:16 kernel: EXT4-fs (dm-4): error count: 17
Jan 7 08:39:16 kernel: EXT4-fs (dm-4): initial error at 1420187311: mb_free_blocks:1345: block 599060
Jan 7 08:39:16 kernel: EXT4-fs (dm-4): last error at 1420189009: ext4_lookup:1044: inode 131313
Jan 7 08:39:16 kernel: EXT4-fs (xvdb1): error count: 10
Jan 7 08:39:16 kernel: EXT4-fs (xvdb1): initial error at 1420187294: ext4_mb_generate_buddy:736
Jan 7 08:39:16 kernel: EXT4-fs (xvdb1): last error at 1420193163: ext4_mb_generate_buddy:736

Changes

Nothing is changed.

Cause

May not be harmful. Depends on how it appears.

Note: The EXT4 filesystem errors shown here are from the VM guest machine running on OVM setup. Here the underlying Repo and Pool are having ocfs2 filesystem. So, when there is a lock occurs on the OCFS2 filesystem, the guest machines can't write to the Repo. The result is an error on EXT4 File System running on a Guest VM.

It can be ignored. But before that, you need to analyze the error and categorize it under the following:

If this error occur on same logical volume device, eg: dm-3, it can be a problem indicating underlying hard disk issue. This disk issue can be an ongoing one or it could indicate a past disk issue. The errors are still displayed because of the fact that EXT4 filesystem stores error messages and reports them again in a particular interval of time. The error message has to be flushed from the filesystem to stop the errors.
If the same error occur on different logical volume device, it can be ignored. In this case, such an error occur due to OCFS2 lock mechanism. Since repo is shared, many devices write to it at the same time. So, once a device is busy, OCFS2 locks the device and if any other device try to access it at that time, such an error saying "I/O error writing" will be displayed. Its just an informational error.

Solution

If the error is shown only for one logical volume device, you need to first check if its an ongoing issue or not. To be more clear, just execute the command below and you will get the exact date and time at which this issue occurred:

# date -u --date="1970-01-01 1420187288 sec GMT"

* The number "1420187288" in the above command is obtained from the error message in the /var/log/messages file. A sample error is pasted below:

Jan 7 08:39:08 kernel: EXT4-fs (dm-0): error count: 8
Jan 7 08:39:08 kernel: EXT4-fs (dm-0): initial error at 1420187288: ext4_mb_generate_buddy:736

The issue actually occurred at the time which you are getting as output of above command. The output of above commands will be some thing like the below:

[root@ovs1 ~]# date -u --date="1970-01-01 1420187288 sec GMT"
Fri Jan 2 08:28:08 UTC 2015

If its is a recent date that you get, then it indicates that issue still occurs. But if its a past date, then it indicates that issue is not occurring now. It can be ignored or flush it.

First run a file system check with the command below to see if there is any filesystem error currently:

# e2fsck <disk under issue>

This will automatically fix the filesystem error. See if error messages stops. If, the issue still persists, try the next option i.e do a filesystem check forcefully with below command:

# e2fsck -f <disk under issue>

If this also did not fix the issue, then it requires to proceed with the third option i.e flushing the file system cache manually with below command:

# e2fsck -F <disk under issue>

If still error messages appear then the need arises to check the underlying disk for any filesystem issues. It indicates a filesystem corruption.

Note: The e2fsprog in Oracle Linux 5 does not have the code in it to handle the EXT4 s_error_count field of the EXT4 superblock. So, to get rid of those messages, you may want to boot the server in rescue mode with OL 6.6 iso and run fsck against those erroneous disks.

References

<NOTE:1506029.1> - Oracle VM: Error "kernel: EXT4-fs warning (device ): ext4_end_bio:: I/O error writing to inode" in Guest Domain
<NOTE:790716.1> - The Error "Buffer I/O error on device XXX, logical block XXX" on Reboot
<BUG:20683895> - EXT4 ERROR MESSAGES KEEP PROMPTING OUT DAILY AFTER FSCK THE PARTITION
<NOTE:1515713.1> - How to prevent Buffer I/O Error during the boot time in Linux 6

Attachments

This solution has no attachment