Not Able To Create Or Modify Or Delete Any Files/folders From "/u04". But Able To Perform The Same On "/u03" filesystem

Asset ID:	1-72-2045868.1
Update Date:	2015-08-31
Keywords:

Solution Type Problem Resolution Sure

Solution 2045868.1 : Not Able To Create Or Modify Or Delete Any Files/folders From "/u04". But Able To Perform The Same On "/u03" filesystem

Applies to:

Exalytics In-Memory Machine X4-4 - Version All Versions to All Versions [Release All Releases]
Linux x86-64

Symptoms

On Exalytics x4-4 Bare Metal with Patch set 5, Filesystem /u04 on node sdl2 went to read only mode following a network issue in the DC (Data Center).

There is /u03 from the same DC which is working fine in node sdl2. The related node sdl1 is having the same shared mount points /u03 and /u04 which are working fine.

While attempting to create/modify option on the /u04 on node fails with the error "Operation not permitted; Filesystem in Read only mode"

The error can be reproduced by attempting to create a new file or by modifying an existing file.

Cause

The network outage has resulted in the iSCSI connection issues due to which this filesystem was turned into read only mode.

The /var/log/messages log file in the sosreport reported the below error message:

Aug 15 21:06:26 sdl2 kernel: ixgbe 0000:91:00.0: eth0: NIC Link is Down
Aug 15 21:06:26 sdl2 kernel: ixgbe 0000:91:00.1: eth1: NIC Link is Down
Aug 15 21:06:26 sdl2 kernel: bonding: bond1: link status down for backup interface eth0, disabling it in 5000 ms.
Aug 15 21:06:26 sdl2 kernel: bonding: bond1: link status down for backup interface eth1, disabling it in 5000 ms.
Aug 15 21:06:31 sdl2 kernel: bonding: bond1: link status definitely down for interface eth0, disabling it
Aug 15 21:06:31 sdl2 kernel: bonding: bond1: now running without any active interface !
Aug 15 21:06:31 sdl2 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Aug 15 21:06:32 sdl2 kernel: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 12535194880, last ping 12535199888, now 12535204896
Aug 15 21:06:32 sdl2 kernel: connection1:0: detected conn error (1011)
Aug 15 21:06:32 sdl2 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) ---------------- > iSCSI connection error
Aug 15 21:06:34 sdl2 kernel: connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 12535197136, last ping 12535202144, now 12535207152
Aug 15 21:06:34 sdl2 kernel: connection2:0: detected conn error (1011)
Aug 15 21:06:34 sdl2 iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3) ------------------- > iSCSI connection errors
Aug 15 21:07:16 sdl2 iscsid: connect to 155.111.184.73:3260 failed (No route to host) ------------------- > iSCSI connection errors
Aug 15 21:07:48 sdl2 last message repeated 12 times
Aug 15 21:08:31 sdl2 last message repeated 15 times
Aug 15 21:08:32 sdl2 kernel: session1: session recovery timed out after 120 secs
Aug 15 21:08:34 sdl2 kernel: session2: session recovery timed out after 120 secs
Aug 15 21:08:37 sdl2 iscsid: connect to 155.111.184.73:3260 failed (No route to host) ------------------- > iSCSI connection errors

Solution

To fix the issue, follow the below steps:

Unmount the filesystem using the umount command.
Mount it again in read write mode.
# mount -o remount,rw /u04

Attachments

This solution has no attachment