After changing System date in ILOM, node hangs on bootup with console showing failure at: Superblock last mount time "SOME TIME IN THE FUTURE"

Asset ID:	1-72-2000224.1
Update Date:	2015-04-27
Keywords:

Solution Type Problem Resolution Sure

Solution 2000224.1 : After changing System date in ILOM, node hangs on bootup with console showing failure at: Superblock last mount time "SOME TIME IN THE FUTURE"

Applies to:

Exadata X4-2 Hardware - Version All Versions and later
Linux OS - Version Oracle Linux 5.9 and later
Exadata X5-2 Hardware - Version All Versions and later
Oracle Exadata Storage Server Software - Version 11.2.3.2.2 to 12.1.2.1.1 [Release 11.2 to 12.1]
Exadata X3-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

After changing System date/time, node hangs on bootup. Console shows failure at: Superblock last mount time ("SOME TIME IN THE FUTURE" Give root password for maintenance (or type Control-D to continue)):

This can occur on any OL version and can occur on Exadata or stand-alone systems.

Example from ILOM console:

Boot hangs at:
***** Switching root
Welcome to Oracle Linux Server
Starting udev: [ OK ]
Setting hostname ExaMyDB01.mycompany.com: [ OK ]
Setting up Logical Volume Management: 4 logical volume(s) in volume group "VGExaDb" now active
[ OK ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/mapper/VGExaDb-LVDbSys1
DBSYS: Superblock last mount time (Thu Apr 14 12:41:41 Give root password for maintenance
(or type Control-D to continue):

Note: The console in the ILOM Graphical interface often lags behind the current statement.

It is advised to ssh to the ILOM and start the text console with the command "start /SP/console"

It displays the current statement being executed.

Changes

Customer set the date/time to a point in the future using the command:

# date -s "14 APR 2016 11:11:11" <<< Future date

then changed it back to current date/time.

# date -s "14 APR 2015 11:11:11" <<< Current date

Cause

When changing the stsrtem date/time the Superblock recorded the last date mounted / accessed.

When reset to an earlier date/time and system is rebooted, system hangs on bootup.

Solution

From console:

1. Enter root password to access system

2. More the /etc/fstab to verify FS type:

[root@ExaMyDB01 ~]# more /etc/fstab
LABEL=DBSYS             /                       ext3    defaults        1 1
LABEL=BOOT              /boot                   ext3    defaults,nodev 1 1
LABEL=DBORA             /u01                    ext3    defaults,nodev 1 1
tmpfs                   /dev/shm                tmpfs   defaults,size=257962m 0
devpts                  /dev/pts                devpts gid=5,mode=620 0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP              swap                    swap    defaults        0 0

3. Issue a df -k to get the file systems names

root@ExaMyDB01 ~]# df -k
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
                    340600744 16403128 306899952   6% /
/dev/mapper/VGExaDb-LVDbOra1
                    340600744 16403128 306899952   6% /u01

4. Clean up the file systems (specify correect fsck for filesystem type from /etc/fstab

# fsck.ext3 -v -f /dev/mapper/VGExaDb-LVDbSys1 >> Correct any bad inodes
# fsck.ext3 -v -f /dev/mapper/VGExaDb-LVDbOra1 >> Correct any bad inodes
# fsck.ext3 -v -f /dev/sda1 >> Correct any bad inodes

[root@ExaMyDB01]# fsck.ext3 -v -f /dev/mapper/VGExaDb-LVDbSys1
e2fsck 1.43-WIP (20-Jun-2013)
Superblock last mount time (Thu Apr 14 12:41:41 2016,
       now = Thu Apr 16 13:12:43 2015) is in the future.
Fix<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

     112715 inodes used (0.26%, out of 43253760)
       3526 non-contiguous files (3.1%)
         98 non-contiguous directories (0.1%)
            # of inodes with ind/dind/tind blocks: 15397/426/0
    5458116 blocks used (6.31%, out of 86507520)
          0 bad blocks
          1 large file

      93191 regular files
      13690 directories
          2 character device files
          0 block device files
          1 fifo
       2003 links
       5766 symbolic links (1585 fast symbolic links)
         56 sockets
------------
     114709 files
[root@ExaMyDB01]

5. # sync

6. # reboot

Attachments

This solution has no attachment