Oracle FS1-2 Flash Storage System
 

PLA-Support>Sun Systems>DISK>Flash Storage>SN-EStor: FSx
 

In this Document

Symptoms

Cause

Solution

References

Applies to:

Symptoms

If the timezone in the FS1 Pilot is set to anything but UTC, the time stamps for events, alerts, and log collections in both the GUI and the system logs will vary depending on which software component in the system generates the event or the log collection. The amount of time variation will depend on the difference between UTC and the configured timezone.  This can result in Events appearing out of order in the Event Log and screen, Alerts appearing out of order, and log collection timestamps and file names being out of order.  

Later log collections such as manual logs may appear to have been taken hours before or after an event driven log collection that they are related to.  Event driven log collection times may appear out of sequence depending on whether the initiating event comes from a Controller or the Pilot software. A typical symptom is that times in the Event Log will be outside the boundaries of the log collection file name. 

For example a log collection AK00999999-151119221311-151119223713-COLD_START_FAILED-e-02-02.tar which begins at 2015-Nov-19-22:13:11 and ends at 2015-Nov-19-22:37:13 may have event time stamps that occur before or after those boundaries. This makes problem determination extremely difficult as the times for system events and log collections cannot be used to determine a sequence of issues and recoveries without checking for operating system times in log filenames and reorganizing all logs to provide the correct sequence. 

On the Pilot shell, a date command must indicate that the time is in UTC.  If it does not, the timezone is wrong and must be corrected.

Cause

There are two causes that would contribute to an incorrect timezone being set:

• An administrator has used the Pilot shell to change the timezone from UTC. This should never be done.

• Pilot HDD replacement or Pilot HDD reimaging leaves an incorrect timezone on the newly imaged Pilot HDD. If this happens, the timezone issue will only occur if that Pilot becomes active. In other words, the problem will appear only after a Pilot failover, software upgrade, or system restart. The timezone information should be checked on the re-imaged Pilot.  

        Bug 22241151 notes the issue of Pilot Kickstart leaving bad timezone information.

        Bug 22233332 notes the issue of intentionally changed timezone.

        Both affect all releases below their fixed release, which includes 6.2.0 and all 6.1.x.

Solution

Contact Oracle Customer Support for assistance to verify and resolve the issue.  The System Event Log should be cleared to prevent old events from reappearing.  Optionally, all existing log collections should be cleared also to avoid confusion. 

The log collections and events can be archived if desired before they are deleted. 

Procedure to assist TSC with changing the localtime/timezone on each Pilot

 Enable SSH or use a KBD/Display to access the Pilot shell. Check both Pilots. Kickstart will tend to set the clock file to Los_Angeles, resulting in PST or PDT on the output of a date command. 

1. Check the date on both Pilots. It must indicate UTC:
   [root@pilot2 ~]# date
   Fri Nov 20 17:58:11 UTC 2015
2. Check /etc/localtime on both Pilots.  It must be a symlink to /usr/share/zoneinfo/UTC  
   [root@pilot2 ~]# ls -l /etc/localtime
   /etc/localtime -> /usr/share/zoneinfo/UTCCheck the /etc/sysconfig/clock file to make sure it has the ZONE="UTC" entry.  There should be two lines, where the line containing Los_Angeles is commented out.
   1. If localtime is not a symlink, remove the localtime file and create the proper symlink:
      [root@pilot2 ~]# cd /etc
      [root@pilot2 ~]# rm -f localtime
      [root@pilot2 ~]# ln -s /usr/share/zoneinfo/UTC localtime
   2. Check that the symlink is correct with ls -l /etc/localtime
      [root@pilot2 ~]# ls -l /etc/localtime
      /etc/localtime -> /usr/share/zoneinfo/UTC
3. If either Pilot has the correct entry, copy the clock file from that Pilot to the one with the wrong entry.   
   [root@pilot2 ~]# cat /etc/sysconfig/clock
   #ZONE="America/Los_Angeles" # Removed for Axiom Software (oraclefs-mfg)
   ZONE="UTC" # Added for Axiom Software (oraclefs-mfg) 
4. Repeat all checks on the other Pilot.
5. Reboot both Pilots and repeat all checks to make sure the timezone is still correct. 
6. Clear the Event Log.  If you do not clear the Event Log, you will see new events appear, but will also see old events that are out of sequence appear as if they are new events.  If you need to save the Event Log, perform a log collection and download it. 
7. Strongly Recommended.  Delete all log collections.  If you need to preserve them, download all collections to convenient storage. 
8. Check NTP.  You may need to reboot the Controllers to get them on the correct time if NTP is configured. fscli time -list -details may help. 

References

This solution has no attachment