Oracle FS1-2 Flash Storage System
 

PLA-Support>Sun Systems>DISK>Flash Storage>SN-EStor: FSx
 

In this Document

Goal

Solution

Procedure

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: customers should not try this procedure without receiving approval from Support. The content should only be shared if WebEx is not possible.

Applies to:

Goal

FS1 release 06.02.04 and higher will not accept any drives as replacements or as enclosure add-ons if the Pilots do not have the appropriate drive firmware package installed. On the other hand, release 06.02.04 and higher support non-disruptive drive firmware updates, but ONLY for drives that are not physically installed in the system. Some customers ignore Patch READMEs regarding installation of drive firmware to prevent problems, or the advice that if there is a disruptive firmware update, always install all of the firmware packages as it adds only a few minutes to the firmware update and can prevent not being able to replace a drive.

There will be instances where the firmware is not installed, and a DE add or a replacement drive is received where there is no firmware for it. The drives will NOT be allowed to be incorporated into the system.

For example, an FS1 on 06.02.09 with 2 enclosures of SSD 1.6TB drives with the old firmware (S311) lost a drive. A drive with the same part number and firmware was dispatched but the customer reported that the new drive is reported as UNSUPPORTED on the FS1 GUI. It was found under Support -> Software and Firmware that no drive firmware packages were installed (the same can be verified in the logs under file pcp_runtime_info.log.* -> Installed S/W versions section).

Solution

Use the appropriate solution based on the following rules:

• The FS1 is in Warning as a drive is missing/failed/unsupported (like in the above example): due to the Warning state, the drive firmware package cannot be installed using the FS1 GUI. Use the procedure in this article.
• The FS1 is in Warning as a new enclosure was added and all the drives are showing as unsupported (for more details, see Document 2190111.1 FS System: Newly added drives in a Drive Enclosure display a status of UNSUPPORTED): due to the Warning state, the drive firmware package cannot be installed using the FS1 GUI.
  There are two solutions:
• • Disconnect the DE to get the FS1 back to a Normal state and open the README link from Document 1967797.1 FS System: How to Download Software and Firmware Updates for the FS1-2 to install a drive firmware package on the pilots without downtime. Beware of the risks associated with disconnecting cables and cabling mistakes.
  • Follow this procedure (not tested on DE add).
    
    
• The FS1 is in a Normal state and the customer is planning to add new enclosures for a drive type that is already physically installed on the FS1. The current drives have the old firmware and the customer would like to get the new enclosures with the latest drive firmware burnt on the fly (no downtime). This is not possible to achieve using the FS1 GUI as the FS1 would reboot to upgrade the firmware on the drives that are in use. Use this procedure as it does not require a downtime: the old firmware package will be uninstalled and the new firmware package will be installed on the Pilots and the drives in use with the old firmware will still be supported (they can be upgraded another time using the GUI).
• Easiest case, the FS1 is in a Normal state and the customer is planning to add new enclosures for a drive type that is not physically installed on the FS1 and the relevant drive firmware package is not installed on the pilots:
  Open the README link from Document 1967797.1 FS System: How to Download Software and Firmware Updates for the FS1-2 to install a drive firmware package on the Pilots without downtime

Procedure

Detailed action plan assuming that the Active Pilot is Pilot1 (the customer can also verify which Pilot is active on the FS1 GUI, under the System tab -> Hardware -> Pilots). Perform a Pilot failover using fscli or rewrite the action plan if Pilot2 is active.

1. In case the issue is similar to the example of the unsupported drive, physically remove the drive (this is optional as a test in the lab shown that it was not necessary to remove the unsupported drive).
2. Download FSCLI.  See Document 1991938.1 FS System: How to obtain and install the fscli tool software.
3. Enable SSH on the public IP address.  See Document 2029847.1 FS System: How to Enable SSH Access to the Pilot:
   
   # fscli login -u administrator -oraclefs 10.0.0.2
   # fscli system -modify -enableSsh 120
   # fscli logout
    
4. Open PuTTY on the physical IP address of the active Pilot.  Set PuTTY to capture the output into a file to be safe.
5. Run the following commands to keep ssh open on the Pilots during this procedure:
   
   [root@pilot1 ~]# /usr/local/sbin/sshConfig enable
   [root@pilot1 ~]# ssh pilot2 /usr/local/sbin/sshConfig enable
    
   1. Stop Pilot management service (the GUI will no longer be available until the end of the procedure but the FS1 will continue to serve data) by running the following commands.  From the active Pilot use ssh to stop it on the standby Pilot, then on itself:
      
      [root@pilot1 ~]# ssh pilot2 service pilotcfg stop
      [root@pilot1 ~]# service pilotcfg stop
       
   2. Check to see if there are any drive firmware packages installed on Pilot1:
      
      [root@pilot1 ~]# rpm -qa | egrep oraclefs-hdd
       
   3. If the above command produces any output, remove the package with the -e (erase) option.  This example is for a Hitachi 8TB FIPS drive firmware package:
      
      [root@pilot1 ~]# rpm -e --nodeps oraclefs-hdd-hita008tb-fip-060208-034100.x86_64
       
      
   4. Repeat the erase command for all the other packages listed in the output.
   5. Confirm all drive firmware packages have been removed:
      
      [root@pilot1 ~]# rpm -qa | egrep oraclefs-hdd
       
   6. Check to see if there are any drive firmware packages in the SW-installed directory:
      
      [root@pilot1 ~]# ls -l /rpms/AxiomONE-SW-installed/oraclefs-hdd*
       
      If any drive firmware packages are found, remove them:
      
      [root@pilot1 ~]# rm -f /rpms/AxiomONE-SW-installed/oraclefs-hdd*
       
   7. Check to see if there are any directories containing drive firmware and or compatibility files:
      
      [root@pilot1 ~]# ls -l /var/images/pds/drivefw/
       
      If any are found, remove them:
      
      [root@pilot1 ~]# rm -rf /var/images/pds/drivefw/*
       
6. ssh to the other Pilot (Pilot 2 in this example) and repeat the above steps to remove all the drive firmware files and packages from that Pilot.
7. Copy the drive firmware packages from the staged to the installed directory:
   1. Get the number of drive firmware packages staged:
      
      [root@pilot2 ~]# ls -l /rpms/AxiomONE-SW-staged/* | egrep -c oraclefs-hdd
      21
       
   2. Copy the files from staged to installed:
      
      [root@pilot2 ~]# cp /rpms/AxiomONE-SW-staged/oraclefs-hdd* /rpms/AxiomONE-SW-installed/
       
   3. Confirm the count is the same:
      
      [root@pilot2 ~]# ls -l /rpms/AxiomONE-SW-installed/* | egrep -c oraclefs-hdd
      21
       
      
      Note: The number of drive firmware packages (21 in this example) might change in future. Always verify the README of the latest GA code available at Document 1967797.1 FS System: How to Download Software and Firmware Updates for the FS1-2
       
      
8. Install the drive firmware packages and verify the count is correct:
   
   [root@pilot2 ~]# rpm -i --force /rpms/AxiomONE-SW-installed/oraclefs-hdd*.rpm
   [root@pilot2 ~]# rpm -qa | egrep -c oraclefs-hdd
   21
   [root@pilot2 ~]# ls -l /var/images/pds/drivefw/ | egrep -v total | wc -l
   21
    
9. Exit back to active Pilot (Pilot 1 in this example) and repeat previous 2 steps to install drive firmware packages on Pilot1.
10. Restart the Pilot management service:
    
    [root@pilot1 ~]# service pilotcfg start
     
11. Wait 2 minutes and run this command:
    
    [root@pilot1 ~]# cat /etc/nodenames
     
    one of the lines would be "172.30.80.2 WN2008fffffffffff2" (not ready) or "172.30.80.2 WN2008fffffffffff2 WN2008000101000000 mgmtnode" (ready)
     
12. Repeat step 11 until Pilot (Pilot 1 in this example) shows mgmtnode:
    
    [root@pilot2 ~]# cat /etc/nodenames
    172.30.80.2 WN2009fffffffffff2 WN2009000101000000 mgmtnode
    172.30.80.129 WN508002000158ba51
    172.30.80.128 WN508002000158ba50 WN2008000101000001
     
13. Restart Pilot management service on standby Pilot (Pilot 2 in this example):
    
    [root@pilot1 ~]# ssh pilot2 service pilotcfg start
     
14. After about 5 minutes reopen the FS1 GUI and insert the drive back in the enclosure (if it was removed in Step 1). The firmware of the drive will be updated (see the Events).
15. Go to the System tab -> Alerts and Events -> System Alerts.  If there is a Foreign Disk Drive, double click on the alert, make sure it is only for the drive that was unsupported, tick the "Accept New Drives" and click OK.
16. The FS1 should be Normal.
    
    Note: A drive that has been removed and reinserted from an enclosure will have its firmware upgraded. This can be done on SSD HotSpares without degrading the RAID levels.
     
17. Disable ssh access on both Pilots:
    
    [root@pilot1 ~]# /usr/local/sbin/sshConfig disable
    [root@pilot1 ~]# ssh pilot2 /usr/local/sbin/sshConfig disable

References

This solution has no attachment