FS System: How to Use the Pilot Rescue Image

Asset ID:	1-71-2148252.1
Update Date:	2018-04-19
Keywords:

Solution Type Technical Instruction Sure

Solution 2148252.1 : FS System: How to Use the Pilot Rescue Image

Applies to:

Oracle FS1-2 Flash Storage System - Version 6.2 to 6.2 [Release 6.2]
Information in this document applies to any platform.

Goal

Provide step by step instructions on how to utilize the FS1-2 rescue image. While this document is restricted to Partners, a customer ready outline of the steps is attached.

Solution

CAUTION: Do not attempt to use this process to re-image a Pilot HDD if the FS1 software version is below R6.2.5. Only use the rescue image that comes with the version of software on the FS1-2.

Background

Beginning with R6.2.5, there will be an additional rescue image in the second part of the patch (2of2 or 2of3). The image is initially zipped to reduce its size and needs to be unzipped in order to get to the ISO image itself. The ISO image CANNOT be used to upgrade an FS1-2 via the GUI or CLI. The ISO image, once transferred to a USB "thumb drive", can be used in 3 different ways:

Rescue - Used to mount a problem Pilot's Hard Disk Drive (HDD) and extract information (logs etc) that cannot be otherwise obtained. In some cases it may also allow other repair work that could result in the restoration of Pilot functionality without a re-image.
HDD replacement - Used to re-image a replacement Pilot HDD without any recabling.
Re-image - Used to correct a corrupted image on either one or both Pilot HDDs.

Note: The HDD replacement and re-imaging of a single Pilot HDD are essentially the same. The only difference is whether the Pilot HDD has been replaced. These are covered in the same procedure below. Re-imaging both Pilots during the same activity is a separate procedure as it requires certain configuration files. It may be possible to extract them from the Pilots using the Rescue method or recent log bundles.

Creating a Bootable Image on a USB Drive

Requirements:

USB Drive 8GB or larger. If it also has an activity LED, IO can be confirmed during the process.
ISO formatted Rescue Image.
Software to transfer the ISO image to the USB drive and make it bootable. Rufus 2.8 was used for the purposes of this document (https://rufus.akeo.ie), but newer versions are available.

Note: It has been reported that version 2.12 has permission problems with Windows 7.

Transfer ISO image to USB drive:

Install rufus per the instructions on their web page.
Determine the version of software being recovered.
Download the 2of2 part of that patch and extract the ISO rescue image using unzip or similar program.
Write the rescue image onto the USB drive:
1. Insert USB drive into any USB port on your Windows system. If you get any pop-ups asking for it to be encrypted, answer No.
2. Launch the rufus application. It should automatically find the USB drive inserted in the previous step. If not, use the Device pull-down at the top to select it.
3. Use the Select Image Icon to load the rescue image:
4. After loading the rescue image, clear out anything in the New volume label (typically it is a single dot), the image type should update to ISO, and the rescue image name should appear at the bottom of the window:
5. Click on the Start button. You will receive a pop-up indicating that all data on the USB drive will be destroyed - click OK to proceed.
  
  Note: It takes Rufus about 15-30 minutes to create the bootable ISO image on the USB drive depending on the laptop and USB drive that are used.

CAUTION: the default operation of the rescue image is to re-image the HDD if the Pilot is rebooted with a rescue image USB drive installed. There will be a 1 minute delay before this happens to allow the user to change this behavior. At this time changing this can only be accomplished by using a keyboard and monitor connected to the Pilot.

Using the Rescue Option

At this time this option requires a monitor and keyboard to successfully implement.

If not already done, power off the Pilot that needs to be re-imaged.

This can be accomplished by either removing both power cords or
By pressing and holding the OK button in front on the left side for 5-7 seconds until the OK LED changes from solid to a slow blink (every 3-5 seconds).

Insert USB drive with rescue image into a USB port on the bad Pilot.
Attach a VGA monitor and USB keyboard to the bad Pilot.
Boot the Pilot by either re-inserting the power cords or pressing the OK button.
When the screen below is observed, press the F8 key to enter the BIOS menu:
From the BIOS menu, select the USB drive with the rescue image on it:
Select the Rescue Flash Storage System option:
The rescue image will prompt for various inputs:
1. Choose a Language:
2. Keyboard Type:
3. Rescue Method:
4. Select Partition:
5. Setup Network:
6. Mount Linux partition:
7. Select partition to mount:
8. Status of mount:
9. Mounted partition confirmation:
10. Start Shell:
Logs or other files can be copied from the mounted Pilot HDD to the USB drive or attempts to repair the Pilot HDD can be made.
Upon conclusion of these activities, reboot the Pilot and remove the USB drive. If a re-image of the Pilot is required, proceed with the steps below.

Using the Re-image Option on a Single Pilot or Pilot HDD Replacement

Insert USB Drive with rescue image into a USB port in the front of the good Pilot. In this example, Pilot 2.
ssh as root to the good Pilot - reference Documents 2029847.1 FS System: How to Enable SSH Access to the Pilot & 2046703.1 FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System.
Mount the USB drive
[root@pilot2 ~]# mkdir /mnt/rescue
[root@pilot2 ~]# mount /dev/sdb1 /mnt/rescue
[root@pilot2 ~}# cd /mnt/rescue/fs_config

Note: X5-2 Pilots may have an internal USB Drive for Oracle System Assistant (OSA). This will result in the device incrementing to sdc1.
Save the config:
[root@pilot2 fs_config]# ./config_mgr buddygm
copy /etc/system_serial_number
copy /etc/system_baseWWN
copy /etc/ntp.conf
copy /etc/adjtime
copy /etc/localtime
copy /etc/nodenames
copy /etc/passwd
copy /etc/resolv.conf
copy /etc/sysconfig/clock
copy /etc/sysconfig/iptables_fs1
copy /etc/shadow
copy /etc/pki/tls/private/localhost.key
copy /etc/pki/tls/certs/localhost.crt
copy /etc/pki/tls/private/server.key
copy /etc/pki/tls/certs/server.crt
copy /etc/udev/rules.d/70-persistent-net.rules
copy /etc/sysconfig/network-scripts/ifcfg-eth0
copy /var/lib/pillar/pcp/pilot-config.xml
copy /var/lib/pillar/pcp/pilot[1..2]
copy /var/lib/axiom/dnsmasq_pmi_hostsfile
[root@pilot2 fs_config]#
cd to the root home directory and unmount the USB drive:

[root@pilot2 fs_config]# cd
root@pilot2 ~]# umount /mnt/rescue
Move the USB drive from the good Plot to one of the front USB ports on the bad Pilot.
Wait 3-5 minutes for the ILOM to update the boot_device list in the BIOS with the USB drive.
Modify the Pilot BIOS to boot from a USB drive:
1. Save the BIOS configuration to a text file:
  [root@pilot2 ~]# echo "changeme" | ubiosconfig export all -U root -H 169.254.2.9 -x buddy_bios_config
  Enter password :
  ********
  Wrote backup of BIOS configuration to 'buddy_bios_config'.
  [root@pilot2 ~]#
2. Edit the BIOS configuration and copy the USB drive entry from the list of boot_devices to the first entry in the boot_order list:
  BEFORE:
  <boot_order>
     <boot_device>
        <description>SAS:PCIE4:E01S00-2F022B0D HGST H101860SFSUN60</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>PXE:NET2:IBA XE Slot 8200 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>PXE:NET3:IBA XE Slot 8201 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
  </boot_order>
  
  
  <boot_devices>
     <boot_device>
        <description>PXE:NET2:IBA XE Slot 8200 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>PXE:NET3:IBA XE Slot 8201 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>SAS:PCIE4:E01S00-2F022B0D HGST H101860SFSUN60</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>USB:USB3:KingstonDT 101 G2 PMAP</description>
        <instance>1</instance>
     </boot_device>
  </boot_devices>
  
  
  AFTER:
  <boot_order>
     <boot_device>
        <description>USB:USB3:KingstonDT 101 G2 PMAP</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>SAS:PCIE4:E01S00-2F022B0D HGST H101860SFSUN60</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>PXE:NET2:IBA XE Slot 8200 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>PXE:NET3:IBA XE Slot 8201 v2320</description>
        <instance>1</instance>
     </boot_device>
  </boot_order>
  
  
  <boot_devices>
     <boot_device>
        <description>PXE:NET2:IBA XE Slot 8200 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>PXE:NET3:IBA XE Slot 8201 v2320</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>SAS:PCIE4:E01S00-2F022B0D HGST H101860SFSUN60</description>
        <instance>1</instance>
     </boot_device>
     <boot_device>
        <description>USB:USB3:KingstonDT 101 G2 PMAP</description>
        <instance>1</instance>
     </boot_device>
  </boot_devices>
3. Import the edited BIOS configuration file back onto the bad Pilot:
  [root@pilot2 ~]# echo "changeme" | ubiosconfig import boot -yx buddy_bios_config -U root -H 169.254.2.9
  Enter password :
  ********
  Preparing to restore XML file to BIOS...
  Done preparing to restore settings in buddy_bios_config to BIOS.
  Restoring BIOS configuration (allow several minutes)..............................................................................................Done.
Power off the Pilot that needs to be re-imaged by removing both power cords. This will also force the updated BIOS from the previous step to be loaded when power is reapplied.

NOTE: If the Pilot is an X5-2 version, move it into the service position, open the top cover and remove the OSA USB drive from the motherboard if one exists. Failure to do so will cause the kickstart to re-image the Pilot to fail as it is hard coded for a singled USB drive. The OSA USB drive does not have to be returned to the motherboard as it does not have a purpose in the FS1-2.
If applicable, replace HDD.
Boot the Pilot by re-inserting the power cords. Re-image of the Pilot should start within 60 seconds and take about 15 minutes to complete.
NOTE: If the Pilot is not back online and the FS1-2 normal/green within 20 minutes, contact Oracle Support for assistance.
Remove USB drive.
Copy Drive firmware packages from the good Pilot to the repaired Pilot and install them:
[root@pilot2 ~]# scp /rpms/AxiomONE-SW-installed/oraclefs-hdd*.rpm pilot1:/rpms/AxiomONE-SW-installed/
oraclefs-hdd-hgst1.6tb-fip-060204-029400.x86_64.rpm 100% 983KB 983.4KB/s 00:00
oraclefs-hdd-hgst1.6tb-nfp-060200-017300.x86_64.rpm 100% 983KB 983.2KB/s 00:00
oraclefs-hdd-hgst400gb-fip-060204-029400.x86_64.rpm 100% 983KB 983.3KB/s 00:00
oraclefs-hdd-hgst400gb-nfp-060200-017300.x86_64.rpm 100% 983KB 983.2KB/s 00:00
oraclefs-hdd-hita004tb-nfp-060204-028100.x86_64.rpm 100% 932KB 932.3KB/s 00:00
oraclefs-hdd-hita008tb-fip-060204-029400.x86_64.rpm 100% 1225KB 1.2MB/s 00:00
oraclefs-hdd-hita008tb-nfp-060202-025800.x86_64.rpm 100% 1274KB 1.2MB/s 00:00
oraclefs-hdd-hita1.2tb-fip-060204-031700.x86_64.rpm 100% 1267KB 1.2MB/s 00:00
oraclefs-hdd-hita300gb-nfp-060200-015700.x86_64.rpm 100% 699KB 699.1KB/s 00:00
oraclefs-hdd-hita900gb-nfp-060200-015700.x86_64.rpm 100% 699KB 699.1KB/s 00:00
oraclefs-hdd-sand1.6tb-nfp-060000-000400.x86_64.rpm 100% 529KB 528.7KB/s 00:00
oraclefs-hdd-sand400gb-nfp-060000-000400.x86_64.rpm 100% 529KB 528.7KB/s 00:00
[root@pilot2 ~]# ssh root@pilot1 rpm -ivh --force /rpms/AxiomONE-SW-installed/oraclefs-hdd*
Preparing... ##################################################
oraclefs-hdd-sand400gb-nfp ##################################################
oraclefs-hdd-sand1.6tb-nfp ##################################################
oraclefs-hdd-hita900gb-nfp ##################################################
oraclefs-hdd-hita300gb-nfp ##################################################
oraclefs-hdd-hita1.2tb-fip ##################################################
oraclefs-hdd-hita008tb-nfp ##################################################
oraclefs-hdd-hita008tb-fip ##################################################
oraclefs-hdd-hita004tb-nfp ##################################################
oraclefs-hdd-hgst400gb-nfp ##################################################
oraclefs-hdd-hgst400gb-fip ##################################################
oraclefs-hdd-hgst1.6tb-nfp ##################################################
oraclefs-hdd-hgst1.6tb-fip ##################################################
[root@pilot2 ~]#
Confirm the Pilot is normal/green.
If an OSA USB drive was removed in step 8 above and the customer is willing, check for and if necessary, remove the OSA USB drive from the active Pilot by executing a failover:
# fscli pilot -forceFailover
Move the Pilot into the service position, unplug both power cords, open the top cover and remove the OSA USB drive from the motherboard.
Return the top cover to the Pilot, plug in both power cords and return the Pilot to its racked position.

Using the Re-image Option to Recover Both Pilots

NOTE: At this time at least one Pilot has to be bootable in order to extract configuration files. Work to obtain this data from a recent log bundle is in progress. This process requires an outage.

Shutdown the FS1-2 and remove the power cords from both Controllers and both Pilots.
Power on the Pilot that is bootable.
Insert USB Drive with rescue image into a USB port in the front of that Pilot. In this example, Pilot 2.
ssh as root to that Pilot - reference Documents 2029847.1 FS System: How to Enable SSH Access to the Pilot & 2046703.1 FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System.
Mount the USB drive
[root@pilot2 ~]# mkdir /mnt/rescue
[root@pilot2 ~]# mount /dev/sdb1 /mnt/rescue
[root@pilot2 ~}# cd /mnt/rescue/fs_config

Note: X5-2 Pilots may have an internal USB Drive for Oracle System Assistant (OSA). This will result in the device incrementing to sdc1.
Save the configuration files:
[root@pilot2 fs_config]# ./config_mgr buddygm
copy /etc/system_serial_number
copy /etc/system_baseWWN
copy /etc/ntp.conf
copy /etc/adjtime
copy /etc/localtime
copy /etc/nodenames
copy /etc/passwd
copy /etc/resolv.conf
copy /etc/sysconfig/clock
copy /etc/sysconfig/iptables_fs1
copy /etc/shadow
copy /etc/pki/tls/private/localhost.key
copy /etc/pki/tls/certs/localhost.crt
copy /etc/pki/tls/private/server.key
copy /etc/pki/tls/certs/server.crt
copy /etc/udev/rules.d/70-persistent-net.rules
copy /etc/sysconfig/network-scripts/ifcfg-eth0
copy /var/lib/pillar/pcp/pilot-config.xml
copy /var/lib/pillar/pcp/pilot[1..2]
copy /var/lib/axiom/dnsmasq_pmi_hostsfile
[root@pilot2 fs_config]#
cd to the root home directory and unmount the USB drive:

[root@pilot2 fs_config]# cd
root@pilot2 ~]# umount /mnt/rescue
At this point, the other Pilot can be recovered (Pilot 1 in this example). Power off Pilot 2.
Attach a monitor and keyboard to Pilot 1.
Insert the USB drive into the front slot of Pilot 1 and plug in the power cords.
When prompted, select F8 to enter the BIOS menu:
Select the USB drive to boot from:
Confirm selection of Reimage Flash Storage Option:
Re-image should take about 15 minutes after which the Pilot will reboot.
Repeat steps to enable SSH and log back into the Pilot.
Check the output of the ver -v command to confirm the version is correct:

[root@pilot1 /]# ver -v

pilot1(Active):
Pilot Apps Build:        060216-054501
Pilot OS version:        060215-052200
OracleFS mfg version:    060215-052200
ebod-7043630 version:    060100-007000
ebod-7044319 version:    060100-007000
Mount the USB drive, rename the buddy Pilot file and unmount the USB drive:

[root@pilot1 fs_config]# cd /mnt/rescue/fs_config/var_files
[root@pilot1 var_files]# ls
dnsmasq_pmi_hostsfile pilot1 pilot-config.xml
[root@pilot1 var_files]# mv pilot1 pilot2
[root@pilot1 var_files]# ls
dnsmasq_pmi_hostsfile pilot2 pilot-config.xml
[root@pilot1 var_files]# cd /
[root@pilot1 /]# umount /mnt/rescue
[root@pilot1 /]#
Power off Pilot (Pilot 1 in this example) and move the USB drive to Pilot 2 and repeat steps 11-16 for Pilot 2.
After Pilot 2 has rebooted confirm the same version:

[root@pilot2 ~]# ver -v

pilot2(Active):
Pilot Apps Build:        060216-054501
Pilot OS version:        060215-052200
OracleFS mfg version:    060215-052200
ebod-7043630 version:    060100-007000
ebod-7044319 version:    060100-007000
Remove the USB drive and power on Pilot 1.
After about 5 minutes, confirm both Pilots (172.30.80.2 = Pilot 1, 172.30.80.3 = Pilot 2) are on the Private Management Interface (PMI) network:

[root@pilot2 ~]# cat /etc/nodenames
172.30.80.3 WN2008fffffffffffa WN2008000101000000 mgmtnode
172.30.80.2 WN2009fffffffffff2
Check the node matrix in pcp.log to confirm the status of the Pilots (the middle digits within all of the parenthesis should be a 6):

[root@pilot2 ~]# egrep "pcp:info.*(2: | 3: )" /var/log/pcp.log | tail -n 2
2018-03-20 17:24:16.844 pilot1 pilotcfgproc: 21210941 12161 pcp:info 3: 1 ( 4 6 0)( 20 6 0)
2018-03-20 17:24:16.844 pilot1 pilotcfgproc: 21210943 12161 pcp:info 2: 1 ( 0 6 0)( 20 6 0)
[root@pilot2 ~]#

This may need to be checked more than once before the proper output is confirmed.
Once the Pilots are booted with their software running, power on Controllers.
After about 12 minutes, the Controller should also show up on the PMI:

[root@pilot2 ~]# cat /etc/nodenames
172.30.80.3 WN2009fffffffffffa WN2008000101000000 mgmtnode
172.30.80.2 WN2008fffffffffff2
172.30.80.128 WN508002000####### WN2008000101000001
172.30.80.129 WN508002000#######
Monitor the node matrix again to confirm that the Controllers have successfully booted:

[root@pilot2 ~]# tailf /var/log/pcp.log | grep -A 5 "node matrix"
2018-03-20 17:47:34.356 pilot2 pilotcfgproc: 14104918 12977 pcp:info fofb: node matrix
2018-03-20 17:47:34.356 pilot2 pilotcfgproc: 14104919 12977 pcp:info node           3          2        128        129
2018-03-20 17:47:34.356 pilot2 pilotcfgproc: 14104920 12977 pcp:info    3:   1 ( 4 6 0)( 0 6 0)( 20 6 0)( 20 6 0)
2018-03-20 17:47:34.356 pilot2 pilotcfgproc: 14104921 12977 pcp:info    2:   1 ( 0 6 0)( 0 6 0)( 20 6 0)( 20 6 0)
2018-03-20 17:47:34.356 pilot2 pilotcfgproc: 14104922 12977 pcp:info 128:   1 ( a0 6 0)( a0 6 0)( c0 6 0)( 80 6 0)
2018-03-20 17:47:34.356 pilot2 pilotcfgproc: 14104923 12977 pcp:info 129:   1 ( 20 6 0)( 20 6 0)( 0 6 0)( 0 6 0)

References

<BUG:27685730> - UNABLE TO KICKSTART PILOT IIMAGE ON HDD AFTER REPLACEMENT
<BUG:27676164> - ASSIST TSC: UNABLE TO UPGRADE FS1-2.

Attachments

This solution has no attachment