FS System: How to Collect an ILOM Snapshot from the Pilot or Controller

Asset ID:	1-71-1963071.1
Update Date:	2018-05-15
Keywords:

Solution Type Technical Instruction Sure

Solution 1963071.1 : FS System: How to Collect an ILOM Snapshot from the Pilot or Controller

Applies to:

Oracle FS1-2 Flash Storage System - Version All Versions and later
Information in this document applies to any platform.

Goal

The Pilots include the script /usr/local/sbin/ilomsnapshot.pl which gives the possibility to collect an ILOM snapshot from the Pilots or the Controllers. Here is the usage for this script:

Usage:

ilomsnapshot.pl -t <target> -d <data_set> [-c [ontroller]]

-t: [self/buddy] Get snapshot from self of buddy
-d: [normal/full/normal-logonly or full-logonly] Data type
-c: [172.30.80.128/129] Controller IP address if get snapshot from controllers
-h: Help Page

e.g. Get normal ILOM Snapshot from Pilot itself:
    ilomsnapshot.pl -t self -d normal

e.g. Get normal ILOM Snapshot from buddy Pilot :
    ilomsnapshot.pl -t buddy -d normal

e.g. Get normal ILOM snapshot from Controller node 172.30.80.128 itself:
    ilomsnapshot.pl -c 172.30.80.128 -t self -d normal

e.g. Get full ILOM snapshot from Controller node 172.30.80.129 via his buddy (i.e. node node 172.30.80.128). Useful if 172.30.80.129's host is down:
    ilomsnapshot.pl -c 172.30.80.128 -t buddy -d full

If the FS1-2 has the release 6.1.x, you will need to upload special binaries and libraries into the FS1. Please refer to the section "Procedure for the release 6.1.x" below.
If the FS1-2 has any release between 6.2.1 and 6.2.8, you will need a special library which contains the fix for the <Bug 20391799>. Please refer to the section "Procedure for the releases 6.2.1 through 6.2.8" below.
If the FS1-2 has the release 6.2.9 or above, please refer to the section "Procedure for the releases 6.2.9 and above" below.

Solution

Procedure for the release 6.1.x

Note: Any Controller reboot or FS1-2 shutdown/restart will require the following steps 1-12 to be re-run. These steps 1-12 require you to download the file "newsnapshot.tar" attached to this document.

Note: All of these commands assume that the active Pilot is Pilot1.

Using WinSCP (or "scp" from Solaris or Linux), copy new tar file "newsnapshot.tar" to the active Pilot and into the /var/tmp/SNAP directory.

Note: You will need to create the directory "SNAP" in /var/tmp.

Example:
bash-3.2$ scp newsnapshot.tar root@<FS1-2 IP>:/var/tmp/SNAP
root@<FS1-2 IP>'s password:
newsnapshot.tar
bash-3.2$
ssh to the active Pilot and cd to /var/tmp/SNAP.
- Please refer to the <Document 2029847.1> FS System: How to Enable SSH Access to the Pilot if ssh is disabled.
- Please refer to the <Document 2046703.1> FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System for the default password.
untar the newsnapshot.tar:

Example:
[root@pilot1 ~]# cd /var/tmp/SNAP
[root@pilot1 SNAP]# tar xvf newsnapshot.tar
ilomsnapshot.pl
libpsg_ilom_snapshot.so
README.txt
[root@pilot1 SNAP]#
Rename the current ilomsnapshot.pl file:

Example:
[root@pilot1 SNAP]# mv /usr/local/sbin/ilomsnapshot.pl /usr/local/sbin/ilomsnapshot.pl.orig
[root@pilot1 SNAP]#
Copy the new ilomsnapshot.pl into place:

Example:
[root@pilot1 SNAP]# cp ilomsnapshot.pl /usr/local/sbin/
[root@pilot1 SNAP]#
Change permissions to 755:

Example:
[root@pilot1 SNAP]# chmod 755 /usr/local/sbin/ilomsnapshot.pl
[root@pilot1 SNAP]#
Rename the ilomsnapshot.pl file on the standby pilot:

Example:
[root@pilot1 SNAP]# ssh root@pilot2 mv /usr/local/sbin/ilomsnapshot.pl /usr/local/sbin/ilomsnapshot.pl.orig
[root@pilot1 SNAP]#
Copy the new ilomsnapshot.pl on the standby Pilot:

Example:
[root@pilot1 SNAP]# scp ilomsnapshot.pl pilot2:/usr/local/sbin/ilomsnapshot.pl
[root@pilot1 SNAP]#
Change the permissions too:

Example:
[root@pilot1 SNAP]# ssh root@pilot2 chmod 755 /usr/local/sbin/ilomsnapshot.pl
[root@pilot1 SNAP]#
Rename current libraries on both Controllers:

Example:
[root@pilot1 SNAP]# ssh root@172.30.80.128 mv /usr/lib64/libpsg_ilom_snapshot.so.1 /usr/lib64/libpsg_ilom_snapshot.so.1.orig
[root@pilot1 SNAP]# ssh root@172.30.80.129 mv /usr/lib64/libpsg_ilom_snapshot.so.1 /usr/lib64/libpsg_ilom_snapshot.so.1.orig
[root@pilot1 SNAP]#
Use scp to copy the libraries to the Controllers and rename:

Example:
[root@pilot1 SNAP]# scp libpsg_ilom_snapshot.so 172.30.80.128:/usr/lib64/libpsg_ilom_snapshot.so.1
[root@pilot1 SNAP]# scp libpsg_ilom_snapshot.so 172.30.80.129:/usr/lib64/libpsg_ilom_snapshot.so.1
[root@pilot1 SNAP]#
Change the permissions to 755:

Example:
[root@pilot1 SNAP]# ssh root@172.30.80.128 chmod 755 /usr/lib64/libpsg_ilom_snapshot.so.1
[root@pilot1 SNAP]# ssh root@172.30.80.129 chmod 755 /usr/lib64/libpsg_ilom_snapshot.so.1
[root@pilot1 SNAP]#
Then you can execute one of the following commands to collect an ILOM snapshot:
- To collect a Controller 1 ILOM snapshot from itself:
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d normal
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d full
- To collect a Controller 2 ILOM snapshot from Controller 1: (this is useful if the Controller 2 host is down)
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d normal
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d full
  
  
  Note: The results are located under /var/images/tds/slammer/<Controller WWN which executed the command>/snapshot/ in the Pilot from where you executed the script.
  
  
  Note: The commands to collect a Pilot ILOM snapshot are not listed because they do not work at this time. This document will be updated with the commands once they work.
  
  Note: The recommendation is to use the argument "-d full" instead of "-d normal". The full option collects additional information like the hwdiag or the fruid. Within the fruid, we find the showpsnc.out which might be important if a Controller does not restart after some type of part replacement.
  
  
  Example:
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d full
  Getting snapshot...
  Waiting for ILOM snapshot(60s)...0
  Waiting for ILOM snapshot(60s)...1
  Waiting for ILOM snapshot(60s)...2
  Waiting for ILOM snapshot(60s)...3
  Waiting for ILOM snapshot(60s)...4
  Waiting for ILOM snapshot(60s)...5
  Waiting for ILOM snapshot(60s)...6
  Waiting for ILOM snapshot(60s)...7
  sending incremental file list
  snapshot_self/
  snapshot_self/fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T08-02-53.zip
  
  sent 2106875 bytes received 35 bytes 4213820.00 bytes/sec
  total size is 2106435 speedup is 1.00
  Finished Snapshot Collection.
  
  [root@pilot1 ~]#
  
  [root@pilot1 ~]# ls -l /var/images/tds/slammer/508002000158C370/snapshot/snapshot_self/
  total 4096
  -rw-r--r-- 1 root root 2075697 Oct 29 07:29 fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T07-23-16.zip
  -rw-r--r-- 1 root root 2106435 Oct 29 08:10 fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T08-02-53.zip
  [root@pilot1 ~]#

Procedure for the releases 6.2.1 through 6.2.8

Download the file lib.tar.gz from this document. It is needed on the FS1 Controllers only.
Enable ssh to the FS1. Refer to the <Document 2029847.1> FS System: How to Enable SSH Access to the Pilot if ssh is disabled.
Copy the file lib.tar.gz into the FS1 under /tmp, refer to the <Document 2046703.1> FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System for the root password:

# scp lib.tar.gz root@<FS1 IP address>:/tmp
ssh to the FS1:

# ssh -l root <FS1 IP address>
Install the library: (The following example shows the Pilot 1 as being the active Pilot)

[root@pilot1 ~] cd /tmp
[root@pilot1 ~] tar xvfz lib.tar.gz
[root@pilot1 tmp]# ssh 172.30.80.128 mv /usr/lib64/libpsg_ilom_snapshot.so.1 /usr/lib64/libpsg_ilom_snapshot.so.1.orig
[root@pilot1 tmp]# ssh 172.30.80.129 mv /usr/lib64/libpsg_ilom_snapshot.so.1 /usr/lib64/libpsg_ilom_snapshot.so.1.orig
[root@pilot1 tmp]# scp libpsg_ilom_snapshot.so.1 root@172.30.80.128:/usr/lib64/
[root@pilot1 tmp]# scp libpsg_ilom_snapshot.so.1 root@172.30.80.129:/usr/lib64/
[root@pilot1 tmp]# ssh 172.30.80.128 chmod 755 /usr/lib64/libpsg_ilom_snapshot.so.1
[root@pilot1 tmp]# ssh 172.30.80.129 chmod 755 /usr/lib64/libpsg_ilom_snapshot.so.1
Collect the ILOM snapshot for a Controller:

Note: The results are located under /var/images/tds/slammer/<Controller WWN which executed the command>/snapshot/ in the Pilot from where you executed the script.

Note: The recommendation is to use the argument "-d full" instead of "-d normal". The full option collects additional information like the hwdiag or the fruid. Within the fruid, we find the showpsnc.out which might be important if a Controller does not restart after some type of part replacement.
- To collect a Controller 1 ILOM snapshot from itself:
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d full
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d normal
  
  Example:
  
  [root@pilot1 tmp]# ilomsnapshot.pl -c 172.30.80.128 -t self -d normal
  Getting snapshot...
  Waiting for ILOM snapshot(180s)...0
  Waiting for ILOM snapshot(180s)...1
  sending incremental file list
  snapshot_self/
  snapshot_self/fs1-2-sin06-a-controller1-sp_1345NM201A_2016-09-26T05-30-36.zip
  sent 1820216 bytes received 35 bytes 3640502.00 bytes/sec
  total size is 1819812 speedup is 1.00
  Finished Snapshot Collection.
  [root@pilot1 tmp]#
- To collect a Controller 2 ILOM snapshot from Controller 1: (this is useful if the Controller 2 host is down)
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d full
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d normal
  
  Example:
  
  [root@pilot1 snapshot]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d normal
  Getting snapshot...
  Waiting for ILOM snapshot(180s)...0
  Waiting for ILOM snapshot(180s)...1
  sending incremental file list
  snapshot_buddy/
  snapshot_buddy/fs1-2-sin06-a-controller2-sp_1345NM2019_2016-09-26T06-12-25.zip
  sent 2123287 bytes received 35 bytes 4246644.00 bytes/sec
  total size is 2122846 speedup is 1.00
  Finished Snapshot Collection.
  [root@pilot1 snapshot]#
Collect the ILOM snapshot for a Pilot:

Note: The results are located under /var/tmp/snapshot/ in the Pilot from where you executed the script.


Note: The recommendation is to use the argument "-d full" instead of "-d normal". The full option collects additional information like the hwdiag or the fruid. Within the fruid, we find the showpsnc.out which might be important if a Controller does not restart after some type of part replacement.


Note: It has been seen during the tests that the first run might complete in 2-3 sec although it does not generate any snapshot. If this happens, you simply need to execute the command once again.
- To collect a Pilot 2 ILOM snapshot from itself:
  
  [root@pilot2 self]# ilomsnapshot.pl -t self -d full
  [root@pilot2 self]# ilomsnapshot.pl -t self -d normal
  
  Example:
  [root@pilot2 self]# ilomsnapshot.pl -t self -d full
  Snapshot log is at /var/tmp/snapshot/self
  Getting snapshot...
  Connected. Use ^D to exit.
  -> set /SP/diag/snapshot dataset=full
  set: Busy
  -> set /SP/diag/snapshot dump_uri=ftp://root:a1s2d3f$@169.254.2.6/snapshot/self
  Collecting a "full" dataset may reset the host. Are you sure (y/n)? y
  set: Busy
  
  -> Session closed
  Command Failed
  ILOM Snapshot is submitting...0
  ILOM Snapshot is submitting...20
  ILOM Snapshot is submitting...40
  ILOM Snapshot is submitting...60
  ILOM Snapshot is submitting...80
  ILOM Snapshot is submitting...100
  ILOM Snapshot is submitting...120
  ILOM Snapshot is submitting...140
  ILOM Snapshot is submitting...160
  ILOM Snapshot is submitting...180
  ILOM Snapshot is submitting...200
  ILOM Snapshot is submitting...220
  ILOM Snapshot is submitting...240
  ILOM Snapshot is submitting...260
  ILOM Snapshot submit finished
  [root@pilot2 self]#
- To collect a Pilot 1 ILOM snapshot from Pilot 2:
  
  [root@pilot2 self]# ilomsnapshot.pl -t buddy -d normal
  
  Example:
  
  [root@pilot2 self]# ilomsnapshot.pl -t buddy -d normal
  Snapshot log is at /var/tmp/snapshot/buddy
  Getting snapshot...
  Connected. Use ^D to exit.
  -> set /SP/diag/snapshot dataset=normal
  set: Busy
  -> set /SP/diag/snapshot dump_uri=ftp://root:a1s2d3f$@169.254.2.10/snapshot/buddy
  set: Busy
  
  -> Session closed
  Command Failed
  ILOM Snapshot is submitting...0
  ILOM Snapshot is submitting...20
  ILOM Snapshot is submitting...40
  ILOM Snapshot is submitting...60
  ILOM Snapshot is submitting...80
  ILOM Snapshot is submitting...100
  ILOM Snapshot is submitting...120
  ILOM Snapshot is submitting...140
  ILOM Snapshot is submitting...160
  ILOM Snapshot is submitting...180
  ILOM Snapshot is submitting...200
  ILOM Snapshot is submitting...220
  ILOM Snapshot is submitting...240
  ILOM Snapshot is submitting...260
  ILOM Snapshot is submitting...280
  ILOM Snapshot is submitting...300
  ILOM Snapshot submit finished
  [root@pilot2 self]#
Logout from the Pilot.
Transfer the snapshot file out of the Pilot:

# scp root@<FS1-2 IP address>:<file> .

Example:

# scp root@<FS1-2 IP>:/var/images/tds/slammer/508002000158C370/snapshot/snapshot_self/fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T07-23-16.zip ~/Desktop/
root@10.186.86.30's password:
fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T07-23-16.zip
#

Procedure for the releases 6.2.9 and above

Enable ssh to the FS1. Refer to the <Document 2029847.1> FS System: How to Enable SSH Access to the Pilot if ssh is disabled.
ssh to the FS1:

# ssh -l root <FS1 IP address>
Collect the ILOM snapshot for a Controller:

Note: The results are located under /var/images/tds/slammer/<Controller WWN which executed the command>/snapshot/ in the Pilot from where you executed the script.

Note: The recommendation is to use the argument "-d full" instead of "-d normal". The full option collects additional information like the hwdiag or the fruid. Within the fruid, we find the showpsnc.out which might be important if a Controller does not restart after some type of part replacement.
- To collect a Controller 1 ILOM snapshot from itself:
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d full
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t self -d normal
  
  Example:
  
  [root@pilot1 tmp]# ilomsnapshot.pl -c 172.30.80.128 -t self -d normal
  Getting snapshot...
  Waiting for ILOM snapshot(180s)...0
  Waiting for ILOM snapshot(180s)...1
  sending incremental file list
  snapshot_self/
  snapshot_self/fs1-2-sin06-a-controller1-sp_1345NM201A_2016-09-26T05-30-36.zip
  sent 1820216 bytes received 35 bytes 3640502.00 bytes/sec
  total size is 1819812 speedup is 1.00
  Finished Snapshot Collection.
  [root@pilot1 tmp]#
- To collect a Controller 2 ILOM snapshot from Controller 1: (this is useful if the Controller 2 host is down)
  
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d full
  [root@pilot1 ~]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d normal
  
  Example:
  
  [root@pilot1 snapshot]# ilomsnapshot.pl -c 172.30.80.128 -t buddy -d normal
  Getting snapshot...
  Waiting for ILOM snapshot(180s)...0
  Waiting for ILOM snapshot(180s)...1
  sending incremental file list
  snapshot_buddy/
  snapshot_buddy/fs1-2-sin06-a-controller2-sp_1345NM2019_2016-09-26T06-12-25.zip
  sent 2123287 bytes received 35 bytes 4246644.00 bytes/sec
  total size is 2122846 speedup is 1.00
  Finished Snapshot Collection.
  [root@pilot1 snapshot]#
Collect the ILOM snapshot for a Pilot:

Note: The results are located under /var/tmp/snapshot/ in the Pilot from where you executed the script.


Note: The recommendation is to use the argument "-d full" instead of "-d normal". The full option collects additional information like the hwdiag or the fruid. Within the fruid, we find the showpsnc.out which might be important if a Controller does not restart after some type of part replacement.


Note: It has been seen during the tests that the first run might complete in 2-3 sec although it does not generate any snapshot. If this happens, you simply need to execute the command once again.
- To collect a Pilot 2 ILOM snapshot from itself:
  
  [root@pilot2 self]# ilomsnapshot.pl -t self -d full
  [root@pilot2 self]# ilomsnapshot.pl -t self -d normal
  
  Example:
  [root@pilot2 self]# ilomsnapshot.pl -t self -d full
  Snapshot log is at /var/tmp/snapshot/self
  Getting snapshot...
  Connected. Use ^D to exit.
  -> set /SP/diag/snapshot dataset=full
  set: Busy
  -> set /SP/diag/snapshot dump_uri=ftp://root:a1s2d3f$@169.254.2.6/snapshot/self
  Collecting a "full" dataset may reset the host. Are you sure (y/n)? y
  set: Busy
  
  -> Session closed
  Command Failed
  ILOM Snapshot is submitting...0
  ILOM Snapshot is submitting...20
  ILOM Snapshot is submitting...40
  ILOM Snapshot is submitting...60
  ILOM Snapshot is submitting...80
  ILOM Snapshot is submitting...100
  ILOM Snapshot is submitting...120
  ILOM Snapshot is submitting...140
  ILOM Snapshot is submitting...160
  ILOM Snapshot is submitting...180
  ILOM Snapshot is submitting...200
  ILOM Snapshot is submitting...220
  ILOM Snapshot is submitting...240
  ILOM Snapshot is submitting...260
  ILOM Snapshot submit finished
  [root@pilot2 self]#
- To collect a Pilot 1 ILOM snapshot from Pilot 2:
  
  [root@pilot2 self]# ilomsnapshot.pl -t buddy -d normal
  
  Example:
  
  [root@pilot2 self]# ilomsnapshot.pl -t buddy -d normal
  Snapshot log is at /var/tmp/snapshot/buddy
  Getting snapshot...
  Connected. Use ^D to exit.
  -> set /SP/diag/snapshot dataset=normal
  set: Busy
  -> set /SP/diag/snapshot dump_uri=ftp://root:a1s2d3f$@169.254.2.10/snapshot/buddy
  set: Busy
  
  -> Session closed
  Command Failed
  ILOM Snapshot is submitting...0
  ILOM Snapshot is submitting...20
  ILOM Snapshot is submitting...40
  ILOM Snapshot is submitting...60
  ILOM Snapshot is submitting...80
  ILOM Snapshot is submitting...100
  ILOM Snapshot is submitting...120
  ILOM Snapshot is submitting...140
  ILOM Snapshot is submitting...160
  ILOM Snapshot is submitting...180
  ILOM Snapshot is submitting...200
  ILOM Snapshot is submitting...220
  ILOM Snapshot is submitting...240
  ILOM Snapshot is submitting...260
  ILOM Snapshot is submitting...280
  ILOM Snapshot is submitting...300
  ILOM Snapshot submit finished
  [root@pilot2 self]#
Logout from the Pilot.
Transfer the snapshot file out of the Pilot:

# scp root@<FS1-2 IP address>:<file> .

Example:

# scp root@<FS1-2 IP>:/var/images/tds/slammer/508002000158C370/snapshot/snapshot_self/fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T07-23-16.zip ~/Desktop/
root@10.186.86.30's password:
fs1-2-sin06-a-controller1-sp_1345NM201A_2015-10-29T07-23-16.zip
#

Attachments

This solution has no attachment