Oracle FS1-2 Flash Storage System
 

PLA-Support>Sun Systems>DISK>Flash Storage>SN-EStor: FSx
 

In this Document

Goal

Solution

Prerequisites

Inhibit Password Expiration

Expired or Forgotten Password

Password Reset Utilities

NOTES for Releases Below 6.1.10

RESOLUTION On Releases Below 6.1.10

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: this article is in the process of being rewritten for external consumption

Applies to:

Goal

The goal of this document is to describe the tools and procedures to reset the primary "administrator", primary support "pillar" or replication account on an FS1 system. The document also describes a method to inhibit password expiration. These three utilities may be used if one or more of the account passwords are lost and, in the case of the administrator and pillar accounts, no valid email address is configured for recovery. The utilities may also be used at system installation if the system was not installed within 30 days of being shipped causing the default password duration time (30 days) to expire.

Solution

Prerequisites

• The FS1 system must be on release 06.01.00 Build 028701 or higher in order to reset the primary "administrator" password with this method.
• The FS1 system must be on release 06.01.05 Build 335 or higher in order to reset the primary "pillar" support password with this method.
• The FS1 system must be on release 06.01.05 Build 335 or higher in order to inhibit password expiration.
• The FS1 system must be on release 06.02.02 Build 269 or higher in order to reset the replication password with this method.
• The Pilot must be able to write to the System Configuration Database--the Persistence VLUN. 

See Internal NOTES section below for issues and resolution of releases on or below 06.01.10-0354.00.

Inhibit Password Expiration

Password expiration should only be disabled in environments where there is adequate physical protection to restrict access to the management network and customer security policies allow disabling expiration.

To inhibit password expiration, set the expiration time to "0" with FSCLI or the GUI.

Expired or Forgotten Password

• If the administrator/pillar password has already expired and if the expired password is known, click on the 'Reset password' link on the Login screen to reset the password. 
• If the administrator/pillar password has already expired and if the expired password is unknown, click on the 'Forgot password' link on the Login screen to reset the password.
• If the pillar account has already expired and no email address was assigned to it, you can log in as administrator and create a new support account as a temporary workaround.

For step-by-step instructions, please review the Reset the Primary System Administrator Password documentation.

• If the administrator password has already expired and if the expired password is unknown and the administrator account has no email address configured, password recovery will not be possible using the above methods.

Password Reset Utilities

Access to the Pilot OS shell on the active Pilot is required. To access the shell, please review Documents 2046703.1 FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System and 2029847.1 FS System: How to Enable SSH Access to the Pilot.

All password reset utilities provide instructions and information on the screen.  They require acknowledgement of the disclaimer before they will run and ignore password history. Running the appropriate utility will reset the password to the factory default with a 24-hour password expiration. After the utility runs, reboot the active Pilot to make the reset effective and store the new account information credentials in the System Configuration Database known as Persistence. After the reboot completes the password and/or password expiration can then be set to the desired values. 

The three password reset utilities are located in /usr/share/pillar/bin and can be used instead to reset the password directly from inside the Pilot shell:

• ResetPrimaryAdminPassword will reset the primary administrator password to "pillar"
• ResetSupportPassword will reset the primary support password to "pillar"
• ResetReplicationServerPassword will reset the replication password to "TwinPeaks"

1. Access the Pilot shell:
   • If administrator password is known, use Document to 2029847.1 FS System: How to Enable SSH Access to the Pilot.
   • If administrator password is not known, use Document 2070735.1 FS System: How to Access FS1-2 ILOMs Using Serial Connection to access the Pilot shell.
      
2. Note the command prompt. It will either be Pilot1 or Pilot2
3. Run cat /etc/nodenames to determine which Pilot is the Active Pilot.
   
   [root@pilot1 ~]# cat /etc/nodenames
   172.30.80.3 WN2009fffffffffffa WN2008000101000000 mgmtnode
   172.30.80.2 WN2008fffffffffff2
   172.30.80.129 WN508002000158ba51
   172.30.80.128 WN508002000158ba50 WN2008000101000001
    
   The line that ends with "mgmtnode" is the active Pilot.  172.30.80.2 = Pilot1,  172.30.80.3 = Pilot2.  In the above example, it is Pilot2.
    
4. If the command prompt does NOT match the active Pilot, ssh to that Pilot:
   
   [root@pilot1 ~]# ssh pilot2
    
5. Run the appropriate Password Reset Utility from the Active Pilot.  This example resets the administrator password:
   
   [root@pilot2 ~]# /usr/share/pillar/bin/ResetPrimaryAdminPassword
   
                   W A R N I N G !
   
   You are running a script that will attempt to reset the password of the Primary Administrator on this OracleFS system. By confirming this action in the shell you have agreed that THIS ACTION MAY VOID ANY SUPPORT AGREEMENT. If you do not agree to this -- or do not otherwise understand what you are doing -- you should type 'no' at the prompt. Support personnel may use the fact of your running this script to substantiate invalidating your support contract. This script is NOT a supported mechanism for managing this system, and RUNNING IT MAY DO IRREPARABLE HARM.
   
   NOTHING SHOULD BE ATTEMPTED HERE BY UNTRAINED SUPPORT PERSONNEL UNDER ANY CIRCUMSTANCES. This appliance is a non-traditional operating system environment, and expertise in a traditional operating system environment in NO WAY constitutes training for supporting this appliance. THOSE WITH EXPERTISE IN OTHER SYSTEMS -- HOWEVER SUPERFICIALLY SIMILAR -- ARE MORE LIKELY TO MISTAKENLY EXECUTE OPERATIONS HERE THAT WILL DO IRREPARABLE HARM. Unless you have been explicitly trained on supporting this appliance via the operating system shell, you should immediately exit this script.
   
   IMPORTANT: This script will set the Primary Administrator's password to 'pillar' and its expiration time to 24 hours. You MUST fail over the pilot for this change to take effect. After the pilot has failed over, you should immediately log in to the system and change the Primary Administrator's password.
   
   Type 'yes' to proceed with this script, or 'no' (or anything else) to exit:
   yes
   
   Success! You should now fail the pilot over for the change to take effect.
   
   [root@pilot2 ~]#
    
6. Reboot the active Pilot to force a failover:
   
   [root@pilot2 ~]#  service pilotcfg restart
   Shutting down pcp_monitor:
   Shutting down pilotcfg:
   
    
   The fscli interface can also be used (must be logged in as pillar):
   
   # fscli pilot -forceFailover
    

NOTES for Releases Below 6.1.10

Ref: Bugs 20562026, 20695269, 20786414, 20957451, and 20737952.    On releases below 6.1.10, using this script can result in Pilot CUs hanging in Boot State Pilot, or the primary administrator account disappearing from the GUI.  This happens because duplicate and/or conflicting records for the "pillar" and/or "administrator" account xml files are created in /var/PillarPilotPersistence/com.pillardata.pmi.message.AccountInfo or in the Persistence VLUN.  As the Pilot boots on a system or Pilot restart, it reconciles the Pilot Persistence accounts with those from the Persistence VLUN, and the conflict on the "Account" portion of the reconcile prevents the Pilot from completing its reconciliation necessary to exit Boot State Pilot. The following snippet from the server.log shows the offending item:  The true,false status in the checkDomainLoadMatrix for the item indicates it is an item that will prevent exit of Boot State Pilot.

If the Password has just expired, and the old password is known, select the reset password on the login screen and use the expired password to log in and change the password. 

If the expired password is not known, then the utility will need to be used.  After using the utility make sure the administrator and pillar accounts are visible in the GUI and the System > Information Status field is not Boot State Pilot.

It may be possible to identify the issue with  cd /var/PillarPilotPersistence/com.pillardata.pmi.message.AccountInfo  and grep administrator *.xml [or grep pillar *.xml].  In some instances, the "administrator" or the "pillar" account will not appear, or there will be two files with <userName>administrator</userName> or <userName>pillar</userName> or two accounts with <role>PRIMARY_ADMINISTRATOR</role> or <role>PILLAR_SUPPORT</role>. 

The server.log will show PacMan attempting to read the accounts from the Persistence VLUN (server.log entries of command="NP_MSG_DB_READ(0x170007)")and updating them in Pilot Persistence (server.log entries of command="PM_MSG_ADD_COMMAND(0x1f0005)").  The NP_MSG_DB_READ response from NP will show what is being read from the Persistence VLUN.  As PacMan attempts to sync that to Pilot Persistence, the "PM_MSG_ADD_COMMAND(0x1f0005)" will get a result code of result="PMI_EALREADY(0x10013)" indicating that PacMan has already reconciled one account in Pilot Persistence. The names and contact information of the account(s) will be in the extended details of the "PM_MSG_ADD_COMMAND(0x1f0005)" entries.

RESOLUTION On Releases Below 6.1.10

NOTE:  Persistence must be online and writeable as you reboot the active Pilot after running the script.  That reboot is what writes the new account information to Persistence and this write is necessary for the utility to work, so be sure to reboot the active pilot.  The hanging in Boot State Pilot is caused by defects in the utilities that leave behind records in the Pilot Persistence Cache /var/PillarPilotPersistence/com.pillardata.pmi.message.AccountInfo on the reboot.  As the Pilot opens Persistence to pull the records from the Accounts table in the Persistence VLUN, the conflicting records prevent Pilot Persistence and Persistence VLUN reconcile, causing the Boot State Pilot hang.   The Persistence VLUN information is always considered the most authoritive information, and must be able to override Pilot Persistence, which it can't.  That is why removing the conflicting files from [both] Pilots will recover the issue. 

Grep the xml files in /var/PillarPilotPersistence/com.pillardata.pmi.message.AccountInfo/  for  PRIMARY_ADMINISTRATOR and PILLAR_SUPPORT.  There should be ONE file on each pilot for each of those two account roles.  If either is missing or there are two files for either role,  delete all xml files that contain those values in the role entries. Check both Pilots. Then reboot the Pilot with "reboot".  As the Pilot fails over, the new active Pilot will read the Persistence VLUN Accounts table and recreate the accounts in Pilot Persistence.   

If duplicate or missing files were found on both Pilots due to having tried a Pilot Failover to cure the issue, as soon as one Pilot recovers, issue reboot to force the other Pilot to read the tables and update the account records.

If engineering assistance is required, provide logs that include a pilot boot attempt.  Also create a tar file containing all of the xml files in /var/PillarPilotPersistence/com.pillardata.pmi.message.AccountInfo/ and a Persistence Accounts Table dump with /usr/share/pillar/bin/NPperf table=32771 which creates the plain text file table8003.dat

This solution has no attachment