Pillar Axiom: Slammer Boot Process & Booting States

Asset ID:	1-79-1511869.1
Update Date:	2018-04-20
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1511869.1 : Pillar Axiom: Slammer Boot Process & Booting States

Applies to:

Pillar Axiom 600 Storage System - Version All Versions and later
Pillar Axiom 500 Storage System - Version All Versions and later
Pillar Axiom 300 Storage System - Version All Versions and later
Information in this document applies to any platform.

Purpose

Applies To: Support personnel who watch the Slammer Booting 0x00H states to help troubleshoot

Use this document to determine at what point a slammer failed to boot. The status of the slammer control unit will start as unknown and during boot the status will display different boot codes. This document will help identify which process it is currently loading.

The slammer control unit should end up with a status of normal at the end, however if the slammer control unit fails, note down the last boot code before it failed. This can be an indication that there is a probable fault with that piece of software the slammer was trying to load.

Scope

NOTE: In order to watch the boot states of a Slammer during an R4 to R5 upgrade, you will need to download and install the Axiom GUI client in advance and make sure port 26008 is open to the Pilot. Once the Axiom is on R5, the only HTTP interface is one that allows downloading the client.

R3 introduces ConMan, the Configuration Manager, that runs in the Slammer and coordinates the activity of several other modules. R5 introduces PACMan, Pillar Axiom Configuration Manager, to replace mcc-core. The boot sequence for MCC-Core or PACMan is identical for booting Slammers.

The R5 GUI will show the Slammer Booting states by name rather than by their hex component number.

Details

Slammer Initial Boot

The Slammer boots from the "A" or rPage copy of PROM resident on the Slammer. There is a separate image called the "B" page which is largely used to boot microdms to inform the Pilot that the A page is corrupt or otherwise unusable. A number of power on tests are run by the PROM, with some of the tests skipped if the boot flags indicate a PODR, Power On Data Recovery, rather than a Power On, since a full memory test would destroy the contents of the two DIMMS that are battery backed memory.

For the Slammer boot process, the most important component in PROM is microdms. The microdms is responsible for booting the Slammer CU to the point where the downloaded Slammer software component DMS can be launched. As the Slammer CU boots, the LEDs are controlled by microdms up to the point where the Pilot Software takes over to manage the initialization of the software modules in the Slammer

Microdms functions:

Manages the boot diagnostics and starts fans and temperature monitoring
Turns on the Slammer Ethernet port on the motherboard that connects thru the FCIM
Turns on the FCIM and sets up the Ethernet switch in the FCIM that connects the motherboard ethernet to the PMI ethernet ports on the FCIM
Sets up the initial Spanning Tree protocol for the PMI ethernet to keep ethernet frames from being propagated infinitely on the PMI. If this isn't done, the redundant connections of the PMI will result in frames being repeated forever and will prevent Slammer boot. This spanning tree is missing from at least some versions of flashslam.ifs, which is why when doing flashslam you may need to connect an ethernet straight from the active Pilot CU to the Slammer CU being updated.
Starts netboot. This sends the DHCP request to the Pilot, passing the WWN of the Slammer and requesting an IP address on the 172.30.80.xx subnet. Once an IP address has been received, netboot sends a request to the Pilot to download the slammer.ifs (ax500 & ax300) or slammer_ax600.ifs (ax600) file. The file name is sent by the netboot and is hard coded in the Slammer PROM.
Slammer CU LED State at netboot Once microdms reaches this state where it is attempting to download the Slammer runtime software a healthy motherboard:
The FLT LED should be off
ACT will be flashing green rapidly, about twice per second.
ST will be flashing green slowly, about once per 1.5 second.
This ST and ACT Blinking Green with no FLT is a very significant state in Slammer CU Boot, as it is the transition where the Slammer CU is no longer in control of its status.
Monitors netboot. If the Slammer CU is unable to obtain an IP address or download the slammer.ifs/slammer_ax600.ifs file within approximately 90 seconds, microdms will power reset the CU to try again. This timeout is in R3 Slammer PROM to attempt to recover a possibly hung ethernet network on either the motherboard or in the FCIM.
Boot progress CONSOLE information is written to a CONSOLE file on the buddy node if possible, to allow troubleshooting boot issues where the TDS component is not able to come up far enough to upload logs into the Pilot.
LED State Indicating Software Initialization If the slammer runtime image download succeeds, microdms jumps to the start location which begins initializing the slammer software and sends a Wanna_Be_Ready to the Pilot. If microdms reaches this state, it blinks the ST LED amber. All further boot progress is controlled by the CHAMP heartbeat interface on the other nodes and the Pilot. The ST LED will blink amber until either the boot fails or the boot completes and the ST LED goes Green as the Slammer transitions to Green and Normal status.

CHAMP Heartbeat

The CHAMP is known by multiple names: CHAMP, Concensus Heartbeat And Mumble Protocol; HBMON, HeartBeat MONitor; or FOFB, FailOver/FailBack. It includes STONITH, Shoot The Other Node In The Head, capability to help remove nodes (the infamous champ reset) causing problems with other nodes. As Slammer CUs boot, the first series of CHAMP/HBMON packets sent are WBR, Wanna Be Ready. MCC-Core/PACMan look at their failure history for the CU before allowing it to proceed. If the failure history has not been exceeded, the Pilot will set the status of the CU to "INIT" which allows it to transition to Node Ready, or in the case of failures, Node Not Ready.

A "node" is a Slammer CU or a Pilot CU. There are two special nodes, the Master Node, which is a Slammer CU, and the Active Pilot, where both cooperate to manage the rest of the nodes. The heartbeats and STONITH are used by the non-master Slammer nodes and the passive Pilot to control the state of the Master Node or Active Pilot. The Slammer nodes do get a vote on whether a Pilot node is allowed to transition to Active Pilot status--a separate discussion.

Once a Slammer CU has transitioned to Node Ready (signalled in the heartbeats), the individual slammer software components are managed by mcc-core or PacMan in the Pilot.

Slammer Components

The list of components is contained in the components.h file in the MCC/ConMan source. If AutoTriage is working, it should also create a compsDB.txt file when running scanlog. Not all components exist in all releases.

Commands sent by MCC/PACMan begin with the hex component number. Events generated by a component begin with the hex component number. During boot, the components are displayed with leading zeros, for example Booting 0x004 is DMS. The components are listed in numerical order, and not in boot sequence. Some of these components are subcomponents of other components and do not appear in the boot sequence in MCC/PACMan or the GUI.

Boot code chart

Boot Sequence

The order in which MCC/PACMan bring up Slammer components. Some of the compents initialize so quickly you rarely see them in the GUI unless that is where the boot hangs or you set a Halt Point.

Booting 0x004 DMS Diagnostic and Monitoring Services: One of the first things this does is build a packet of info to pass to the pilot to look for memory and other hardware mismatches, or the need to burn PROM. It also updates the FCIM firmware if needed.
Booting 0x008 PI Private Interconnect: Turns on the FCIM fabric switches [or hub in ax300] and starts looking for bricks and queues on the fly firmware updates if required.
Booting 0x00F TDS Trace and Debug Services: Sets up the memory for the components to put their trace info in and the dumper. Dumps info to Pilot files for warm starts, boots, etc. If the boot never reaches TDS, troubleshooting is done via the CONSOLE files which are sent to the buddy if at all possible, so possibly it can send them to the Pilot.
Booting 0x01C ConMan Configuration Manager: New with R3. R2 and below would go to AM or booting 0x005. Sets up for BBM recovery, reads/reconciles COD in from disk. This reconciliation uses the view of geomap allocations from COD, from the ConMan allocation map (crmap), and from the Brick allocation map (brmap) On upgrades from R3 to R4 or R4 to R5 does COD Conversion. The r3 and r4 upgrade audits use the code from the higher release in "report" mode.
Booting 0x018 DSC Distributed Services Component: New in R3, where each of the R3 point releases have slightly more services added to this component. A distributed lock manager used by the Array Manager and Block Services to coordinate bring up of those components amongst all nodes.
Booting 0x005 AM Array Manager: In R1 and R2, this component read the COD from disk and reconciled. In R3 and higher, ConMan does that reconciliation. AM sets up the low level geomaps that BS will use to open Persistence and start the configuration of VLUNs under the user level file systems and LUNs. Keeps the AM Metadata VLUN for each FS or LUN so any bad blocks can be tracked even across drive replacements.
Booting 0x006 BS Block Services: Configures and starts Persistence VLUN and all "user" oriented VLUNs including those that support SAN LUNs, File Systems,, etc, but does not start Clones or Repositories (doesn't have enough info to do this, those are started by their respective components).
Booting 0x010 PE or Booting 0x017 NP Persistence or New Persistence: NP came in as of R3. You may see both states on R3 and higher. Both are just called Persistence or System Root Configuration. This contains the user level configuration of File Systems, LUNs, Callhome, Link Aggregation, Scheduled Tasks, Initiator Tables, etc. On a clean cold start after a clean shutdown, this is where automatically assigned File Systems or LUNs would be rebalanced across Slammer Nodes.
Booting 0x019 RMR Remote Mirror: A no-op up to and including R5, not used in RepLite. For future releases.

Here, the boot sequence branches, depending on whether you are configuring SAN or NAS, or in mixed Axioms both. Some of these will not be seen in SAN only or NAS only systems. If both are present, the following sequence applies.

Booting 0x00D SAN Storage Area Network: Brings up the SAN and iSCSI interfaces, including loading new HBA/NIC firmware if applicable. Loads the Pilot IP address into SAN to be passed to clients in SCSI Inquiry responses, which is how APM clients know which pilot to log into for which LUN. Sets up Mapping/Zoning and turns on the LUNs and their Clones/Repositories. SAN only systems are pretty much ready to rock and roll at this point.
Booting 0x009 MFS Meta File System: Configure the NAS File Systems. This includes the PDSFS, an internal file system on the NAS Slammers that is used to store NAS configuration information. Run a quick file system check on the file systems and their snapshots and clones.
Booting 0x007 VS Virtual Server: Configures Link Aggregation, Routes, etc. Lock Managers, and the TCP/IP stack for the File Servers
Booting 0x011 XPAL Cross-Protocol Adapter Layer: Services common to both NFS and CIFS, character sets, user mapping/no mapping mode, combined Windows and NFS authentication
Booting 0x00A NFS Network File System: Brings up NFS server services on those File Servers configured with NFS support. Reads exports and host exports from Persistence and configures into NFS servers.
Booting 0x00C CIFS Common Internet File System: Starts the Microsoft File Sharing services on File Servers configured with NFS support. Reads shares, domain accounts, and Server Comment configuration items from Persistence and brings up the Microsoft services.
Booting 0x012 SIM SCSI Interface Module: Brings up the SCSI or FC HBA in the NAS Slammer if present, for NDMP operations to tape drives. Shortly will have the ability to update the firmware in the NAS FC HBA.
Booting 0x00E BE Backup Engine: The NAS component used to drive NDMP backup and restores.
Booting 0x01B FPI Fast Path Internal: The final component in NAS bringup, runs a goodly portion of the NAS services in the fast path for performance.

From here, the Pilot checks all components, including the status of recovery from Battery Back Memory, and then sends a global message to Component 0, which is all components in any given slammer. This message is "allow performance mode" which only provides the permission for those components to transition LUNs and File Systems into Normal mode as opposed to Conservative. Any one of those components can keep the resources Conservative if it has any reason to doubt the wisdom of transitioning to Battery Backed Memory caching for LUNs and Journaling for File Systems. Then the Pilot sends out a scan for tape devices to the SIM component in NAS Slammers. As of R4.1/2, it also starts up the RepLite managers in the Pilot to do the replication thing with the Slammers.

Attachments

This solution has no attachment