Sun ZFS Storage 7320
 
Sun Storage 7720 Unified Storage System
 
Sun ZFS Storage 7420
 
Sun ZFS Storage 7120
 

PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
 

In this Document

Goal

Solution

Target Audience

You will need

Overview

Detailed Steps

BIOS SETTINGS FOR 7120, 7320, 7420 and 7720

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: Partner doc to be referenced from ATR CAP

Applies to:

Goal

This document provides the steps that Partner engineers need to take for replacing the motherboard for Sun ZFS Storage Appliance 7x20 (7120, 7320, 7420, 7720).

• Scope

This procedure is applicable to the following products: 7120, 7320, 7420, 7720 (includes cluster models)

• Prerequisites

Partner engineers who are trained on the x4270 M2, x4170, x4470 platforms.

Solution

This document outlines the steps to replace a motherboard in a Sun ZFS Storage Appliance 7x20 (7120, 7320, 7420, 7720).

After completing the following procedure, the system will have a replacement motherboard, will be running with the approved 7x20 series appliance BIOS and have the correct BIOS settings.  

Target Audience

Sun or Partner engineers who are trained on the x4270 M2, x4170, and x4470 platforms.

You will need

• This document
• Alcohol Cleaner
• Wipes
• Heat Sink Compound/Thermal Paste
• SP Firmware/BIOS (see table below for obtaining S/W)
• Laptop for serial connection and browser access to SP
• Torx T10 screwdriver for 7120 systems

This procedure will require you to access the SP via a serial connection and across the network using a browser. Both of these steps could be done using customer equipment if they have the necessary kit setup, however don't assume this and bring along a laptop with appropriate cables.

Overview

If you are familiar with motherboard swaps in x4270 M2, x4170, and x4470 platforms, the physical swapping of the motherboard is identical.

Full details on the official process can be found in the appropriate service manual: (to access the SSH 7x20 listings, login to MOS)

• 7120 Sun Fire X4270 M2 SSH 7120 Motherboard replacement procedure
• 7320 Sun Fire X4170 M2 SSH 7320 Motherboard replacement procedure
• 7420 Sun Fire X4470 SSH 7420 Motherboard replacement procedure
• 7420 Sun Fire X4470 M2 SSH 7420 Motherboard replacement procedure
• 7720 Sun Fire X4470 SSH 7720 Motherboard replacement procedure
• The difference in repairs is the BIOS settings. Use [Article ID 1357409.1 ] (Login to MOS to access doc)

What is special here is a specific SP firmware and BIOS, coupled with a particular BIOS configuration.

• Generic Physical Swap
  • Note SP Network Configuration
  • Swap motherboards
  • Configure SP's network
• Appliance Specific Steps
  • Load SP firmware/BIOS
  • Configure BIOS
  • Resync SP password
  • Cluster BIOS Matching
  • Load SP Firmware/BIOS

If you are swapping a motherboard in a 7320/7420/7720 cluster, then there is the possibility that the newly replaced motherboard will now have a newer revision BIOS than the partner head. To address this, once the motherboard replacement is complete, repeat the steps again for the partner head starting at section Load SP Firmware/BIOS. This includes instructions for identifying the BIOS version.

Detailed Steps

1. Note SP Network Configuration

If the SP is available before starting work, it is a good idea to make a note of the network configuration of the SP. This will make life easier when configuring the replacement SP; it is physically located on the motherboard therefore requires the network configuration restoring.

If the SP is unavailable, you will need to speak to the customer about getting the network configuration for the SP. You will need an IP address, subnet mask and default router/gateway.

2. Swap Motherboard

The 7120 and 7320 share the same motherboard, the 7420 and 7720 are the same physical controller.
The directions for changing the motherboard can be found on the inside of the system lid. You will also need cleaning alcohol, wipes and heat sink compound. Experience says that the heat sink compound (in a syringe) is not always shipped with motherboards from logistics. Use the alcohol and wipes to remove the old heat sink compound from the top of the CPU(s) and heat sink(s) before inserting into new motherboard.

3. Move onboard USB memory stick

7x20 systems come with a USB stick inserted into an internal onboard USB interface. This is typically a small blue USB memory stick. At present this device is not used (it is there for a future product enhancement), but nevertheless it should be moved over when changing the motherboards. Don't forget to do this!

4. Torx T10 Screwdriver Required for RHDDs on 7120

The 7120 system drives are held in a small caddy in the rear of the system. This caddy sits in place of PCI slots 2 and 5. It is held in place with 2 large screws (green crossheads screwed into motherboard), and a small Torx T10 screwed into PCI backpanel stiffener. Make sure you have the Torx T10 as it would be shame to be defeated when replacing a motherboard in a 7120 by such a small screw!

5. SPECIAL CARE WITH 7420/7720 INTERNAL SATA CABLES

The SATA cables (for the disks at the front of the x4470) run from an onboard connector towards the rear, and then under the power supplies, before popping out and connecting to the disk backplane. In addition to the official x4470 service procedures, it is worth pointing out that extra care should be taken with:

• Undoing Power Supply Cover Plate: Tiny screw and easily damaged
• Removing/Inserting the large motherboard: SATA cables can easily be damaged

6. Note Cable Configuration

The official procedure for swapping motherboards in these systems is generic. The 7x20 series systems have specific configurations for how the HBA or SAS cables are attached to the motherboard. It is strongly recommended that careful note is taken of how the original system was cabled.

Particular attention should be paid to the 7720 system; each head has 8 SAS cables which are connected to 2 RCC units in the cabinet, plus the onboard igb2 and igb3 interfaces should be patched into the RCC network ports. These, plus other networking and serial connections means each head will have a lot of cables, so take care to label them as they are disconnected.

7. Confirm Chassis Serial Number Updating

7x20 systems use SP Firmware 3.x which uses containers to store the Chassis Serial Number (known as a TLI, or Top Level Identifier). Containers are merely the FRU boards in the system and depending on the platform there are different containers:

• 7120/7320 (platform x4170/x4275)
  • Primary: Power Distribution Board (PDB)
  • Backup1: Motherboard (MB)
  • Backup2: Disk Backplane (DBP)
• 7420/7720 (platform x4470)
  • Primary: Fan Board (FB)
  • Backup1: Disk Backplane (DBP)
  • Backup2: Motherboard (MB)

7.1 Replacing a Backup Container

NOTE: This is the common case when replacing a motherboard.

When replacing a backup container, the SP software will automatically take care of updating the Chassis Serial Number from the other containers.

Before powering on, use a serial connection into the SP (labelled SER MGT). When the power supply is inserted the SP will begin booting:

There will be quite a lot of output as the SP boots. You are looking to see the message:

This indicates the SP has automatically updated the replaced motherboard with the previous Chassis Serial Number.

7.2 Replacing Primary Container

NOTE: This is only required if changing a PDB on a 7120/7320 or a Fan Board on a 7420/7720.

However, if the primary container is replaced, then an additional manual service procedure is required. Rather than repeat the detailed steps here, you should read and understand the documents:

How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1)
How to update product serial number on systems which implement Top Level Identifier functionality (Doc ID 1280913.1)

The first document/procedure will also require access to the ILOM Password Generator. Passwords are valid for 48 hours and can be generated for a specific time in the future. Please ensure you read these two documents and have an appropriate ILOM password ahead of time if changing a primary container.

When the SP boots, you will see something like:

 Error: Product Identification data is missing. Product IDs will be set to unknown. Service must update all containers.

The following example shows the case of changing the primary container (Fan Board) in a 7720, a similar procedure is appropriate for a 7120/7320 and a PDB change:

-> show /SYS product_serial_number

  /SYS
    Properties:
    product_serial_number = 0000000000

-> set SESSION mode=service
Short Form Password:**** **** ****
Currently in service mode.

-> showpsnc
Primary: fruid:/~//SYS/FB
Backup 1: fruid:/~//SYS/DBP
Backup 2: fruid:/~//SYS/MB

Element           | Primary           | Backup 1          | Backup 2
------------------+-------------------+-------------------+-------------------
Container Status   Not Valid           Not Valid           Not Valid
PPN                597-0865-01         597-0865-01         597-0865-01
PSN                1033DF008D          1033DF0086          1033DF0086
Product Name       SUN FIRE X4470 SERVER SUN FIRE X4470 SERVER SUN FIRE X4470 SERVER
-> copypsnc  BACKUP1 PRIMARY
-> showpsnc
Primary: fruid:/~//SYS/FB
Backup 1: fruid:/~//SYS/DBP
Backup 2: fruid:/~//SYS/MB

Element           | Primary           | Backup 1          | Backup 2
------------------+-------------------+-------------------+-------------------
Container Status   Valid               Not Valid           Not Valid
PPN                597-0865-01         597-0865-01         597-0865-01
PSN                1033DF0086          1033DF0086          1033DF0086
Product Name       SUN FIRE X4470 SERVER SUN FIRE X4470 SERVER SUN FIRE X4470 SERVER
-> show /SYS product_serial_number 
 
  /SYS 
    Properties: 
    product_serial_number = 0000000000

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

-> show /SYS product_serial_number

  /SYS
    Properties:
    product_serial_number = 1033DF0086 

8. Configure SP's Network and Hostname

Now configure the SP networking (substitute appropriate addresses and subnet mask) and SP hostname (the SP hostname is optional and not a requirement):

NOTE: THESE ADDRESSES ARE EXAMPLES. USE THE VALUES NOTED DOWN EARLIER

9. Load SP Firmware/BIOS

The SP firmware/BIOS may not be correct for the 7x20 series. Check the SP firmware version when logging in:

NOTE: The newly replaced board will boot with an SP hostname based on the Chassis Serial Number that the board shipped with. Part of this procedure includes a note to reset the SP hostname, otherwise on subsequent boots of the SP it will be based on the updated (ie correct) Chassis Serial Number.

Next check the BIOS with:

The latest  "fru_version" here should be:

NOTE:
The "SW Version" relates to the internal numbering for the BIOS/SP firmware bundle.

Review the BIOS overview for current releases via the table above

The 7120 and 7320 systems share the same system board, hence use the same BIOS. The 7420 and 7720 are the same physical controller.

The SP firmware and BIOS are updated at the same time from a single image. It is handy to have a terminal window with a serial connection to the SP open at this stage. The updating is done via a browser, but the subsequent reboot can be observed via the open serial connection.

To update, point a browser at the SP, log in as root and follow these steps:

• 1. Click on main Maintenance tab
• 2. Firmware Upgrade will be the default and correct subtab
• 3. Click on "Enter Upgrade Mode"
• 4. Confirm this action with the pop up
• 5. Click on "Browse" and select the appropriate image from your local filesystem
• 6. Click on "Upload"
• 7. Wait for upload to complete and the verification to succeed
• 8. You will now see a Summary Table of the SP firmware and BIOS versions (Existing vs New). Confirm that "Preserve existing configuration" is checked for the SP Firmware.
• 9. Click on "Start Upgrade"
• 10. Confirm this action with the pop up
• 11. Now wait for the upgrade to proceed. If the head was up at this point, it will be cleanly shutdown. Warning! Do not interrupt the update. Leave the browser undisturbed until the update is complete.
• 12. When finished, you will see "Upgrade Complete" and the SP will reboot. This is where it is handy to have the SP serial window open.

10. Configure BIOS

The SP firmware and BIOS will now have been updated to the correct 7x20 version. Now you must configure some specific BIOS settings. Boot the head and enter setup with:

-> start /SYS
Are you sure you want to start /SYS (y/n)? y
Starting /SYS

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y

Serial console started.  To stop, type ESC (

Once you see the initial BIOS banner, hit CONTROL-E a few times; this will trigger the BIOS Setup menu after the initialisation.

'You can drop back to the SP with ESC-( NOTE: Escape, followed by shift 9 -- at least open parenthesis is usually on shift 9. Once into the BIOS Setup screen, start by loading factory defaults.

To do this, use the right arrow key to move over to the "Exit" menu. Down arrow to "Load Optimal Defaults" and, then again to confirm the popup asking "Load Optimal Defaults". Now follow the specific instructions for the appropriate appliance:

BIOS SETTINGS FOR 7120, 7320, 7420 and 7720

• 7120 BIOS

Disable PCI-E Option-ROM scanning for all non-NET slots in Boot Menu.

Enable Persistent Boot Mode.

Select HDD boot devices.

Use the right arrow key to page over to the "Boot" menu. Use the down arrow to select:

Option ROM Enable

Press return, and then in the sub-menu use the down arrow to select:

PCI-E Slot 0 Option ROM:       [Enabled]

Press return and select "Disabled", followed by return. This will now appear as:

PCI-E Slot 0 Option ROM:       [Disabled]

Repeat this for all PCI-E slots (1-5) in this sub-menu.

Press ESC to return to the top-level "Boot" menu and enter Expert Mode by pressing CONTROL-U. This will prompt with:

At this point select OK to enter Expert Mode.

Once Expert Mode has been enabled, enter the sub-menu:

Boot Setting Configuration

Use the arrow key to move down and select:

Persistent Boot Mode           [Disabled]

Press return and select "Enabled", followed by return. This will now appear as:

Persistent Boot Mode           [Enabled]

Press ESC to return to the top-level "Boot" menu. Use right arrow to page over to "Exit". Press return for the default "Save Changes and Exit", and again to confirm the action with the pop up.

This will cause the appliance to reboot. You will need to re-enter the BIOS Setup in order to set the boot device priority. Once again use CONTROL-E to enter BIOS Setup when the BIOS screen first loads.

NOTE: Attempting to set the boot device priority without performing this extra reboot may result in an incorrectly configured boot device priority.

Once the BIOS Setup is running use the arrow keys to navigate to the "Boot" menu and enter the sub-menu:

Boot Device Priority

You do not need to re-enable Expert Mode. Set the first two boot devices to be HDD:P0 and HDD:P1 using the "+" key to rotate the list and move these devices to the top, such that you have something similar to:

1)  [P B HDD:P0-SEAGATE ST95000NSSUN500G 101]
2)  [P B HDD:P1-SEAGATE ST95000NSSUN500G 101]
3)  [P B PXE:IBA GE Slot 0100 v1331]
4)  [P B PXE:IBA GE Slot 0101 v1331]
5)  [P B PXE:IBA GE Slot 0700 v1331]
6)  [P B PXE:IBA GE Slot 0701 v1331]
7)  [  B PORT4]
8)  [  B PORT5]
9)  [  B USB Front 0]
10) [  B USB Front 1]
11) [  B USB Rear 0]
12) [  B USB Rear 1]
13) [P B USB:02.82;01  Unigen PSA4000]

Press ESC to return to the top-level "Boot" menu.

Exiting BIOS Setup : Use right arrow to page over to "Exit". Press return for the default "Save Changes and Exit", and again to confirm the action with the pop up.

• 7320 BIOS

Disable PCI-E Option-ROM scanning for all non-NET slots in Boot Menu.

Enable Persistent Boot Mode.

Select HDD boot devices.

Use the right arrow key to page over to the "Boot" menu. Use the down arrow to select:

Option ROM Enable

Press return, and then in the sub-menu use the down arrow to select:

PCI-E Slot 0 Option ROM:       [Enabled]

Press return and select "Disabled", followed by return. This will now appear as:

PCI-E Slot 0 Option ROM:       [Disabled]

Repeat this for all PCI-E slots (1 and 2) in this sub-menu.

Press ESC to return to the top-level "Boot" menu and enter Expert Mode by pressing CONTROL-U. This will prompt with:

At this point select OK to enter Expert Mode.

Once Expert Mode has been enabled, enter the sub-menu:

Boot Setting Configuration

Use the arrow key to move down and select:

Persistent Boot Mode           [Disabled]

Press return and select "Enabled", followed by return. This will now appear as:

Persistent Boot Mode           [Enabled]

Press ESC to return to the top-level "Boot" menu. Use right arrow to page over to "Exit". Press return for the default "Save Changes and Exit", and again to confirm the action with the pop up.

This will cause the appliance to reboot. You will need to re-enter the BIOS Setup in order to set the boot device priority. Once again use CONTROL-E to enter BIOS Setup when the BIOS screen first loads.

NOTE: Attempting to set the boot device priority without performing this extra reboot may result in an incorrectly configured boot device priority.

Once the BIOS Setup is running use the arrow keys to navigate to the "Boot" menu and enter the sub-menu:

Boot Device Priority

You do not need to re-enable Expert Mode. Set the first two boot devices to be HDD:P0 and HDD:P1 using the "+" key to rotate the list and move these devices to the top.

You should have something similar to (this is most likely correct and doesn't need changing):

1)  [P B HDD:P0-SEAGATE ST95000NSSUN500G 102]       
2)  [P B HDD:P1-SEAGATE ST95000NSSUN500G 103]
3)  [P B PXE:IBA GE Slot 0100 v1331]
4)  [P B PXE:IBA GE Slot 0101 v1331]
5)  [P B PXE:IBA GE Slot 0700 v1331]
6)  [P B PXE:IBA GE Slot 0701 v1331]
7)  [P   HDD:P2-TOSHIBA THNS512GG8BBAA]
8)  [P   HDD:P3-TOSHIBA THNS512GG8BBAA]
9)  [  B PORT4]
10) [  B PORT5]
11) [  B USB Front 0]
12) [  B USB Front 1]
13) [  B USB Rear 0]
14) [  B USB Rear 1]
15) [P B USB:02.82;01  Unigen PSA4000] 

Press ESC to return to the top-level "Boot" menu.

Exiting BIOS Setup : Use right arrow to page over to "Exit". Press return for the default "Save Changes and Exit", and again to confirm the action with the pop up.

• 7420 BIOS

Disable PCI-E Option-ROM scanning for all non-NET slots in Boot Menu.

Disable I/O allocation for all but Cluster.

Enable Persistent Boot Mode.

Select HDD boot devices.

Use the right arrow key to page over to "PCIPnP" menu. Use the down arrow to highlight:

PCI-E Slot 2 Option ROM:       [Enabled] 

Press return and select "Disabled", followed by return. This will now appear as:

PCI-E Slot 2 Option ROM:       [Disabled]

Repeat this for all slots (there are more off the bottom).

Just below these OPROM settings are a group of settings which allow IO allocation to be disabled per-slot. Disable all but the final entry, which will likely need enabling. The last few lines should look like:

I/O Allocation for Slot 7      [Disabled]          
I/O Allocation for Slot 8      [Disabled]        
I/O Allocation for Slot 9      [Disabled]        
I/O Allocation for Cluster     [Enabled]

Use the right arrow key to page over to "Boot" menu and enter Expert Mode by pressing CONTROL-U. This will prompt with:

At this point select OK to enter Expert Mode.

Once Expert Mode has been enabled, enter the sub-menu:

Boot Setting Configuration

Use the arrow key to move down and select:

Persistent Boot Mode           [Disabled]

Press return and select "Enabled", followed by return. This will now appear as:

Persistent Boot Mode           [Enabled]

Press ESC to return to the top-level "Boot" menu. Use right arrow to page over to "Exit". Press return for the default "Save Changes and Exit", and again to confirm the action with the pop up.

This will cause the appliance to reboot. You will need to re-enter the BIOS Setup in order to set the boot device priority. Once again use CONTROL-E to enter BIOS Setup when the BIOS screen first loads.

NOTE: Attempting to set the boot device priority without performing this extra reboot may result in an incorrectly configured boot device priority.

Once the BIOS Setup is running use the arrow keys to navigate to the "Boot" menu and enter the sub-menu:

Boot Device Priority

You do not need to re-enable Expert Mode. Set the first two boot devices to be HDD:P0 and HDD:P1 using the "+" key to rotate the list and move these devices to the top, such that you have something similar to (this is most likely correct and doesn't need changing):

1)  [P B HDD:P0-SEAGATE ST95000NSSUN500G 102]       
2)  [P B HDD:P1-SEAGATE ST95000NSSUN500G 102]       
3)  [  B HDD2]  
4)  [  B HDD3]  
5)  [  B HDD4]                                      
6)  [  B FMOD]  
7)  [  B USB0]                                 
8)  [  B USB1]                                      
9)  [P B PXE:IBA GE Slot 6100 v1335]                
10) [P B PXE:IBA GE Slot 6101 v1335]  
11) [P B PXE:IBA GE Slot 8100 v1335]                
12) [P B PXE:IBA GE Slot 8101 v1335]                
13) [P   HDD:P2-TOSHIBA THNS512GG8BBAA]  
14) [P   HDD:P3-TOSHIBA THNS512GG8BBAA]             
15) [P   HDD:P4-TOSHIBA THNS512GG8BBAA]             
16) [P   HDD:P5-TOSHIBA THNS512GG8BBAA]             
17) [P   USB:ExtPort0:Unigen PSA4000]  

Press ESC to return to the top-level "Boot" menu.

Exiting BIOS Setup : Use the arrow right to move to the "Exit" menu. Press return for the default "Save Changes and Exit", and return again to confirm the action with the pop up.

• 7720 BIOS

Same as 7420 above.

11.Resync SP Password

Finally, resync the SP password to match the root password of the NAS head. Have the customer complete this final step. Remember: you exit back to the SP using ESC-(

With the SP password matching the root password, the motherboard replacement is complete.

NOTE:
The SP has a minimum password length of 8 which is not enforced by the appliance system software. ie, if the customer has the hopelessly simple password "abc", then this will be rejected by the SP. To resolve this, the customer will need set a new password from the appliance, which in turn will update the SP password directly.

References

This solution has no attachment