How to replace a Virtualized Multi-Fabric 10GbE M2 NEM (Network Express Module)

Asset ID:	1-71-1408733.1
Update Date:	2018-04-04
Keywords:

Solution Type Technical Instruction Sure

Solution 1408733.1 : How to replace a Virtualized Multi-Fabric 10GbE M2 NEM (Network Express Module)

Applies to:

Sun Netra 6000 System - Version All Versions to All Versions [Release All Releases]
Sun Blade 6000 System - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

How to replace a Virtualized Multi-Fabric 10GbE M2 NEM (Network Express Module)

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:(IS A SITE ENGINEER AVAILABLE?):
Blade 6000 Chassis, NEM, REM

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 2

FIELD ENGINEER INSTRUCTIONS

PROBLEM OVERVIEW: How to replace an Virtualized Multi-Fabric 10GbE M2 NEM Card

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? : Schedule maintenance window with customer's agreement. SAS/Ethernet interfaces ready for NEM removal.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

-----------------------------------------------

This procedure is ONLY for Virtualized Multi-Fabric 10GbE Network Express Module M2 NEMs. The procedure for Virtualized Multi-Fabric 10GbE NEMs is available in the document:
How to replace a Virtualized Multi-Fabric 10GbE NEM (Network Express Module) (Doc ID 1401598.1)

1. Identify the NEM model

Hydra = Sun Blade 6000 Virtualized Multi-Fabric 10GbE Network Express Module
Hydra2 = Sun Blade 6000 Virtualized Multi-Fabric 10GbE M2 Network Express Module

Verify the NEM model using the CMM ILOM Web Interface

- In a web browser, enter the IP address of the CMM
- Select CMM from the CMM ILOM left side menu
- Select Components -> /CH/NEM0
- Select Components -> /CH/NEM1

Verify the NEM model using the CMM ILOM CLI

Connect to the ILOM using using a terminal emulator from a terminal or PC running directly to the CMM serial port on your chassis

or connect to the chassis Ethernet network management port using a secure shell (SSH)

Execute one the following commands depending on which NEM requires maintenance:

-> show /CH/NEM0 -d properties
-> show /CH/NEM1 -d properties

Example:

-> show /CH/NEM0 -d properties
/CH/NEM0
Properties:
        type = Network Express Module
        ipmi_name = NEM0
        system_identifier = dt214-124
        fru_name =SUN BLADE 6000 VIRTUALIZED MULTI-FABRIC 10GE NEM M2
        fru_version = FW 3.0.10.16, SAS 5.3.6.0
        fru_part_number = 540-7961-02
        fru_extra_1 = FW 3.0.10.16, SAS 5.3.6.0
        fault_state = Faulted
        load_uri = (none)
        clear_fault_action = (none)
        prepare_to_remove_status = NotReady
        prepare_to_remove_action = (none)
        return_to_service_action = (none)
Commands:
cd
show

Refer to the Sun Blade 6000 Virtualized Multi-Fabric 10GbE M2 Network Express Module User's Guide for using Integrated Lights Out Manager (ILOM) with the Virtualized NEM:
http://docs.oracle.com/cd/E19530-01/

2. Caution

If a NEM fails, you need to replace it. The server modules must be informed if you are replacing a non-redundant NEM.
If the chassis is powered on and you are not replacing the NEM within 60 seconds, install an NEM filler panel to ensure proper system operation.
If you are replacing a functional SAS-2 NEM (for example replacing a Multi-Fabric NEM with a 10GbE Virtualized M2 NEM) and another SAS-2 NEM is also installed in the chassis, make sure that multipathing at the OS level is enabled on all active server blades in the chassis before you remove the existing NEM. This ensures that the primary path to disks is not lost, which can cause an OS panic. For servers running Windows 2003, which does not support multipathing, shut down IO to all disks on the disk module that are not in hardware RAID volumes.

Hot-plug removal/insertion procedures have to be followed to avoid losing the RAID configuration on the REM

3. Replacement Procedure

When removing a Virtualized M2 NEM from a powered on chassis, you must prepare PCIe and SAS interfaces for the NEM hot plug actions, and make sure that you have backed up the SAS assignment configuration. If you are hot-swapping NEM, the Gigabit Ethernet connectivity provided by that NEM will be lost. Take appropriate action pulling out the corresponding network cable from each blade.

1. Prepare the NEM for removal

1.1 Check to see if there is an Ethernet interface associated with the NEM on the server module. Perform the action associated with the appropriate operating system:

Solaris:
# ifconfig -a | grep hxge

Linux
# ifconfig -a | grep eth

Windows
Click the "Safely Remove Hardware" icon on the lower right task bar.
Sun Blade 6000 10Gbe Networking Controller

1.2 Disable all interfaces between the blade and the NEM. Perform the action associated with the appropriate operating system:

Solaris
# ifconfig hxgeX down
# ifconfig hxgeX unplumb

Linux
# ifconfig ethX down

Windows
- Double-click the Safely Remove Hardware icon at the lower right of the Windows taskbar.
- The Safely Remove Hardware dialog appears.
- Select the Sun Blade 6000 10 Gbe Networking Controller and click Stop.
- Click Close to finish.
- The NEM(s) can be safely removed.

1.3 Make sure that the corresponding Ethernet interface is disabled using same commands than Step 1.1

2. Prepare SAS Interfaces for Hot Plug Removal

The following actions must be performed in every module in chassis connected to NEM. For hosts in the chassis running Oracle Solaris 10 OS 09/10 or later, the hotplugd(1M) daemon must be started before attempting to initiate a hot-plug action.

2.1 Back up the SAS zoning configurations:

- Log into the CMM ILOM web interface.
- With CMM selected in the left pane, click the Maintenance tab for the CMM.
- Click the Backup/Restore tab.
- Select Backup from the Operating drop-down list.
- Fill out the information on the page to create your backup file.

Log into the CMM ILOM CLI.
Change to the /SP/config directory.
-> cd /SP/config
-> set passphrase=passphrase
To initiate the Backup operation, enter the command:
-> set dump_uri=transfer_method://username:password@ipaddress/directorypath/filename.config

2.2 Ensure that all I/O activity to the component has been stopped before proceeding with the hot plug action. If I/O activity is not stopped, interrupted service or data loss can occur.

2.3 Ensure that the appropriate multi-path drivers are installed if your servers are not using a hardware RAID solution.

3. In the rear of the chassis, locate the NEM that you want to remove.

If you are replacing a NEM in a powered-off chassis, go to Step 7.
If you are replacing a NEM in a powered-on chassis, follow the rest of the procedure.

4. Initiate Virtualized M2 NEM Hot-Plug removal

4.1 Press the Virtualized M2 NEM attention button (ATTN) to initiate a hot-plug insertion or hot-plug removal. The green Power/OK LED slow blinks while the action is taking place.

Note:
- If the button is pressed a second time within 5 seconds after the green Power/OK LED begins blinking, the hot-remove or insert request is aborted.
- User can push the ATTN button to bring the NEM online. The process to bring the NEM online may take up to 3 minutes to complete and not complete until the Power/OK LED is solid green.

4.2 An alternative to pushing the ATTN button is to use the Prepare to Remove using the ILOM:

- Log into the CMM ILOM web interface
- Click on CMM, then click on the System Information -> Components tab
- Click on the radio button next to the NEM that you want to remove or return to service
- Choose the option "Prepare to Remove" from the Actions menu to prepare the NEM for hot removal

- Log into the CMM ILOM CLI.
- Change directories to the NEM to be removed or the NEM that was just inserted:
-> cd /CH/NEMx
where NEMx is the NEM to be removed or inserted
Execute the command:
-> prepare_to_remove_action = enable

4.3 If the green Power/OK returns to solid green, this means that the NEM did not go off-line, probably because some blades are ready for removal while others are not

5. View the behavior of the Power/OK LED and OK-to-Remove LED:

The Ready to Remove action is successful if the following occurs within a 65 minute time-frame:

The NEM green Power/OK begins to slow

The OK-to-Remove blue LED illuminates and the Power OK button turns off.

In the ILOM System Information -> Components screen: The Return to Service option is available in the Actions drop-down list for the NEM and the Ready to Remove status column for the NEM is Ready.

In the ILOM CLI show /CH/NEMx command output, the following status is listed in the Properties section: prepare_to_remove_status = Ready

If /CH/NEMx/OK reads Off in either the ILOM web interface or CLI.

The Ready to Remove action is not successful if the following occurs:

The NEM green Power/OK LED begins to slow blink,then returns to solid green, and the OK-to-Remove LED does not go on.

In the ILOM System Information -> Components screen, the Return to Service option does not become available in the Actions drop-down list and the Ready to Remove status column for the NEM still reads Not Ready.

In the ILOM CLI show /CH/NEMx command output, the following status is listed in the Properties section: prepare_to_remove_status = NotReady

If /CH/NEMx/OK reads On in either the ILOM web interface or CLI.

6. Do one of the following:

If the Ready to Remove action is successful, proceed with Step 7.
If the Ready to Remove action is not successful, do not remove the NEM. Follow the below instructions to Force NEM Graceful Removal to remove the NEM:

Once a hot plug removal has been initiated, it must be completed as indicated by the illumination of the blue OK-to-Remove LED. If the green Power/OK returns to solid green, this means that the NEM did not go off-line.

Use the instructions in from Step 1 (Prepare the NEM for removal) to Prepare the PCIe Interface for Hot Removal to ensure that all blades have unplumbed their hxge interfaces.

7. Label the cables so that you can install them in the same location, then remove all cables from the NEM.

8. Press together and hold the buttons on both the right and left ejector levers.

9. To unlatch the NEM from the chassis, open the ejector levers by extending them outward.

10.Holding the open ejector levers, pull the NEM toward you until you can pull the rest of the module out by hand. Wait at least 60 seconds before installing the new NEM.

11.Insert the new NEM into the chassis.

12. Do one of the following:

If you have replaced a NEM in a powered off chassis, go to Step 17.
If you have replaced a NEM in a powered on chassis, follow the rest of the procedure.

13. After the NEM has been inserted into a chassis slot, wait until the Power/OK LED goes to standby blink. Standby blink indicates that the NEM is in standby mode.

14. Press the ATTN button once or use the ILOM Return to Service command. It can take up to 3 minutes for the NEM to come online.

Note - If you push the ATTN button twice within 5 seconds, the green Power/OK LED remains in standby blink mode, and the request for bringing the NEM online is cancelled.

- Log into the CMM ILOM web interface.
- Click on CMM, then click on the System Information -> Components tab.
- Click on the radio button next to the NEM that you want to remove or return to service. For example. /CH/NEM0.
- Choose the option "Return to Service" from the Actions menu to Return the NEM to normal operation after installing it.

- Use the ILOM Command Line Interface
- Log into the CMM ILOM CLI.
- Change directories to the NEM to be removed or the NEM that was just inserted: -> cd /CH/NEMx
where NEMx is the NEM to be removed or inserted.
- Execute the command:
-> return_to_service_action = enable

15. View the behavior of the Power/OK LED or the ILOM interface.

The NEM is online and the PCIe interfaces are available to the blades when the following occurs:

The NEM green Power/OK LED illuminates solid.
In the ILOM web interface System Information -> Components screen, the Prepare to Remove option is available in the Actions drop-down list for the NEM.
In the ILOM CLI show /CH/NEMx output, the following status shows in the Properties section: prepare_to_remove_status = NotReady.

16. See Restoring SAS Configurations After NEM Installation to see if any additional actions need to be taken for the SAS interfaces.  See User Manual for details https://docs.oracle.com/cd/E19530-01/html/821-0875/gjwky.html#scrolltoc.

17. Reconnect all the cables in their original locations.

4. Reference information

Sun Blade 6000 Modular System Documentation: http://docs.oracle.com/cd/E19938-01/index.html

Sun Blade 6000 Virtualized Multi-Fabric 10GbE Network Express Module: http://docs.oracle.com/cd/E19677-01/

Sun Blade 6000 Virtualized Multi-Fabric 10GbE M2 Network Express Module: http://docs.oracle.com/cd/E19530-01/

Integrated Lights Out Manager (ILOM) 3.0 Documentation: https://docs.oracle.com/cd/E19860-01/

References

<NOTE:1401598.1> - How to Replace a Virtualized Multi-Fabric 10GbE NEM (Network Express Module)
<NOTE:1397311.1> - Sun Storage 6Gb SAS PCIe RAID HBA Internal Card (Niwot) Home Page

Attachments

This solution has no attachment