Oracle SuperCluster T5-8 Full Rack
 
Oracle SuperCluster M7 Hardware
 
Oracle SuperCluster T5-8 Half Rack
 
Oracle SuperCluster M6-32 Hardware
 
Oracle SuperCluster T5-8 Hardware
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 
Tools>Primary Use>Configuration
 

This article explains how to add Fibre Channel adapters to a Root Domain within SuperCluster.  
Adding FC HBAs to Root Domains allows them to be used in SuperCluster IO Domains, thus delivering Fibre Channel to this domain type.
Once added to the system, they can be assigned to new IO domains using the IO Domain Creation Tool.

In this Document

Goal

Solution

Purpose

Scope

Details

Prerequisites

Overview

Detailed Instructions

Identify the IO slot (T5-8) or CMIOU (M7) to host the FC adapter.

Physically install the FC card in this location.

Verify that the FC adapter is recognized by Solaris in the root domain

Create Virtual Functions (VF) for the FC adapter

Verify that VFs have been created as expected

Trigger and verify the inclusion of the newly created VFs in the IO resource pool of the SuperCluster Virtual Assistant

Saving the New Hardware Configuration

Additional Reading

References

Applies to:

Goal

This document assists in the configuration of Fibre Channel functionality for IO domains in SuperCluster.

Solution

Purpose

This document describes how to

• add new or additional fibre channel adapters (FC HBAs) to root domains within a SuperCluster configuration.
• configure virtual functions based on the capabilities of the specific adapter
• trigger the addition of these virtual functions into the pool of available virtual hardware

Scope

This document is intended for SuperCluster owners and administrators who have the privilege the create IO Domains.  It will describe how to add FC adapters and create the virtual functions.  It does not cover the physical installation of the FC adapters or the creation of IO domains or other SuperCluster administration tasks.

Details

Prerequisites

• The SuperCluster environment must be at software level "July2016 QFSDP" or later.  Specifically:
  • The control domain and root domains must run Solaris 11.3 SRU 7.5 or later
    • To check, use the command "pkg info -l entire|grep Version".  The system in the example below shows Solaris 11.3 SRU 7.6:
      
      root@sc01-rootadm0103:~# pkg info -l entire|grep Version
            Version: 0.5.11 (Oracle Solaris 11.3.7.6.0)
  • On M7, the system firmware must be 9.7.1.b or later
    You can check the firmware level while logged into the system's ILOM using the command
    -> show /System system_fw_version
    On the control domain, you can also use the command
    ipmitool sunoem cli "show /System system_fw_version"
  • On T5-8, the system firmware must be 9.5.4.a or later
    To check the firmware level, use the command "prtdiag -v|grep Firmware".
    You can also use the two methods described above for M7.
  • The IO domain subsystem (exafamily) software must be version 2.2.0.896 or later.
    To check this version, use the command "pkg info system/platform/supercluster/osc-domcreate|grep Branch"
    This command must be run on the master control domain.  It will fail on other domains as the package is only installed on the master control domain.
  • The package "pkg:/system/storage/fc-utilities" needs to be installed in each root domain that will own FC resources in order to check the FC adapter's firmware version.  This package is not installed by default and can be installed manually for this purpose.
• The firmware of the FC card must support SR-IOV.
  See DocID 1325454.1 for details.
  • Minimum firmware versions as of 2017-05-04 are:
    • QLogic:
      • Model: 7023303
      • Min. firmware: 3.60.17
    • Emulex:
      • Model: 7101684
      • Min. firmware: 1.1.60.1
  • To check the firmware level of the FC adapter (once it is installed), use the command "fcinfo hba-port" while logged into the root domain.  For example:
    root@sc01-rootadm0103:~# fcinfo hba-port|egrep "Manu|Model|Firmware"
            Manufacturer: Emulex
            Model: 7101684
            Firmware Version: 7101684 10.6.230.0
            Manufacturer: Emulex
            Model: 7101684
            Firmware Version: 7101684 10.6.230.0
    This tells us that this Emulex Card with model number 7101684 is running Firmware version 10.6.230.0.  This is higher than what is listed in the above mentioned Document, so the firmware level is ok.
  • To update the FC adapter firmware, please consult the corresponding documentation.
    • Updating the Qlogic 16GBit FC HBA Firmware
      https://docs.oracle.com/cd/E24651_01/html/E24460/z40036871045586.html#scrolltoc
    • Updating the Emulex 16GBit FC HBA firmware
      https://docs.oracle.com/cd/E24649_01/html/E24462/z40005861008729.html#scrolltoc
  • After updating the firmware, make sure that SR-IOV has been enabled on the adapter.  See below for an example of how to verify this.
  • For some Qlogic adapters, it might be necessary to also update the "Firmware Preload Table".  This can be achieved with the firmware update tool available from Qlogic.  See DocID 1950167.1 and the adapter release notes for details.
• The SAN switch must support NPIV
  See the Oracle VM Server for SPARC Admin Guide for details.

Overview

 These are the steps to add and configure new or additional Fibre Channel adapters in a root domain:

1. Identify the IO slot (T5-8) or CMIOU (M7) to host the FC adapter.
2. Physically install the FC card in this location. 
   The physical installation, including any component shutdown required is not covered in this document.
   Note that hotplug operations are not supported on SuperCluster.  You should plan a shutdown of the physical domain to add the FC adapters.
3. Restart the root domain to which the FC adapter has been added.
4. Verify that the FC adapter is recognized by Solaris in the root domain
5. Create Virtual Functions (VF) for the FC adapter
6. Verify that VFs have been created as expected
7. Trigger and verify the inclusion of the newly created VFs in the IO resource pool of the SuperCluster Virtual Assistant.

Detailed Instructions

Identify the IO slot (T5-8) or CMIOU (M7) to host the FC adapter.

Resources for identifying the right IO slot (T5-8) or CMIOU (M7):

• Use the “Physical Hosts” tab in the SuperCluster Virtual Assistant to identify the names of the physical host (T5-8) or physical domain (M7) and the names of the root domains. For example, on a SuperCluster M7, root domain “sc01-rootadm0104” might be located in PDom 1.
• Use the command “virtinfo -d” in the root domain to obtain the LDoms domain name for this domain. For example:
  root@sc01-rootadm0104:~# virtinfo -d
  Domain name: ssccn1-dom3
    
  This shows that the domain name is ssccn1-dom3.
• From the control domain of the physical host, use the “ldm ls-io” command to identify the corresponding physical location. For example:
  root@sc01dbadm0101:~# ldm ls-io|grep "PCIE "|grep ssccn1-dom3
  /SYS/CMIOU3/PCIE2 PCIE pci_15 ssccn1-dom3 OCC
  /SYS/CMIOU3/PCIE3 PCIE pci_16 ssccn1-dom3 OCC
  /SYS/CMIOU3/PCIE1 PCIE pci_18 ssccn1-dom3 EMP
    
  (Note the extra space in the argument to the first grep command above!)
  This shows that this root domain uses CMIOU3, and slot “PCIE1” on CMIOU3 is currently empty, ready to host the FC adapter.
• Use the guidance from the Service Manual of your system to shut down the root domain and prepare for physical installation of the adapter.
  Note: There might be IO domains depending on the IO services of this root domain. These IO domains must be shut down prior to shutting down the root domain. You can view the list of dependent IO domains in the details view of the root domain (from the Physical Hosts view) within the SuperCluster Virtual Assistant.
    
  Note: While it might be feasible to only shut down a single root domain and/or perform a hot-plug operation to add the FC adapter, this is not be supported in a SuperCluster environment.

Physically install the FC card in this location.

The physical installation, including any component shutdown required is not covered in this document. Please use the guidance provided in the installation guide of the adapter.

Restart the root domain to which the FC adapter has been added.  Use the guidance from the Service Manual of your system to shut down the root domain and prepare for physical installation of the adapter.

Verify that the FC adapter is recognized by Solaris in the root domain

To verify that the FC adapter is installed and recognized correctly, check the FC availability with the “ldm ls-io” command in the control domain of the physical host. For example, if you installed one FC adapter in slot "/SYS/CMIOU3/PCIE1” from the previous example, you can verify this with:

The last two lines show that the FC adapter has two physical ports (also called physical functions or PF). This verifies that the physical installation was successful. Note the names for the two ports of the FC adapter: /SYS/CMIOU3/PCIE1/IOVFC.PF0 and /SYS/CMIOU3/PCIE1/IOVFC.PF1

If this should fail, please check back with the prerequisites section above.

Create Virtual Functions (VF) for the FC adapter

Virtual Functions are the virtual entities which the FC adapter will provide to the PCIe subsystem, allowing IO domains to share the physical ports. The number of VFs possible depends on the IO adapter. To check how many VFs are supported by the FC adapter installed, use the “ldm ls-io” command and check for “maxvfs”. For example, the above adapter provides two physical functions. This example shows how to query the first one:

The output shows that this adapter (Emulex, Model: 7101684) supports 8 virtual functions per port.  The Qlogic adapter (Model: 7023303) supports 16 VFs per port.  The SuperCluster Virtual Assistant (SVA) supports up to 16 VFs per port.

You will now need to create these virtual functions to make them available to the IO Domain subsystem. To do so, use the command “ldm create-vf -n <num_of_VFs_to_create> <FC_PF>” for each of the ports (or physical functions) you installed. For example:

Verify that VFs have been created as expected

To verify that the VFs have been created, again use the “ldm ls-io” command. For example:

# ldm ls-io -l /SYS/CMIOU3/PCIE1/IOVFC.PF0
NAME                                       TYPE   BUS      DOMAIN    STATUS
----                                       ----   ---      ------    ------
/SYS/CMIOU3/PCIE1/IOVFC.PF0                PF     pci_18   ssccn1-dom2 
[pci@312/pci@1/SUNW,emlxs@0]
    maxvfs = 8
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF0            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,2]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:fb:04:fa
        node-wwn = 20:00:00:14:4f:fb:04:fa
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF1            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,3]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:fb:b8:9b
        node-wwn = 20:00:00:14:4f:fb:b8:9b
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF2            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,4]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:f8:45:60
        node-wwn = 20:00:00:14:4f:f8:45:60
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF3            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,5]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:f8:26:5a
        node-wwn = 20:00:00:14:4f:f8:26:5a
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF4            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,6]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:fb:d9:ee
        node-wwn = 20:00:00:14:4f:fb:d9:ee
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF5            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,7]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:fb:2c:e4
        node-wwn = 20:00:00:14:4f:fb:2c:e4
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF6            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,8]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:f8:68:37
        node-wwn = 20:00:00:14:4f:f8:68:37
        bw-percent = 0
/SYS/CMIOU3/PCIE1/IOVFC.PF0.VF7            VF     pci_18             
[pci@312/pci@1/SUNW,emlxs@0,9]
    Class properties [FIBRECHANNEL]
        port-wwn = 10:00:00:14:4f:fb:86:12
        node-wwn = 20:00:00:14:4f:fb:86:12
        bw-percent = 0

Trigger and verify the inclusion of the newly created VFs in the IO resource pool of the SuperCluster Virtual Assistant

As a final step, we need to trigger the FC VFs to be recognized by the SuperCluster Virtual Assistant. This job is performed by the automated Health Monitor checks which run on the SuperCluster Virtual Assistant once every hour and every time a new IO Domain is created or destroyed.

If you do not wish to wait for the hourly run of the Health Monitor, you can create a dummy IO Domain to trigger the Health Monitor execution, which in turn process the addition of the new FC resources.

After the new FC HBA has been added and virtualized per the above steps, the next time the Health Monitor runs, the new FC resources will be detected by the Health Check called "Query the Resource Allocation Engine for Resource Changes". This Health Check will show up in the Health Monitor navigation tab with status "Automatically Resolved", as shown in the screenshot below:

Clicking on the "View" link in the "Details" column will display information on the new FC VFs that have been added to the SuperCluster Virtual Assistant. You should check for and recognize the names of the VFs from the output of “ldm create-vf” from above:

The FC resources are now available to use in new IO Domain deployments.   The current software version does not support adding FC VFs to previously existing IO domains.

Saving the New Hardware Configuration

After successfully adding the VFs to the configuration, this new configuration needs to be saved in the system's service processor.  Otherwise, it would be lost after the next power cycle of the system.  This needs to be done from the same control domain where the configuration changes were applied.  This is done in two steps:

1. Come up with a name for the new configuration
2. Save the configuration

The name should be something descriptive.  To save the configuration, use the command "ldm add-spconfig <config name>".  For example:

Saving this configuration will trigger a health check event in SVA, which can be manually acknowledged.

This completes the adding of VFs to the root domain.

Additional Reading

• Oracle VM Server for SPARC Admin Guide
  https://docs.oracle.com/cd/E69554_01/html/E69557/index.html
• SuperCluster Documentation
  
  • T5-8
    https://docs.oracle.com/cd/E40166_01/index.html
    
    • T5-8 Service Manual
      https://docs.oracle.com/cd/E35078_01/html/E35082/index.html
  • M6-32
    https://docs.oracle.com/cd/E41531_01/index.html
    • M6-32 Service Manual
      https://docs.oracle.com/cd/E24355_01/pdf/E41217.pdf
  • M7
    https://docs.oracle.com/cd/E58626_01/index.html
    • M7-8 Service Manual
      https://docs.oracle.com/cd/E55211_01/html/E55215/index.html
• Fibre Channel Adapter Documentation
  • QLogic 16GBit FC HBA Installation Guide
    https://docs.oracle.com/cd/E24651_01/html/E24460/index.html
    • One-Time Firmware Update
      https://docs.oracle.com/cd/E24651_01/html/E24460/z40007c01004751.html
  • Emulex 16GBit FC HBA Installation Guide
    https://docs.oracle.com/cd/E24649_01/html/E24462/index.html

References

This solution has no attachment