SPARC T7 / M7 Servers : How to install Solaris on a Physical Domain using VersaBoot

Asset ID:	1-71-2094649.1
Update Date:	2017-10-11
Keywords:

Solution Type Technical Instruction Sure

Solution 2094649.1 : SPARC T7 / M7 Servers : How to install Solaris on a Physical Domain using VersaBoot - iSCSI over IPoIB

Applies to:

SPARC T7-1 - Version All Versions to All Versions [Release All Releases]
SPARC T7-4 - Version All Versions to All Versions [Release All Releases]
SPARC T7-2 - Version All Versions to All Versions [Release All Releases]
SPARC M7-8 - Version All Versions to All Versions [Release All Releases]
SPARC M7-16 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

VersaBoot introduces a new boot model that enables Solaris to store its root filesystem on storage that is not directly bootable by firmware,
such as iSCSI/IPoIB. The aspect of VersaBoot that enables this capability is that the boot archive is stored in separate storage that is directly
bootable by firmware, on the M7 platform the eUSB device on each CMIOU board can be used.

Solution

Below is an example from the M7 DCU PDOM, but should be applicable to the other M7 configurations and the T7 platforms.

The Solaris installation is started at single user mode exit out to a shell prompt.

It is from the shell prompt the InfiniBand interfaces need to be configured.

Welcome to the Oracle Solaris installation menu

1 Install Oracle Solaris
2 Install Additional Drivers
3 Shell
4 Terminal type (currently xterm)
5 Reboot

Please enter a number [1]:3

 
Configure the InfiniBand IB interfaces.

# dladm show-ib
   LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
   net3 10E0000132F148 10E0000132F14A 2 up -- -- FFFF
   net2 10E0000132F148 10E0000132F149 1 up -- -- FFFF

# dladm create-part -l net2 -P 0xFFFF ib0

# dladm show-part
LINK PKEY OVER STATE FLAGS
ib0 FFFF net2 unknown ----

# ipadm create-ip ib0

# ipadm create-addr -a 192.168.28.2/24 ib0/v4

At this point you should be able to communicate to the storage, which in this example is at 192.168.28.1

# ping 192.168.28.1


Start the iscsi smf services.

# svcs iscsi/initiator
STATE STIME FMRI
disabled 15:05:53 svc:/network/iscsi/initiator:default

# svcs -d iscsi/initiator
   STATE STIME FMRI
   disabled 15:05:53 svc:/network/service:default
   online 15:06:04 svc:/network/loopback:default
   online 15:06:11 svc:/milestone/network:default

# svcadm enable network/service
# svcadm enable iscsi/initiator

# svcs iscsi/initiator
STATE STIME FMRI
online 15:16:32 svc:/network/iscsi/initiator:default

 
The initiator node iqn name can be now found at this point.

# iscsiadm list initiator-node
   Initiator node name: iqn.1986-03.com.sun:01:0010e05db9cf.568d2d51
   Initiator node alias: -
   Login Parameters (Default/Configured):
   Header Digest: NONE/-
   Data Digest: NONE/-
   Max Connections: 65535/-
   Authentication Type: NONE
   RADIUS Server: NONE
   RADIUS Access: disabled
   Tunable Parameters (Default/Configured):
   Session Login Response Time: 60/-
   Maximum Connection Retry Time: 180/-
   Login Retry Time Interval: 60/-
   Configured Sessions: 1


The initiator node iqn name should be configured as needed in the storage appliance. 
On a ZFS appliance this needs to be added in the SAN - iSCSI - Initiators menu.


Exit from the single used shell, back to the Solaris installation menu

# exit

Then continue with the Solaris installation.

Welcome to the Oracle Solaris installation menu

1 Install Oracle Solaris
2 Install Additional Drivers
3 Shell
4 Terminal type (currently xterm)
5 Reboot

Please enter a number [1]:1

Discovery Selection

Select discovery method for disks

Local Disks Discover local disks

iSCSI Discover iSCSI LUNs <------- Select this option in bold

Add the IP address for the iSCSI Discovery target.

Disks

Where should Oracle Solaris be installed?
Minimum size: 3.0GB Recommended minimum: 5.0GB

Type Size(GB) Boot Device
──────────────────────────────────────────────────────────────────────
unknown 1490.4 /SYS/CMIOU12/PCIE3/NVME INTEL
iSCSI 600.0 c0t600144F093355C6E0000566BD7470006d0

Once the iSCSI disk is selected the boot pool menu should be displayed, this is from a single DCU M7 Pdom with 2 CMIOUs hence 2 eUSB disks available. All of the eUSB disks available should be selected.

Important : Best practices are to always have 2 or more devices in the boot pool.To avoid any problem when replacing the CMIOU or the eUSB disk itself.

See SPARC M7 Servers : VersaBoot (iSCSI over IPoIB) - CMIOU/eUSB replacement considerations (Doc ID 2107700.1)

Boot Pool

Which disks should be selected as boot pool?
Minimum size: 0.5GB

Selected Dedicated Device Size(GB) Type
─────────────────────────────────────────────────────────────────────────
Yes Yes /SYS/CMIOU13/EUSB-DISK 1.9 usb
Yes Yes /SYS/CMIOU12/EUSB-DISK 1.9 usb
No No /SYS/CMIOU12/PCIE3/NVME 1490.4 unknown

Continue with the Solaris installation and reboot on the newly installed OS.

The Boot processes should show the eUSB disk is being used for the first part of the boot.

Boot device: /pci@340/pci@2/usb@0/storage@1/disk@0,0:a File and args:
SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved.
NOTICE: Configuring iSCSI to access the root filesystem...

Once the OS is booted the format output on the installed Solaris host shows the iSCSI LUN and the two eUSB disks.

# format
Searching for disks...done

   AVAILABLE DISK SELECTIONS:
   0. c2t0d0 <MICRON-eUSB DISK-1112-1.89GB>
   /pci@340/pci@2/usb@0/storage@1/disk@0,0
   /dev/chassis/SYS/CMIOU12/EUSB-DISK/disk
1. c1t1d0 <INTEL-SSDPEDME016T4S-8DV1-1.46TB>
   /pci@33d/pci@1/nvme@0/disk@1
   /dev/chassis/SYS/CMIOU12/PCIE3/NVME/disk
   2. c4t0d0 <MICRON-eUSB DISK-1111-1.89GB>
   /pci@345/pci@2/usb@0/storage@1/disk@0,0
   /dev/chassis/SYS/CMIOU13/EUSB-DISK/disk
   3. c0t600144F093355C6E0000566BD7470006d0 <SUN-ZFS Storage 7420-1.0-600.00GB>
   /scsi_vhci/ssd@g600144f093355c6e0000566bd7470006

Checking the ZFS configuration for the root and boot pool, both eUSB disks are in the mirrored bpool.

# zpool status
   pool: bpool
   state: ONLINE
   scan: none requested
   config:

      NAME STATE READ WRITE CKSUM
      bpool ONLINE 0 0 0
            mirror-0 ONLINE 0 0 0
            c4t0d0 ONLINE 0 0 0
            c2t0d0 ONLINE 0 0 0

   pool: rpool
   state: ONLINE
   scan: none requested
   config:

      NAME STATE READ WRITE CKSUM
      rpool ONLINE 0 0 0
          c0t600144F093355C6E0000566BD7470006d0 ONLINE 0 0 0

The IB interface used for the root pool is named sys-root0.

# dladm
   LINK CLASS MTU STATE OVER
   net1 phys 1500 unknown --
   net5 phys 1500 up --
   net3 phys 65520 up --
   net2 phys 65520 up --
   sys-root0 part 65520 up net2       <--------------
   net0 phys 1500 unknown --
   sp-host0 vnic 1500 up net5

References:

Additional information can be found in the Solaris 11 installation documentation.

Installing Oracle® Solaris 11.3 Systems

VersaBoot: Oracle Solaris Boot Process

For the boot device path decoding for the M7 SPARC M7 Series Servers: Device Paths (Doc ID 2063247.1)

Also see SPARC T7 / M7 Servers : Information about VersaBoot - iSCSI over IPoIB (Doc ID 2094741.1)

M7 Boot video on Otube

Attachments

This solution has no attachment