VersaBoot - iSCSI over IPoIB : How to Access Solaris Boot Environment When Booted from rcdrom/net

Asset ID:	1-71-2197347.1
Update Date:	2017-10-31
Keywords:

Solution Type Technical Instruction Sure

Solution 2197347.1 : VersaBoot - iSCSI over IPoIB : How to Access Solaris Boot Environment When Booted from rcdrom/net

Applies to:

Solaris Operating System - Version 11.3 and later
SPARC M7-8 - Version All Versions and later
SPARC M7-16 - Version All Versions and later
SPARC T7-1 - Version All Versions and later
SPARC T7-2 - Version All Versions and later
Oracle Solaris on SPARC (64-bit)

Goal

VersaBoot introduces a new boot model that enables Solaris to store its root filesystem on storage that is not directly bootable by firmware,
such as iSCSI/IPoIB. The aspect of VersaBoot that enables this capability is that the boot archive is stored in separate storage that is directly
bootable by firmware, on the M7/M8 and T7/T8 platforms the eUSB device on each CMIOU or motherboard board can be used.

When booted from Solaris text-install iso image or from net (through AI server), the iSCSI LUN for the root pool is not visible so the pool cannot be imported.

The purpose of this document is to describe the required steps to import the root pool and mount the Solaris boot environment.

This can be required if the system does not allow logging in and you need to make changes to the BE from an alternate boot environment.

Solution

1. Retrieve the information about the location of the iSCSI LUN

This info is part of the OBP. The specific OBP variable 'os-root-device' defines devices and root file systems for root pools.

{c00} ok printenv os-root-device
os-root-device=osroot-type=ZFS/iSCSI/IPv4/IPoIB;osroot-iscsi-port=3260;osroot-iscsi-target-ip=192.168.28.1;osroot-iscsi-partition=a;osroot-iscsi-initiator-id=iqn.1986-03.com.sun:01:0010e05db9cf.57ed31de;osroot-iscsi-lun=0;osroot-iscsi-target-name=iqn.1986-03.com.sun:01:0010e05db9cf.57ed31de;osroot-subnet-mask=255.255.255.0;osroot-host-ip=192.168.28.2;osroot-path=/pci@33c/pci@1/pciex15b3,1003@0:port=1,pkey=FFFF,protocol=ip

The sames in a more readable form:

os-root-device=osroot-type=ZFS/iSCSI/IPv4/IPoIB;
osroot-iscsi-port=3260;
osroot-iscsi-target-ip=192.168.28.1;
osroot-iscsi-partition=a;
osroot-iscsi-initiator-id=iqn.1986-03.com.sun:01:0010e05db9cf.57ed31de;
osroot-iscsi-lun=0;
osroot-iscsi-target-name=iqn.1986-03.com.sun:01:0010e05db9cf.57ed31de;
osroot-subnet-mask=255.255.255.0;
osroot-host-ip=192.168.28.2;
osroot-path=/pci@33c/pci@1/pciex15b3,1003@0:port=1,pkey=FFFF,protocol=ip

2. Boot from alternate boot device (rcdrom / net )

For issues to boot from rcdrom/net, please consult Platform support group

SPARC M7-16, No Keyboard
Copyright (c) 1998, 2016, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.40.1, 477.0000 GB memory installed, Serial #106805695.
Ethernet address 0:10:e0:5d:b9:cf, Host ID: 865db9bf.

{c00} ok devalias
screen /pci@345/pci@1/pci@0/pci@c/display@0
fallback-miniroot /pci@345/pci@1/pci@0/pci@8/usb@0/storage@2/disk@0
rcdrom /pci@345/pci@1/pci@0/pci@8/usb@0/storage@2/disk@0
disk /pci@33d/pci@1/nvme@0/disk@1
virtual-console /virtual-devices/console@1
name aliases

-> show -d properties /Servers/PDomains/PDomain_3/SP/services/kvms/host_storage_device
/Servers/PDomains/PDomain_3/SP/services/kvms/host_storage_device
Properties:
mode = miniroot
status = operational

rcdrom is configured to boot on fallback-miniroot

For more details : SPARC T7 / M7 Servers : Information about VersaBoot - iSCSI over IPoIB <Document 2094741.1>

If the mode is set to miniroot, change mode to 'disabled' to boot from local ISO image or to 'remote' to boot from nfs/samba remote ISO image

The location of the text-install ISO image needs to be set from ILOM GUI - for more details : SPARC S7/T7 rKVM Storage redirection fails to boot with sol-11_3-text-sparc.iso <Document 2102941.1>

For the purpose of this document, I've set up the mode to "remote" and use "nfs://xxx.xxx.xxx.xxx:/export/iso/11/s11.3/ga/sol-11_3-text-sparc.iso" for the location of my ISO image.

If there is an AI server that serves this client, you can do

{c00} ok boot net:dhcp

so there is no need to specify the location of an ISO image.

-> cd /Servers/PDomains/PDomain_3/SP/services/kvms/host_storage_device
/Servers/PDomains/PDomain_3/SP/services/kvms/host_storage_device

-> set mode=remote

-> cd /HOST3

-> set bootmode script="setenv auto-boot? false"

-> start console -f
Are you sure you want to start /HOST3/console (y/n)? y
Warning: User root currently has write permission to this console
and forcibly removing them will terminate any current write actions and
all work will be lost. Would you like to continue (y/n)? y

Serial console started. To stop, type #.

{c00} ok reset-all
{c00} ok boot rcdrom
Boot device: /pci@345/pci@1/pci@0/pci@8/usb@0/storage@2/disk@0 File and args:
SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Probing for device nodes ...
Preparing image for use
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
Done mounting image
USB keyboard
1. Arabic 15. Korean
2. Belgian 16. Latin-American
3. Brazilian 17. Norwegian
4. Canadian-Bilingual 18. Portuguese
5. Canadian-French 19. Russian
6. Danish 20. Spanish
7. Dutch 21. Swedish
8. Dvorak 22. Swiss-French
9. Finnish 23. Swiss-German
10. French 24. Traditional-Chinese
11. German 25. TurkishQ
12. Italian 26. UK-English
13. Japanese-type6 27. US-English
14. Japanese
To select the keyboard layout, enter a number [default 27]:

[...]

Welcome to the Oracle Solaris installation menu

1 Install Oracle Solaris
2 Install Additional Drivers
3 Shell
4 Terminal type (currently xterm)
5 Reboot

Please enter a number [1]:3

3. Setup the network over IB

# dladm show-ib
LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
net7 10E0000132F148 10E0000132F14A 2 up -- -- FFFF
net6 10E0000132F148 10E0000132F149 1 up -- -- FFFF

# dladm show-part
#

# dladm create-part -l net6 -P 0xFFFF ib0

# dladm show-part
LINK PKEY OVER STATE FLAGS
ib0 FFFF net6 unknown ----

# ipadm create-ip ib0
# ipadm create-addr -a 192.168.28.2/24 ib0/v4

At this point you should be able to communicate to the storage, which in this example is at 192.168.28.1

# ping 192.168.28.1
192.168.28.1 is alive

4. Setup iSCSI

For issues to discover the iSCSI LUN, please consult with Storage support group

# svcs iscsi/initiator
STATE STIME FMRI
online 14:24:13 svc:/network/iscsi/initiator:default

If the service is not online, then execute

# svcadm enable -r iscsi/initiator

Setup iSCSI device

# iscsiadm add static-config iqn.1986-03.com.sun:01:0010e05db9cf.57ed31de,192.168.28.1:3260
# iscsiadm modify discovery --static enable

The iSCSI LUN should be visible from format utility

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c2t0d0 <MICRON-eUSB DISK-1112-1.89GB>
/pci@340/pci@2/usb@0/storage@1/disk@0,0
/dev/chassis/SYS/CMIOU12/EUSB-DISK/disk
1. c1t1d0 <INTEL-SSDPEDME016T4S-8DV1-1.46TB>
/pci@33d/pci@1/nvme@0/disk@1
/dev/chassis/SYS/CMIOU12/PCIE3/NVME/disk
2. c4t0d0 <MICRON-eUSB DISK-1111-1.89GB>
/pci@345/pci@2/usb@0/storage@1/disk@0,0
/dev/chassis/SYS/CMIOU13/EUSB-DISK/disk
3. c0t600144F093355C6E0000566BD7470006d0 <SUN-ZFS Storage 7420-1.0-600.00GB>
/scsi_vhci/ssd@g600144f093355c6e0000566bd7470006

5. Import the root pool

# zpool import
pool: bpool
id: 9243399647925667558
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

bpool ONLINE
mirror-0 ONLINE
c2t0d0 ONLINE
c4t0d0 ONLINE

pool: rpool
id: 11508226960569345719
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

rpool ONLINE
c1t1d0 ONLINE

pool: rpool
id: 7966350490478334165
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

rpool ONLINE
c0t600144F093355C6E0000566BD7470006d0 ONLINE

Import the appropriate rpool

# zpool import 7966350490478334165

# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c0t600144F093355C6E0000566BD7470006d0 ONLINE 0 0 0

errors: No known data errors

6. Mount the Solaris BE

# beadm list
be_find_current_be: failed to find current BE name
be_find_current_be: failed to find current BE name
BE Flags Mountpoint Space Policy Created
-- ----- ---------- ----- ------ -------
solaris R - 1.93G static 2016-09-29 15:34

# mkdir /a
# beadm mount solaris /a

Confirm that the BE is now mounted on /a

# mount | egrep '^/a'
/a on rpool/ROOT/solaris read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/mountpoint=/a/dev=4bd0006 on Tue Oct 25 14:34:17 2016
/a/var on rpool/ROOT/solaris/var read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/mountpoint=/a/var/dev=4bd0007 on Tue Oct 25 14:34:17 2016

At this point you can access the filesystem and perform any action you need. e.g.

# pkg -R /a verify
# pkg -R /a fix

7. Cleanup and reboot from VersaBoot

Update boot archive

# bootadm update-archive -vR /a

Unmount BE and take the system to OBP

# cd /
# beadm unmount solaris
# init 0

-> cd /Servers/PDomains/PDomain_3/SP/services/kvms/host_storage_device
/Servers/PDomains/PDomain_3/SP/services/kvms/host_storage_device

-> set mode=miniroot
Set 'mode' to 'miniroot'

From OBP, reset system. You can boot from VersaBoot config now

{c00} ok reset-all
{c00} ok boot

References

<NOTE:2094649.1> - SPARC T7 / M7 Servers : How to install Solaris on a Physical Domain using VersaBoot - iSCSI over IPoIB
<NOTE:2094741.1> - SPARC T7 / M7 / M8 Servers : Information about VersaBoot - iSCSI over IPoIB
<NOTE:2102941.1> - SPARC S7/T7 rKVM Storage redirection fails to boot with sol-11_3-text-sparc.iso
<NOTE:1912037.1> - Zpool Import and Mount of BE When Booted From Alternate Device in Solaris 11

Attachments

This solution has no attachment