SPARC T3-1
 

PLA-Support>Sun Systems>SPARC>CMT>SN-SPARC: T3
 

There are times when the PCI card being used by the Operating system boot disks has to be moved.
It may be the slot we are using is not supported for the card. and the card has to be relocated to another PCI slot.
The document outlines the considerations and actions required.

Applies to:

Goal

To enable the booting of a Server, after changing PCIe Slot Of The Boot Disk's SCSI SAS Host Bus Adapter (HBA) Card

When the niwot card slot location is changed, the controller definition of original HW raid volumes changes at the Open Boot Prom (OBP) 

The System can no longer boot up successfully, as OBP no longer points to a valid device definition.

Even correcting the OBP device path may not ensure a successful boot, if the Operating system device tree has not been updated as well.

This document attempts to give a high level overview of the process, considerations required and show in some detail the reconfiguration at the Server Open Boot Prom (OBP) using a simple example of a HW Raid Volume, with no logical volume manager being used, for example Solaris Volume Manager (SVM) or Veritas Volume Manager or Oracle's ZFS

Solution

General Considerations are required before proceeding to ensure success, avoid problems and potential data corruption.

It is important to determine the configuration of the Operating system boot disks with regard to file system types and the use of any logical volume manager software for mirroring,

before the PCIe card is relocated, to understand the impact and work required to reconfigure the Operating System, before, during and after any physical changes.

Seek assistance from Oracle Support if there is any doubt. 

General High Level Steps

We wil provide a simple example of the steps highlighted with "*"

   Identify what kind of filesystem and if the boot disks are under the control of any logical volume manager or h/w raid used for the purposes of mirroring.

   If any form of mirring is used determine if extra steps are required to ensure data integrity - for example reconfiguration to not use any logical volume manager.

   Take backups of the boot disks before proceeding.

   Record the current boot configuration.

   Reconfigure the Operating System disks if pre work is required - like removing the logical volume manager.

   Configure the system to not "auto-boot" from power on.

   Shut down the system, power it off and relocate the PCI card, Power on the system 

*  1) Determine at the OBP the new device path/definition to the boot disk, and create a new device alias to point the Open Boot Prom to the boot disk

*  2) Boot the system either from cdrom, network jumpstart image or the target boot disks in failsafe mode, and mount the root filesystem

*  3) Reconfigure the root filesystem as needed, this may include ; -

                 rebuilding the device tree using devfsadm

                 editing /etc/vfstab to point to the changed boot disk device name,

                 editing /etc/vfstab to point to the changed primary swap device

                 configuring the crash dump utility (dumpadm) to point to the correct dump device

 *  4) Once the system succesfully boots, with the SAS HBA in its new location, re-introduce any logical volume manager (if deconfigured) and remirror the disks as needed.

Ensure the "auto-boot" option is renabled, and ensure the system boots succesfully from any mirror disks by testing.

Example of a reconfiguration exercise where the Root disk volume is a simple UFS filesystem residing on a HW Raid configuration. No Logical Volume Manager was being used.

      We are moving the card from slot 4 to slot 3

1)  Boot the Server to the OBP prompt, then determine the new device path.

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y

Serial console started.  To stop, type #.

...

{0} ok 

1.1) From OK prompt, find NEW hw raid volume path using either probe-scsi-all, or show-disks.

{0} ok show-disks
a) /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@2/disk   
b) /pci@400/pci@2/pci@0/pci@4/scsi@0/disk
c) /pci@400/pci@1/pci@0/pci@8/LSI,mrsas@0/disk                  <     <<<<<<<<< This means that Niwot card is in slot 0
d) /pci@400/pci@1/pci@0/pci@4/scsi@0/disk
e) /iscsi-hba/disk
q) NO SELECTION 
Enter Selection, q to quit: q

1.2) From OK prompt, create hw raid volume device aliase using nvalias

{0} ok nvaliase  test /pci@400/pci@1/pci@0/pci@8/LSI,mrsas@0/disk     <<<<<<< to create alias for disk in new path

2) Boot the server in failsafe mode, from newly created alias, and mount the root filesystem.

    We use the following procedure to rebuild device tree information. 1008598.1 - How to rebuild the path_to_inst file and /dev/dsk and /devices directories

    However in our case we boot the original Operating System disk in failsafe mode, where as the above document shows alternative methods.

    The above document is highlighted as it contains very important advice regarding actions to take when the Operating a system disks are under control of a logical volume manger.

See also

{0} ok boot test -F failsafe
Boot device: /pci@400/pci@1/pci@0/pci@8/LSI,mrsas@0/disk  File and args: -F failsafe

FW package: 12.12.0-0048
SunOS Release 5.10 Version Generic_147440-01 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Configuring devices.
Searching for installed OS instances...

Multiple OS instances were found. To check and mount one of them
read-write under /a, select it from the following list. To not mount
any, select 'q'.

  1  /dev/dsk/c0t0d0s0             Oracle Solaris 10 8/11 s10s_u10wos_17b SPARC


Please select a device to be mounted (q for none) [?,??,q]: 1
mounting /dev/dsk/c0t0d0s0 on /a
** /dev/rdsk/c0t0d0s0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cylinder Groups

CORRECT BAD CG SUMMARIES FOR CG 2060? y      <<<<< let fsck repair the file system. This will enable it, to mount the said fs on /a 

CORRECTED SUPERBLOCK SUMMARIES FOR CG 2060
CORRECTED SUMMARIES FOR CG 2060
FRAG BITMAP WRONG 
FIX? y

CORRECTED SUPERBLOCK SUMMARIES FOR CG 2061
CORRECTED SUMMARIES FOR CG 2061
FRAG BITMAP WRONG (CORRECTED)

... cut .....

CORRECT GLOBAL SUMMARY
SALVAGE? yes

Log was discarded, updating cyl groups
402970 files, 13379136 used, 559771215 free (10063 frags, 69970144 blocks, 0.0% fragmentation)

***** FILE SYSTEM WAS MODIFIED *****

Starting shell.
# 

3) Update the device tree and /etc/vfstab to ensure the Operating system points to the correct devices

  Modify /etc/vfstab file to reflect to appropriate disk information for the root device and primary swap

  Modify the dump device to point to the new location using "dumpadm -r /dev/..."  See the dumpadm man page for details

  Remove the device entries and recreate them using devfsadm, then reconfigure reboot to the system disk:

# rm /a/dev/dsk/c* /a/dev/rdsk/c*
# devfsadm -r /a -p /a/etc/path_to_inst
# umount /a
# reboot -- -r

4) Perform a reboot of the Operating System from the relocated HBA

References

This solution has no attachment