How to restore a LDOM config across M5/M6 expandable property change

Asset ID:	1-72-1640383.1
Update Date:	2017-10-04
Keywords:

Solution Type Problem Resolution Sure

Solution 1640383.1 : How to restore a LDOM config across M5/M6 expandable property change

Applies to:

SPARC M6-32 - Version All Versions to All Versions [Release All Releases]
SPARC M5-32 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

If you change the expandable property from true to false or vice versa, after logical domains(LDOMs) are created, you will see similar messages like the ones below, during POST. This will prevent the LDOMs from booting.

:
2014-03-26 06:15:18 24:0:0> DEBUG:   Storing PlatMD to memory
2014-03-26 06:15:18 24:0:0> DEBUG:   stored plat md to 0x65e0000 (mdp=1d5d850) size=22288
2014-03-26 06:15:19 24:0:0> WARNING: HOST expandable property must be set to true in order to boot config
2014-03-26 06:15:20 24:0:0> WARNING: Unable to boot config_kon3 due to missing resources   *
2014-03-26 06:15:20 24:0:0> DEBUG:   Trying to fall back to degraded config
2014-03-26 06:15:20 24:0:0> DEBUG:   Can't open degraded cfg "config_kon3" - rv = -6
2014-03-26 06:15:20 24:0:0> DEBUG:   Degraded config doesn't exist
2014-03-26 06:15:20 24:0:0> WARNING: Falling back to factory-default
2014-03-26 06:15:20 24:0:0> NOTICE: Booting config = factory-default
2014-03-26 06:15:20 24:0:0> DEBUG:   Updating mdset-boot-reason prop: "1""config_kon3""factory-default"
2014-03-26 06:15:20 24:0:0> DEBUG:   bootconfig differs from last boot
2014-03-26 06:15:20 24:0:0> DEBUG:   clearing TOD offset
2014-03-26 06:15:20 24:0:0> DEBUG:   clear_tod_offset_in_PDomain_info
2014-03-26 06:15:20 24:0:0> DEBUG:   ldm_set_bootedconfig_name: New bootedcfg factory-default, last bootedcfg config_kon3
:

* "config_kon3" here is an original LDOM config stored at the ILOM. Upon changing the expandable property, the PDomain will fail back to "factory-default" and the original LDOM config will be degraded during POST.

Assuming the ldmd/recovery_mode is not enabled. (Ref: ldmd/recovery_mode )

Changes

PDomain expandable values changed from true to false or vice versa after LDOMs were created.

Cause

After LDOMs are created, the expandable value should NOT change. The reason is that, the PDomain type(expandable= true/false) affects the physical address assignment of the devices in the PDomain.

If someone does change the expandable value After LDOMs are created (ie, expandable from true to false or vice versa) the LDOM configurations are set to "factory-default" and you will not be able to boot the LDOMs after the PDomain is restarted.

Solution

If you Must change the expandable values, then you will need to save the constraints in an XML file first, Prior to the expandable value changes.

Follow these steps to create the constraints in an XML file in the primary domain to allow the LDOMs to recover after the expandable value changes.

Here are the steps to restore a LDOM config across the expandable property change. Please read "Caution" before staring this procedure.
(Assuming a single DCU PDomain (expandable = true) with LDOM, the ldmd/recovery_mode is not enabled. Changing the expandable property from true to false.)

Step#1) Stop and unbind all LDOM(s).

     primary# ldm list-domain
     NAME             STATE      FLAGS   CONS    VCPU MEMORY   UTIL NORM UPTIME
     primary          active     -n-cv-     UART    48      12G       0.0% 0.0% 2h 22m
     ldom1            active     -n----     5000     16 2G       0.0% 0.0% 1h 57m
     primary# ldm stop-domain -a
     primary# ldm unbind-domain ldom1
     primary# ldm list-domain
     NAME             STATE      FLAGS   CONS    VCPU MEMORY   UTIL NORM UPTIME
     primary          active     -n-cv- UART 48      12G         0.7% 0.7% 2h 24m
     ldom1            inactive   ------            16    2G
     primary#

Step#2) Create an XML file, alldomains.xml, which contains the constraints for all the domains on the system.

primary# ldm list-constraints -x > /ldombackup/alldomains.xml

Step#3) Shutdown/Stop the PDomain.

-> stop /hostX

Step#4) Change expandable property to false.

-> set /hostX expandable=false
Set 'expandable' to 'false'

Step#5) Start the PDomain and console.

-> start /hostX
-> start /hostX/console

      You will see like the following messages during POST.

     :
     2014-03-26 06:50:15 24:0:0> WARNING: HOST expandable property must be set to true in order to boot config
     2014-03-26 06:50:16 24:0:0> WARNING: Unable to boot config_kon3 due to missing resources
     2014-03-26 06:50:16 24:0:0> DEBUG:   Trying to fall back to degraded config
     2014-03-26 06:50:16 24:0:0> DEBUG:   Can't open degraded cfg "config_kon3" - rv = -6
     2014-03-26 06:50:16 24:0:0> DEBUG:   Degraded config doesn't exist
     2014-03-26 06:50:16 24:0:0> WARNING: Falling back to factory-default
     2014-03-26 06:50:16 24:0:0> NOTICE: Booting config = factory-default
     2014-03-26 06:50:16 24:0:0> DEBUG:   Updating mdset-boot-reason prop: "1""config_kon3""factory-default"
     :

     i.e. The original LDOM config stored at ILOM will be degraded during POST.

Step#6) Once you boot the domain, verify that there are no LDOMs configured and the system is in the factory-default configuration.

     # ldm list-domain
     NAME             STATE      FLAGS   CONS    VCPU MEMORY   UTIL NORM UPTIME
     primary          active     -n-c-- UART    384    4193024M 0.1% 0.1% 38m
     # ldm list-config | grep "factory-default"
     factory-default [current]
     #

Step#7) Restore the LDOM config from the XML file.

# ldm init-system -r -i /ldombackup/alldomains.xml

The primary domain is restarted automatically.

Step#8) Once you boot the domain, verify that the LDOM config is restored.

     primary# ldm list-domain
     NAME             STATE      FLAGS   CONS    VCPU MEMORY   UTIL NORM UPTIME
     primary          active     -n-cv- UART    48 12G     2.8% 2.8% 4m
     ldom1            inactive   ------          16    2G
     primary# ldm list-bindings

Step#9) Save the LDOM config to the ILOM, verify that the LDOM config is properly set.

primary# ldm add-config config_restore
primary# ldm list-config

Step#10) Bind and start the LDOM(s).

     primary# ldm bind-domain ldom1
     primary# ldm start-domain -a
     NAME             STATE      FLAGS   CONS    VCPU MEMORY   UTIL NORM UPTIME
     primary          active     -n-cv-     UART    48      12G     0.1% 0.1% 2h 36m
     ldom1            active     -t----      5000    16   2G        6.3% 6.3% 3m
     primary#

Caution - The ldm init-system command might not correctly restore a configuration in which physical I/O commands have been used. Such commands are ldm add-io, ldm set-io, ldm remove-io, ldm create-vf, and ldm destroy-vf. For more information, see ldm init-system Command Might Not Correctly Restore a Domain Configuration on Which Physical I/O Changes Have Been Made in Oracle VM Server for SPARC 3.1.1 and 3.1 Release Notes and the reference:1575852.1 for procedure.

The Oracle VM Server for SPARC 3.1 Admin guide and release notes can be found here:

Oracle VM Server for SPARC 3.1 Administration Guide

Oracle VM Server for SPARC 3.1.1 and 3.1 Release Notes

References

<NOTE:1575852.1> - Using the ldm init-system Command to Restore Domain Configurations on Which Physical I/O Changes Have Been Made
<NOTE:1464421.1> - Configuration, Save & Restore Setup and Troubleshooting of Oracle VM Server for SPARC (LDom)
<NOTE:1674918.1> - LDOM fails to start after configuration changes or faults

Attachments

This solution has no attachment