Check ldom manager (ldmd)

#svcs ldmd

Video Demo CLI Example

SPARC T5
SPARC M5/M6-32
SPARC M10

Check vntsd is running

Check Default Services are running

Check ldm software 

Usage :ldm -V

Example

#ldm -V

Logical Domains Manager (v 3.0.0.3)
Hypervisor control protocol v 1.11
Using Hypervisor MD v 1.4

System PROM:
Hostconfig  v. 1.3.5.c      @(#)Hostconfig 1.3.5.c 2014/03/21 19:04
Hypervisor  v. 1.12.5.b     @(#)Hypervisor 1.12.5.b 2014/02/07 13:52         
OpenBoot    v. 4.35.5.a     @(#)OpenBoot 4.35.5.a 2014/01/31 14:12
 

 
[ check ldoms manager package in Solaris 11 ]

Usage: pkg info ldomsmanager

Example

eis@m10-1-mysyd04-a:~$ pkg info ldomsmanager
              Name: system/ldoms/ldomsmanager
           Summary: Logical Domains Manager
       Description: LDoms Manager - Virtualization for SPARC T-Series
          Category: System/Virtualization
             State: Installed
         Publisher: solaris
           Version: 3.1.1.0.5
     Build Release: 5.11
            Branch: 0.175.2.0.0.34.0
    Packaging Date: March  3, 2014 02:18:20 PM 
              Size: 3.87 MB
              FMRI: pkg://solaris/system/ldoms/
 ldomsmanager@3.1.1.0.5,5.11-0.175.2.0.0.34.0:20140303T141820Z
    eis@m10-1-mysyd04-a:~$ 
    eis@m10-1-mysyd04-a:~$ 

add virtual console concentrator (vcc)

add virtual network switch (vsw)
   

Usage: ldm add-vsw  net-dev=<device>  <vsw-name> <ldom-name>
            
Example

add virtual disk server (vds)

add virtual storage device to virtual disk service

Usage: ldm add-vdsdev <backend>  <volume@service>
            
where

backend: specify a physical disk path. In addition to a file, you can specify the path of a disk, disk slice etc for the backend.

volume: specify a volume name that is unique within the virtual disk server

service: specify a virtual disk service name
            
Example: 
            
### Primary domain. 
### Add zfs filesystem to existing Guest domain                    

            
In this example  a 5G backend zfs partition is created called ckt-disk2
The volume name is ckt-disk2
The vds is called primary-vds0
         

remove virtual console concentrator (vcc)

remove virtual network switch (vsw)

Usage: ldm remove-vsw  <vsw-service-name>
            
Example     

remove virtual disk server (vds)

Usage: ldm remove-vds  <service-name>
            
Example

remove virtual storage device to virtual disk  service

start ldom manager
 

start vntsd

list resources binded to a Guest Domain   

Usage: ldm  list-bindings <ldom-name>
            
Example

how to identify if the current domain?

[Control,Guest,Service or Root]          

how to check status of I/O device

Usage: ldm list-io         

Example

how to check logical domain (ldom) status

Usage: ldm list-domain -o domain <guest-domain-name>
            
Example

list the status of all the guest domains on the system
 

Usage: ldm list
            
Example

how to manually list the LDOM config on a  system

Usage: ldm list-bindings /  ldm list-bindings  <ldom-name>
            
Example       

list current save LDOM config in Solaris

Usage: ldm list-spconfig
            
Example

when do we need to initiate an LDOM reconfig reboot

Usage: ldm start-reconf primary

The "start-reconf" needs to be executed on the primary domain if 
changes [ add/remove ] is to be made to the resources [ cpu,io
and memory] that is currently assigned to it.
            
Example
            

Check CPU activation

Usage: ldm list-permits

Example           

Check Autoreplacement policy for CPU

Usage:  svccfg -s ldmd listprop ldmd/autoreplacement_policy_cpu
            
Example
            
 o display autoreplacement_policy_cpu property value  use the  following command:
            
    #svccfg -s ldmd listprop ldmd listprop ldmd/autoreplacement_policy_cpu           

Example (from M10-1 system)         

       
Example to change replacement policy :           

issue send break

Usage: CTRL+]   type  “send break” followed by return then another return
            
Example

stop Guest Domain
 

Usage: ldm stop < guest domain name >
            
Example

ubind Guest Domain 
 

Usage: ldm unbind <guest-domain-name>
            

Example

Add Guest Domain
  

Usage :ldm add-domain  <guest domain>
                       

Example

assign cpu threads to Guest Domain

Usage: ldm add-vcpu  <number>  <guest-domain-name>
            

Example

assign vcpu units of cores
 

Usage: ldm add-core, ldm set-core  <number> <ldom>
            
Note: CPUs cannot be assigned to an ldom in units of cores and in units of threads simultaneously. 
Choose either of these units of assignment for each ldom        

assign memory to Guest Domain

Usage:  ldm add-memory <number>  <guest-domain-name>
            ldm set-memory <number>   <guest-domain-name>
            
Note that the Guest domain OS must be up and running to do memory DR.
                      
Example          

add vnet device to Guest Domain
 

Usage: ldm add-vnet  <if-name> <vsw-name> <ldom-name>

adds a virtual network device to a domain
                              
Example

assign disk resource to Guest Domain
 

assign ISO resource to Guest Domain

root@t5-8-sin06-a:# ldm stop ldg0

root@t5-8-sin06-a:ldm add-vdiskserverdevice /export/home/sol-11_2-text-sparc.iso    iso@primary-vds0

root@t5-8-sin06-a:ldm add-vdisk iso iso@primary-vds0 ldg0
root@t5-8-sin06-a:/export/home# ldm start ldg0
LDom ldg0 started
root@t5-8-sin06-a:~# ldm ls -o disk primary
NAME             
primary          

VDS
    NAME             VOLUME         OPTIONS          MPGROUP        DEVICE
    primary-vds0     vol1                                           /ldg0-disk
                     iso                                            /export/home/sol-11_2-text-sparc.iso

root@t5-8-sin06-a:~# telnet localhost 5000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldg0" in group "ldg0" ....
Press ~? for control options ..
NOTICE: Entering OpenBoot.
NOTICE: Fetching Guest MD from HV.
NOTICE: Starting additional cpus.
NOTICE: Initializing LDC services.
NOTICE: Probing PCI devices.
NOTICE: Finished PCI probing.

SPARC T5-8, No Keyboard
Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.37.0.build_29, 64.0000 GB memory available, Serial #83521195.
Ethernet address 0:14:4f:fa:6e:ab, Host ID: 84fa6eab.



{0} ok
{0} ok
{0} ok
{0} ok
{0} ok
{0} ok
{0} ok ~
~ ?
{0} ok
{0} ok
telnet> quit
Connection to localhost closed.
root@t5-8-sin06-a:~# telnet localhost 5000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldg0" in group "ldg0" ....
Press ~? for control options ..

~ ?
{0} ok show-disks
a) /reboot-memory@0
b) /pci@440/pci@1/pci@0/pci@c/SUNW,qlc@0,2/fp@0,0/disk
c) /virtual-devices@100/channel-devices@200/disk@1    <<<------ ISO
d) /virtual-devices@100/channel-devices@200/disk@0
e) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: q
{0} ok

{0} ok devalias
iso                      /virtual-devices@100/channel-devices@200/disk@1   <<<----
vdisk0                   /virtual-devices@100/channel-devices@200/disk@0
vnet0                    /virtual-devices@100/channel-devices@200/network@0
net                      /virtual-devices@100/channel-devices@200/network@0
disk                     /virtual-devices@100/channel-devices@200/disk@0
virtual-console          /virtual-devices/console@1name                     aliases

{0} ok boot iso -s    
Boot device: /virtual-devices@100/channel-devices@200/disk@1  File and args: -s
SunOS Release 5.11 Version 11.2 64-bit
Copyright (c) 1983, 2014, Oracle and/or its affiliates. All rights reserved.
Booting to milestone "milestone/single-user:default".
Remounting root read/write
Probing for device nodes ...

Remove a Guest Domain
 

Remove disk resource from Guest Domain
  

Remove virtual network device from a Guest Domain

Remove CPU threads from a Giuest Domain
      

Usage:   ldm remove-vcpu [-f|--force] [-c|--core] <number> <ldom-name>
            
            
Example

Remove virtual cpu units in cores from a  Guest Domain
      

Usage: ldm remove-core [-f|--force]  <number>|cid=<id>[,<id>[,...]]  <ldom-name>
            
Example

Remove memory from a Guest Domain

start a Guest Domain

Usage:  ldm start (-a | -i <file> |  <ldom>...
            

Example          

save ldom configuration to the SP

Usage:  ldm add-spconfig [-r  <autosave>] <config_name>
            
            
Example

backup of existing configuration from the control domain

Usage: ldm list-constraints -x > < file.xml >

Example

identitify physical resources bindings

usage: ldm list-constraints
           
Example

jack@m5-32-sca11-a-pdom01:~$ ldm list-constraints
DOMAIN
primary
            
UUID
  9d1d44bb-ec6e-41bb-f8e6-c56e8b8621d2
            
MAC
  00:10:e0:24:6a:b1
            
HOSTID
  0x86246a9d
            
CONTROL
  failure-policy=ignore
  extended-mapin-space=on
  cpu-arch=native
  rc-add-policy=
  shutdown-group=0
            
VCPU
COUNT:    288
            
MEMORY
SIZE:    3144448M
            
CONSTRAINT
  threading=max-throughput
            
VARIABLES
 boot-device=/pci@700/pci@1/pci@0/pci@c/pci@0/pci@4/scsi@0/disk@w5000cca0162f5cc5,0:a
 pm_boot_policy=disabled=0;ttfc=22746;ttmr=0;
 use-nvramrc?=true
            
IO
  DEVICE                           OPTIONS
  pci_16                          
  pci_17                          
  pci_18                          
  pci_19                          
  pci_24                          
  pci_25                          
  pci_26                          
  pci_27                          
  pci_20                          
  pci_21                          
  pci_22                          
  pci_23                   

bind resources to a Guest Domain
 

unbind resources to a Guest Domain

login to the console of a Guest Domain
  

Usage: telnet localhost < console port # >
            
            
Example       
              

Exit from a Guest Domain console
 

Usage: CTRL+] then q
            
Example

Enable/Disable console loggging function for  a Guest Domain

Note: OVM 3.0 required 

usage: ldm set-vcons log=off | on  <ldom-name>
           ldm ls - o console < guest domain >

The guest gomain console log will be stored in the following directory

/var/log/vntsd

Display current console settings of a Guest  Domain
 

Collect System information via Explorer command        

Data collection from SPARC M5/M6/M7/M8 SP (service processor)

Data collection from SPARC M10 SP (service processor)

Data collection from SPARC T3/T4/T5/T7 SP (service processor)

Collect ILOM snapshot on T3-x and T4-x systems without an active network management connection (Doc ID 1518044.1)

Troubleshooting data needed for T3-x, T4-x & T5-x servers (Doc ID 1470580.1)

list all Service Domains in the platform from           Solaris

Usage: ldm ls [-e] [-l] [-S] [-o <format>] [-p] [<ldom>...]

format' is one or more of:
console,core,cpu,crypto,disk,domain,memory,network,physio,resmgmt,serial,status

          
Example:               
     

list all LDOM config from SP with timestamp        

Usage: [T3/T4/t5]  show /HOST/domain/configs date_created -t      

Note: For SPARC T3/T4 and T5  "/HOST/domain/configs" will list all the saved configurations and "/HOST/bootmode config"  for the current and active config.    

Example T3/T4/T5    

-> show /HOST/domain/configs date_created -t
Target        |     Property    | Value                                                                   
---------------------------------+--------------+---------------------
/HOST/domain/configs/mydomain    | date_created | 2013-11-13 12:50:27
/HOST/domain/configs/guestdom1   | date_created | 2013-11-18 15:26:58

-> 

Usage  [M5/M6/M7/M8]    show/HOSTx/domain/configs date_created -t  [where x=0..3 ]  

Note: For SPARC M5/M6/M7/M8  "/HOSTx/domain/configs" will list all the saved configurations and "/HOSTx/bootmode config"  for the current and active config.

Example M5/M6

list current LDOM config from SP
  

Usage [ T3/T4/T5]  show /HOST/domain/config -t

Example T3/T4/T5

-> show /HOST/bootmode config -t
Target                    | Property         | Value                                                                   
--------------------------+------------------+--------------------
/HOST/bootmode            | config           | guestdom1

-> 

Usage  [M5/M6/M7/M8]     show/HOSTx/domain/configs  -t  [where x=0..3 ]   

Example M5/M6/M7/M8     

Usage [M10]  showdomainconfig -p PPAR-ID  where PPAR-ID=0 ...15

Check virtualization info
     

Usage: virtinfo -a
             
The virtinfo  utility provides  current domain  information
            
Primary domain output

           
Guest domain output

Usage:  set  /HOST/send_break_action=dumpcore

Usage:  set  /HOSTn/send_break_action=dumpcore           
               where n=0 ...3

Example

-> pwd
Current default target: /HOST0
             
-> set send_break_action=dumpcore
Set 'send_break_action' to 'dumpcore'
             
-> start /HOST0/console
Are you sure you want to start /HOST0/console (y/n)? y
             
Serial console started.  To stop, type #.
             
           
panic[cpu508]/thread=2a108c77c60: sync initiated
             
sched: trap type = 0x0
pid=0, pc=0x0, sp=0x0, tstate=0x0, context=0x0
o0-o7: 0, 0, 0, 0, 0, 0, 0, 0

Usage:  XSCF>reset -p <ppar_id> -g primary  panic

Crash a guest domain from the control domain

Usage:  ldm panic-domain <ldom-name>

Example

eis@m10-1-mysyd04-a:~$ ldm panic-domain ldom2
eis@m10-1-mysyd04-a:~$ ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    8     8G       0.6%  27d 1h 13m 
ldom1            active     -n----  5000    4     6G       0.0%  11d 18h 57m
ldom2            active     -t----  5001    4     4G       100%  13d 21h 48m
eis@m10-1-mysyd04-a:~$ telnet localhost 5001
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
             
Connecting to console "ldom2" in group "ldom2" ....
Press ~? for control options ..
             
bookable-192.168-57-212.au.reddot.com console login: 
Jul 9 12:41:57 192.168-57-212.reddot.oracle.com savecore:Decompress the crash dump with 
Jul 9 12:41:57 192.168-57-212.au.reddot.com 'savecore -vf /var/crash/vmdump.0'

 Crash Guest Domain from Service Processor

 Usage: reset [ [-q]  -{y|n}]  -p ppar_id  -g domainname  sir
            reset [ [-q]  -{y|n}]  -p ppar_id  -g domainname  panic
             
where ppar_id =0 ...15
domainname = guest domain name
sir = resets ldom
panic = crashes the guest domain

Example

XSCF> reset -p 0 -g ldom2 panic
PPAR-ID :00
GuestDomain to panic : ldom2
Be sure to execute "ldm add-spconfig" before using this 
command when you have changed the ldm configuration.
Otherwise, an unexpected domain might be reset.
Continue? [y|n] :y
00 ldom2 : Resetting

Note:  Use XSCF command "showdomainstatus" to check result of panic.

Guest Domain Console

# 
# uname -a
SunOS 192.168-57-212.au.reddot.com 5.10 Generic_147147-26 sun4v sparc sun4v
# 
panic[cpu0]/thread=2a10001fc80: Panic - Generated at user request
000002a10001f6a0 unix:process_nonresumable_error+2d8 (2a10001f890,0,ff,40,5,40)
%l0-3: 0000000000000100 0000000003000000 0000000000000001000000000180c600
%l4-7: 0000000000000000 0000000100000000 00000000ffffffff0000000000000000
000002a10001f7e0 unix:ktl0+64 (0,1,0,100,1010101,12)
%l0-3: 000000000180c000 0000000000000000 00000000110014040000000001029aa4
%l4-7: 0000000000000000 0000000000000000 0000000000000000000002a10001f890
000002a10001f930 unix:cpu_halt+f4 (180c000, 0, 19dd9b8,19dd888, 180c000, 0)
%l0-3: 00000000019d73a4 0000000000000001 00000000000000160000000000000000
%l4-7: 0000000000000000 0000000000000002 000000000180c1780000000000000001
000002a10001f9e0 unix:idle+128 (1866c00, 4, 180c000,ffffffffffffffff,1,1865800)
%l0-3: 00000000019d7380 000000000000001b 0000000000000000ffffffffffffffff
%l4-7: 0000000000000001 0000000001a6ec00 000000000180c1780000000001045bac
             
syncing file systems... done
dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content:kernel
0:17 100% done
100% done: 62437 pages dumped, dump succeeded
rebooting...
Resetting...


SPARC M10-1, No Keyboard
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.34.0, 4096 MB memory available, Serial #xxxx1300.
[ 1.24.0 ]
Ethernet address x:xx:xx:xx:96:24, Host ID: xxxxx624.  

to reset (XIR)  control domain from SP

If there is a Hung in the control domain, the first step is to send a break

Step #1

SPARC T3/T4/T5

Usage : send /HOST/send_break_action=break
            start /HOST/console

Example

-> cd /HOST
->send send_break_action=break
->start /HOST/console
            
{0} ok boot
Boot device:             
/pci@300/pci@1/pci@0/pci@4/pci@0/pci@c/scsi@0/disk@w5000cca0162d6525,0:a File and args: 
SunOS Release 5.11 Version 11.1 64-bit
Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.
Hostname: t5-8-sin06-a
            
t5-8-sin06-a console login: Debugging requested; hardware watchdog suspended.
c)ontinue, s)ync, r)eset? 
 

 
If sync is selected, a coredump would be generated. This would be similar to 
"-> set /host/send_break_action=dumpcore".

If step#1 does not unhung a domain, then next step would be to send XIR to the procs.
In this step, XIR command are sent to all the processors . Also the physical domain will be powered off.
            
Step #2 (Note /SYS gets powered off)

SPARC T3/T4/T5

Usage: set SESSION=restricted
           xir sys_reset

Example

-> set SESSION mode=restricted
            
WARNING: The "Restricted Shell" account is provided solely
to allow Services to perform diagnostic tasks.
 
[(restricted_shell) t5-8-sin06-a-sp:~]# xir
usage:    xir             resume|sys_reset|guest_reset|guest_core|guest_debugger
          xir display [-p <physcpu>] [-s <strand>]             [-t] <filename>
          xir display  -t   # display only trap PCs
          xir list          # list XIR files
          xir delete <filename>
[(restricted_shell) t5-8-sin06-a-sp:~]# xir  sys_reset
]xir_send: sys_reset
starting XIR: sys_reset
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
XIR filename: xir_ser.1
[(restricted_shell) t5-8-sin06-a-sp:~]# 
[(restricted_shell) t5-8-sin06-a-sp:~]# exit
exit

Behaviour is the same as the T series machine.If there is a hung
control/primary domain, first step is to consider to issue a break command:

Step#1

SPARC M5/M6/M7/M8

Usage : set  /HOSTn/send_break_action=break
            start /HOSTn/console  

Example

-> cd  /HOST0 
-> set send_break_action=break
Set 'send_break_action' to 'break'
             
-> start /HOST0/console       
Are you sure you want to start /HOST0/console (y/n)? y
          
Serial console started.  To stop, type #.
             
c)ontinue, s)ync, r)eset? r
Resetting...
hcall_mach_sir called

 
If step 1 does not unhung a domain, the next step would be to send XIR to
all procs in the domain.In this step, XIR command are sent to all the processors .
Also the physical domain will be powered off.

Step 2: Mx-32 :  (SP)


Usage: set SESSION=restricted
           xir -d <PDOM#> sys_reset

Example

->set SESSION mode=restricted
[(restricted_shell) m5-32-sca11-a-sp1:~]
[(restricted_shell) m5-32-sca11-a-sp1:~]# xir
usage:    xir -d domain_id resume|sys_reset|guest_reset|guest_core|guest_debugger
          xir display [-p <physcpu>] [-s <strand>][-t] <filename>
          xir display  -t   # display only trap PCs
          xir list          # list XIR files
          xir delete <filename>
[(restricted_shell) m5-32-sca11-a-sp1:~]# xir list
"list" & "delete" not yet supported on this platform
XIR files will be included in Snapshot
[(restricted_shell) m5-32-sca11-a-sp1:~]# xir -d 0 sys_reset <<<---- PDOM0 gets powered off
preparing to send XIR (sys_reset) to domain 0
            
cognizant SP for /SYS/DCU0: /SYS/SPP0
starting XIR: sys_reset
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
get XIR filename: GM busy (5009, 1): sleeping 5
XIR filename: xir_ser.1
[(restricted_shell) m5-32-sca11-a-sp1:~]# exit
exit

SPARC M10

Usage: XSCF>reset -p <ppar_id> -g primary panic

reset command of the XSCF  cause a panic in  the control domain.
             
Example 

XSCF>reset -p 0 -g primary panic

To check the status that the panic has occurred in the   control domain:
             

to reset (XIR) Guest domain from SP   

to reset a hard partition
         

SPARC M5/M6/M7/M8

Usage: set /HOSTn/send_break_action=break
           start /HOSTn/console

SPARC M10

Usage: reset -p <ppar_id> por  | xir
             
Example for PPAR-ID 00            

Check the ppar status             

to check failed cpu or memory components from Control Domain
 

to collect hypervisor dump
        

Hypervisor dump is a snapshot of the hypervisor memory image.
             
- Collected at hypervisor abort or
- when administrator request XIR (external initiated reset )  from XSCF
              
From XSCF

XSCF>reset -p ppar_id xir    

On LDOM control Domain

ldm set-hvdump [hvdump=on|off] [hvdump-reboot=on |  off]     

Check Setting

eis@m10-1-mysyd04-a:~$ ldm list-hvdump
hvdump=on
hvdump-reboot=off
eis@m10-1-mysyd04-a:~$