Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-71-1991360.1
Update Date:2016-03-23
Keywords:
Solution Type  Technical Instruction Sure

Solution  1991360.1 :   SuperCluster - Avoiding system/pools maintenance state when modifying zone pool CPU count after using setcoremem  

Related Items 
    

  • Oracle SuperCluster T5-8 Full Rack
    •  
  • SPARC SuperCluster T4-4 Full Rack
    •  
  • Oracle SuperCluster T5-8 Half Rack
    •  
  • SPARC SuperCluster T4-4
    •  
  • SPARC SuperCluster T4-4 Half Rack
    •  
  • Oracle SuperCluster T5-8 Hardware
    •  
  • Oracle SuperCluster M6-32 Hardware
    •  
Related Categories 
    

  • PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
    •  




CPU-resource pools are used on SuperCluster to tune the number of CPU-cores visible in Solaris Zones. When reducing the number of CPU-cores available in a  Domain that contains Zones with CPU-resource pools, care must be taken to resize the pools so that the Domain can reboot without the pools service going into maintenance mode.  While the basics of this procedure can also be used when a Domain is adding CPU-cores so zones get more resources, that can be done after reboot without issue.  If the pools service goes into maintenance due to miscalculation or incorrect steps, a workaround is provided.

In this Document
Goal
Solution
  Main Steps:
 Instructions:

Applies to:

Oracle SuperCluster M6-32 Hardware - Version All Versions and later
SPARC SuperCluster T4-4 - Version All Versions and later
Oracle SuperCluster T5-8 Hardware - Version All Versions and later
Oracle SuperCluster T5-8 Full Rack - Version All Versions and later
Oracle SuperCluster T5-8 Half Rack - Version All Versions and later
Oracle Solaris on SPARC (64-bit)

Goal

When the number of CPU-cores available in a SuperCluster Domain is reduced, care must be taken if this Domain is hosting Solaris Zones with CPU-resource pools. The size of the CPU-resource pools used by the Zones must be adjusted accordingly or the pools service (a.k.a. SMF system/pools service) can end up in maintenance state. This document provides instructions on how perform such a modification flawlessly.

On SuperCluster the number of CPU-cores available in a Domain is adjusted with the setcoremem tool.

  

Solution

 Main Steps:

 1. Determine current and future pool CPU counts and sizes. The global-zone requires a minimum of 2 cores for 1 Host Channel Adapter (HCA), or 4 cores for 2 HCA. On SuperCluster one core includes 8 CPU.
 2. Reduce the pools as necessary to meet new constraints using poolcfg and saving to /etc/pooladm.conf .
 3. Re-read pool configuration from pooladm.conf into dynamic state using "pooladm -c"
 4. Re-write dynamic state so that pooladm.conf no longer contains previous dynamic state information using "pooladm -s"
 5. Verify both dynamic state and configuration file contain desired content.
 6. In case the above fails and system/pools comes up in maintenance mode, use one of 2 possible workarounds.

 

Instructions:

This is a simple example removing 8 CPUs from one pool. Most real world cases will have many pools affected.

1. Look at existing pools and sizes:

root@domain_shrinkage:~# poolstat
                              pset
 id pool                 size used load
 39 pool_shrk_z01_...     8 0.00 0.00
 38 pool_shrk_z02_...    40 0.00 0.81
  0 pool_default         16 0.00 0.35

root@domain_shrinkage:~# grep 'type="pset' /etc/pooladm.conf
  <res_comp type="pset" sys_id="1" name="pset_shrk_z01_18384" default="false" min="40" max="40" units="population" comment="" ref_id="pset_1">
  <res_comp type="pset" sys_id="2" name="pset_shrk_z02_183148" default="false" min="8" max="8" units="population" comment="" ref_id="pset_2">
  <res_comp type="pset" sys_id="-1" name="pset_default" default="true" min="1" max="65536" units="population" comment="" ref_id="pset_-1">

 
2. Shrink pools as necessary to allow for above constraints on global-zone (pool_default) CPU resources after setcoremem resizing:

root@domain_shrinkage:~# poolcfg -c 'modify pset pset_shrk_z02_18384 ( uint pset.min=32;uint pset.max=32 )' /etc/pooladm.conf

  

If you use poolcfg -dc to directly modify kernel state instead of as above specifying writing direclty to the /etc/pooladm.conf file, you will need to run "pooladm -s;pooladm -c;pooladm -s" in place of two pooladm commands below :

 

3. This is the critical step; until Bug ID 20480175 is addressed, the extra "pooladm -s" command is needed to make sure no pset.size remains too large which could cause the system/pools SMF service to come up in maintenance state after setcoremem resize.  (The facility does handle dynamic resizing of pset_default pset.size upon reboot.) :

root@domain_shrinkage:~# pooladm -c


4. This is the critical step to correct any dynamic size data in pooladm.conf so that the pools service comes up after resize. Rewrite /etc/pooladm.conf: 

root@domain_shrinkage:~# pooladm -s

 
5. Verify that current in-memory state is as expected:

root@domain_shrinkage:~# poolstat
                              pset
 id pool                 size used load
 39 pool_shrk_z01_...     8 0.00 0.00
 38 pool_shrk_z02_...    32 0.00 0.86
  0 pool_default         24 0.00 0.35

  
If any pset.size > that pool's max value, something is wrong so rerun above pooladm -c; pooladm -s commands and try again before shutting down:

root@domain_shrinkage:~# grep 'max' /etc/pooladm.conf | awk '{print $4 " " $7}'; grep pset.size /etc/pooladm.conf
name="pset_shrk_z02_18384" max="32"
name="pset_shrk_z01_18358" max="8"
name="pset_default" max="65536"
    <property name="pset.size" type="uint">32</property>
    <property name="pset.size" type="uint">8</property>
    <property name="pset.size" type="uint">24</property>

 
If all pset.size values are <= the associated pset max value as in this example, and you have enough CPUs available to fulfill you should be able to shutdown your domain, run setcoremem to reduce CPU count, and the domain should boot up with system/pools SMF service online and new CPU counts as specified:

root@domain_shrinkage:~# svcs pools
STATE          STIME    FMRI
online         Feb_04   svc:/system/pools:default


6. If a miscalculation occurs or a misstep occurs and the service comes up in maintenance mode:

Disable the pools service, make a copy elsewhere then delete /etc/pooladm.conf, then enable the pools service. rewrite a blank /etc/pooladm.conf, and regenerate all pools and psets.

root@domain_shrinkage:~# svcadm disable system/pools
root@domain_shrinkage:~# mv /etc/pooladm.conf /root/pooladm.conf.saved
root@domain_shrinkage:~# svcadm enable system/pools

Create a default pooladm.conf:

root@domain_shrinkage:~# pooladm -s /etc/pooladm.conf

Recreate all pools and psets via poolcfg (you can obtain needed information from poolstat before reboot or from pooladm.conf.saved above):

root@domain_shrinkage:~# poolcfg -c 'create pool pool_shrk_z01_18384' /etc/pooladm.conf
root@domain_shrinkage:~# poolcfg -c 'create pool pool_shrk_z02_18384' /etc/pooladm.conf

root@domain_shrinkage:~# poolcfg -c 'create pset pset_shrk_z01_18384 ( uint pset.min=8;uint pset.max=8 )' /etc/pooladm.conf
 root@domain_shrinkage:~# poolcfg -c 'create pset pset_shrk_z02_18384 ( uint pset.min=32;uint pset.max=32 )' /etc/pooladm.conf


Associate psets with pools:

root@domain_shrinkage:~# poolcfg -c 'associate pool pool_shrk_z01_18384 (pset pset_shrk_z01_18384)' /etc/pooladm.conf
root@domain_shrinkage:~# poolcfg -c 'associate pool pool_shrk_z02_18384 (pset pset_shrk_z02_18384)' /etc/pooladm.conf

  
 Read modified pooladm.conf file back into dynamic state, flush dynamic data in conf file, verify everything is as needed:

root@domain_shrinkage:~# pooladm -c
root@domain_shrinkage:~# pooladm -s
root@domain_shrinkage:~# poolstat



Attachments 
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback