SuperCluster - Guest IO domain may fail to boot after upgrading to Solaris 11.3 SRU 11.6 or above

Asset ID:	1-72-2235476.1
Update Date:	2017-02-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2235476.1 : SuperCluster - Guest IO domain may fail to boot after upgrading to Solaris 11.3 SRU 11.6 or above

Applies to:

SPARC SuperCluster T4-4 - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster Specific Software - Version 2.x to 2.x [Release 2.0]
Oracle SuperCluster M7 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on SPARC (64-bit)
Sun SPARC Sun OS

Symptoms

Oracle SuperCluster IO domains may fail to boot after upgrading to Solaris 11.3 SRU 11.6 or above (as part of Oracle SuperCluster latest QFSDP) due to BUG 22311453, if the system has already configured with more than 127 mac addresses per PF.

PDom console messages:

WARNING: failed configuring an alternate MAC address for ldoms-net1.vf0 : err28, diag 24
WARNING: failed configuring an alternate MAC address for ldoms-net0.vf0 : err28, diag 24

Also the IO domain using the VF from the PF with more than 127 mac-address created will hang in transition state probing for PCI devices.

root@PDom:~# ldm ls
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-cv- UART 16 31232M 0.7% 0.7% 19m
ssccn2-dom1 active -n---- 5001 64 129536M 0.0% 0.0% 1h 20m
ssccn2-dom2 active -n---- 5002 144 260608M 3.3% 3.3% 1h 20m
ssccn2-dom3 active -n--v- 5003 16 31488M 0.1% 0.1% 1h 20m
ssccn2-io-PDom-exa13 active -t---- 5010 32 64G 3.1% 3.1% 1m

IO Domain console messages:

root@PDom:~# telnet localhost 5010
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
.
Connecting to console "ssccn2-io-PDom-exa13" in group
"ssccn2-io-PDom-exa13" ....
Press ~? for control options ..
NOTICE: Entering OpenBoot.
NOTICE: Fetching Guest MD from HV.
NOTICE: Starting additional cpus.
NOTICE: Initializing LDC services.
NOTICE: Probing PCI devices.

Identify the PFs on the PDom:

root@PDom:~# ldm ls-io

For each PF, identify the max number of mac(alt-mac)-addresses allocated for each VF within the PF:

root@PDom:~# ldm ls-io -l <pf_name>

Example:

root@PDom:~# ldm ls-io -l /SYS/RCSA/PCIE14/IOVNET.PF0

If the allocated mac(alt-mac)-addresses count for a VF within the PF is more then 127, then the guest IO domain which uses the VFs from this PF will fail to boot.

Changes

LDom v3.4 introduced in Solaris 11.3 SRU 11.6 puts a hard limit of 127 mac-address per PF.

Cause

Oracle SuperCluster is already configured with more than 127 mac-addresses using previous Solaris SRU versions (LDoms v3.3) where this limit did not exist, will have an impact on their IO domains not booting due to this bug/restriction introduced in LDoms v3.4.

Solution

If the above symptoms match then, the solution is to reduce the mac(alt-mac)-addresses count using the following commands, so the guest IO domain can boot up.

Identify the VF which has an mac(alt-mac)-address which is not used on the system.

NOTE: On the PDom, use 'ldm ls-io -l | grep mac' to identify the mac addresses available on the system and use 'dladm show-phys -m' and dladm show-vnic -m' to identify if the mac addresses are in use. As the guest IO domain question is down, it is not possible to identify if any of the mac address are used for zones inside the IO domain unless the domain is booted up. So there is a possibility that you may remove the mac address which may be used for the zones within the guest IO domain in question. It is in the best interest to bring up the guest IO domain first so the zone using the deleted mac address can be fixed later.

Depending on the number of additional mac addresses created on the system, reduce the mac(alt-mac)-address count to below 127.

Example: We have a 2 VF0's (one on each PF0/1) with 12 alt-mac addresses assigned, to reduce the count to 10, execute:

root@PDom:~# ldm set-io alt-mac-addrs=auto,auto,auto,auto,auto,auto,auto,auto,auto,auto /SYS/CMIOU5/PCIE2/IOVNET.PF0.VF0
root@PDom:~# ldm set-io alt-mac-addrs=auto,auto,auto,auto,auto,auto,auto,auto,auto,auto /SYS/CMIOU5/PCIE2/IOVNET.PF1.VF0

Once the value is set then stop/start the IO domain in question. Once the domain is booted up, verify the zones and network configuration and fix if there are any issues.

On the PDom, save the SP config to make the changes persistent across reboots.

root@PDom:~# ldm add-spconfig <New-SP-Config-Name-Date-Time-Stamp>

After the new SP config is saved, next time if we open SuperCluster IO domain creation tool which runs a health check the BUI will report an unexpected SP config. This warning can just be cleared in the BUI so the health check then accepts the new SP config.

References

<BUG:25577951> - ORA-27102: OUT OF MEMORY SVR4 ERROR: 22: INVALID ARGUMENT PREVENTING PLAT PATCH
<BUG:25572153> - LDOM CONSOLE IS NOT CONNECTING EVEN AFTER REBOOTING THE PRIMARY DOMAIN PDOM-4.

Attachments

This solution has no attachment