Description

Sun SPARC Enterprise M8000/M9000 Servers with XCP firmware revision 1092 or 1093 will fail to start after an XSCF reboot (applynetwork/rebootxscf) if a gateway is misconfigured, which indicates the gateway IP address is not on the same subnet as the IP address for the LAN interface.

During the XSCF reboot sequence, the XSCF (eXtended System Control Facility) database will become corrupted as a consequence of the misconfigured gateway. There will be no immediate impact to the running domains, but if no XSCF is available, the domains cannot reboot.

The XSCF with the corrupted database is the one with the misconfigured gateway(s). It is possible that the database on either one of the two or both XSCFs are corrupted.

If both XSCFs have incorrect routes leading to XSCF database corruption, no XSCF will be available and a platform power cycle will be required to recover.

Occurrence

This issue can occur on the following platforms:

• Sun SPARC Enterprise M8000/M9000 Servers with XCP 1092 or 1093 firmware

To determine the XCP firmware version on one of these systems, do the following:

XSCF> version -c xcp

XCP 1093 output will appear similar to the following:

XSCF#0 (Active )
XCP0 (Reserve): 1093
XCP1 (Current): 1093
XSCF#1 (Standby)
XCP0 (Reserve): 1093
XCP1 (Current): 1093

Symptoms

Should the described issue occur, erroneous routes configured on XCP 1091 and lower will produce errors similar to the following on the XSCF console during the XSCF reset sequence:

[output omitted]
execute S11network.sheth0: PHY is Intel LXT972A (1378e2)
ERROR: failed to configure the routeing.
The network system may not work correctly.
[output omitted]

Workaround

Prior to installing the affected firmware revisions 1092 or 1093, it is important that the gateway configuration is verified to be sure to avoid this issue at the next reboot.

Once firmware revision 1092 or 1093 is installed, it is critical that any configuration changes (setroute/setnetwork) are carefully checked to ensure that this issue is not invoked at the next reboot.

It is possible to manually verify the network configuration which will avoid the XSCF database corruption by doing the following:

1. Log into the Active XSCF as a user with 'platadm' privilege

2. Change one of the network settings and use 'applynetwork -n' to see routes defined in the XSCF database via the following command:

XSCF> setnetwork -c down xscf#0-lan#0; setnetwork -c up xscf#0-lan#0; applynetwork -n

verify there are no routes with a gateway not located on the local network (i.e., same subnet) as in the following example:

XSCF> setnetwork -c down xscf#0-lan#0; setnetwork -c up xscf#0-lan#0; applynetwork -n

The following network settings will be applied:

xscf#0 hostname  :m8000-xscf0
xscf#1 hostname  :m8000-xscf1
DNS domain name  :domain.com
nameserver       :10.244.240.150

interface        :xscf#0-lan#0
status           :up
IP address       :10.244.128.27                                <===lan interface IP address (subnet)
netmask          :255.255.255.0                                <===netmask defines subnet range
!!  route        :-n 0.0.0.0 -m 0.0.0.0 -g 10.244.128.1
!!  route        :-n 0.0.0.0 -m 0.0.0.0 -g 128.244.128.1       <===bad gateway is here

In the example above, gateway 128.244.128.1 is not reachable on the xscf#0-lan#0 10.244.128.xxx subnet.

3. Delete the bad routes:

XSCF> setroute -c del -n 0.0.0.0 -m 0.0.0.0 -g 128.244.128.1 xscf#0-lan#0

TIP: copy/paste the route from applynetwork output when constructing 'setroute -c del'

4. Apply the changes with "applynetwork -y", and verify that all routes are reachable on the interface subnet:

XSCF> applynetwork -y

5. Re-verify all routes are reachable on the interface subnet (i.e., gateway is same subnet):

XSCF> rebootxscf -y

Once the configuration is verified correct, you may safely upgrade to XCP 1092 or 1093 or safely proceed with XSCF reset.


This issue is addressed in the following release:

• XCP 1100 firmware (or later) for Sun SPARC Enterprise M8000/M9000 Servers

and can be downloaded from the following site:

http://www.oracle.com/technetwork/systems/patches/firmware/release-history-jsp-138416.html

History

29-Sep-2010: Workaround Release
01-Dec-2010: Republish - Issue is now Resolved
28-Nov-2011: No content change; maintenance update
08-May-2013: Checked for currency/relevance; no change in content


It's possible to verify the network configuration by collecting an XSCF
snapshot and then unpacking this snapshot on the OPL Snapshot analysis
Toolset at https://cores2-web.oraclecorp.com/opltools/ The tool has been augmented with
automation to verify the network configuration. The tool will provide
the command(s) to fix the problem when applicable. Note that the snapshot
must be properly unpacked in order to check the configuration.

Once your configuration is verified correct you may safely upgrade to
XCP 1092 or 1093 or safely proceed with XSCF reset.

When the XSCF is unable to start, the following message can be observed
in the XSCF console logs:

execute S60checktestdb[317]: ERR: Database problems detected: (ret=-9010).
Cleaning up
scdb_init_all: -9010, Database verify bad
root: ERROR: Database problems detected: Inconsistency.  Cleaning up
Initiating shutdown
...
XSCF BOOT STOP (recover by NFB-OFF/ON)

Eng Support: There is a special procedure which requires a complete
platform power cycle to remove the condition that triggers this issue.
Replacing XSCF hardware, or any other hardware for that matter,
is not going to solve this issue.

Please send technical questions to the following email:
sunalertpublication_us_grp@oracle.com
and copy the Responsible Engineer listed below

Internal Contributor/Submitter: stephane.dutilleul@oracle.com
Internal Eng Responsible Engineer: alex.aftandilian@oracle.com
Internal Services Knowledge Engineer:david.mariotto@oracle.dom
Internal Eng Business Unit Group: Systems Group - OPL
Internal Escalation ID: 73220322, 73374292, 73361718, 73477978, 73501428,
73531790, 73522408

References

This solution has no attachment