Sun Fire[TM] 12K/15K/20K/25K: System controller : SMS fomd process does not start up.

Asset ID:	1-72-1963443.1
Update Date:	2017-12-06
Keywords:

Solution Type Problem Resolution Sure

Solution 1963443.1 : Sun Fire[TM] 12K/15K/20K/25K: System controller : SMS fomd process does not start up.

Applies to:

Sun Fire E20K Server - Version All Versions to All Versions [Release All Releases]
Sun Fire E25K Server - Version All Versions to All Versions [Release All Releases]
Sun Fire 15K Server - Version All Versions to All Versions [Release All Releases]
Sun Fire 12K Server - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

The system controller rebooted. After the reboot the fomd process would not start up

platform messages from the other system controller

FailoverMgr.cc 2297] Taking over main role because remote SC is unresponsive or down
Dec 15 01:25:57 2014 sc1 fomd[3904]: [8519 21817596565452167 NOTICE FailoverMgr.cc 2631] Failover deactivated
Dec 15 01:26:02 2014 sc1 fomd[3904]: [8570 21817602299206634 NOTICE FailoverMgr.cc 2356] Reset the remote SC <<<<<<

platform messages from the rebooted SC file shows the reset/reboot

Dec 15 01:31:36 sc0 genunix: [ID 540533 kern.notice] #015SunOS Release 5.10 Version Generic_150400-02 64-bit
Dec 15 01:31:36 sc0 genunix: [ID 700403 kern.notice] Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.
Dec 15 01:31:36 sc0 genunix: [ID 678236 kern.info] Ethernet address = 0:14:4f:44:c4:f0
Dec 15 01:31:36 sc0 unix: [ID 673563 kern.info] NOTICE: Kernel Cage is ENABLED

platform messages from the rebooted SC shows fomd will not start up

Dec 15 01:24:30 2014 sc0 ssd[3979]: [1312 21818904386740456 ERR StartupManager.cc 3025] software component failed: name=fomd
Dec 15 01:24:30 2014 sc0 ssd[3979]: [1304 21818904462871158 NOTICE StartupManager.cc 2744] software component start-up initiated: name=fomd
Dec 15 01:24:32 2014 sc0 hwad[3992]: [500 21818906560556386 WARNING DoorClient.cc 563] door_call failed: door=/var/opt/SUNWSMS/SMS1.6/doors/fomd, retries=1, ecode=9
Dec 15 01:24:32 2014 sc0 hwad[3992]: [50042 21818906707201993 ERR InterruptHandler.cc 293] send event failed: event=1, text=null, ecode=9
Dec 15 01:24:34 2014 sc0 ssd[3979]: [1311 21818908562783461 WARNING StartupManager.cc 2852] software component failed to respond: name=fomd
Dec 15 01:24:34 2014 sc0 ssd[3979]: [1306 21818908565179199 WARNING StartupManager.cc 2854] software component start-up failed: name=fomd
Dec 15 01:24:34 2014 sc0 ssd[3979]: [1308 21818908565975438 WARNING StartupManager.cc 2855] software component hard shutdown initiated: name=fomd, signal=SIGKILL
Dec 15 01:24:34 2014 sc0 ssd[3979]: [1304 21818908605280867 NOTICE StartupManager.cc 2744] software component start-up initiated: name=fomd
Dec 15 01:24:36 2014 sc0 esmd[8679]: [500 21818910771696101 WARNING DoorClient.cc 563] door_call failed: door=/var/opt/SUNWSMS/SMS1.6/doors/fomd, retries=5, ecode=9
Dec 15 01:24:42 2014 sc0 ssd[3979]: [1311 21818916950562977 WARNING StartupManager.cc 2852] software component failed to respond: name=fomd
Dec 15 01:24:42 2014 sc0 ssd[3979]: [1306 21818916951656039 WARNING StartupManager.cc 2854] software component start-up failed: name=fomd
Dec 15 01:24:42 2014 sc0 ssd[3979]: [1308 21818916952173725 WARNING StartupManager.cc 2855] software component hard shutdown initiated: name=fomd, signal=SIGKILL
Dec 15 01:24:42 2014 sc0 ssd[3979]: [1304 21818916973287607 NOTICE StartupManager.cc 2744] software component start-up initiated: name=fomd

Changes

Cause

fomd appears to have a built in 255 bytes limitation on the /etc/group entry size when reading in the group.

See 1010533.1 - SunFire[TM] 12K/15K/E20K/E25K servers: System Management Software (SMS) Daemons Overview for more informatio about FOMD

If we do a wc -c on the customer dmngxxxx group we hit 254 bytes, so with a bit of padding it enough to push fomd over the edge.

Note that dmng is just an example - it could be any 250+byte (approx) entry in the /etc/group file.

As development to SMS has long since stopped their only option will be to manage the size of the group and make sure it doesn't go above this size limit.

In this instance for this customer the problem entry in the /etc/group file was (editted for readability )

dmngxxxx::96:sms-svc,aa8xxxx,bg4xxxx,dp8xxxx,kj7xxxx,lm3xxxx,ma7xxxx,ms0xxxx,nd4xxxx,nv6xxxx,pp9xxxx,pv4xxxx,se0xxxx,sj5xxxx,sk2xxxx,sk9xxxx,sl3xxxx,sn1xxxx,sp3xxxx,ss5xxxx,ybxxxx,yl5xxxx,sg5xxxx,exp16999
$ grep dmngxxxx group | wc -c
254

Explorer adds a user (eg exp16999) to the end of the entry. A short term fix is the removal of the explorer user (exp16999) from the entry in question, then fomd can start ok (but please see the Solution below)

Solution

1. Check /etc/group for an entry which has > 250 bytes and if found, ask the customer to amend it so that the entry is less than 250 bytes.

2. stop / start SMS.

Note that often it can be installing / running explorer, which adds a user ( eg exp16999 ) to an entry, which causes this problem (where it was not happening previously) so this issue can occur as a result of an unrelated problem (where an explorer from the system controller has been asked for).

A short term fix is the removal of the explorer user (in this case ",exp16999" ) from the 250+ byte entry in the /etc/group file and then to stop/start sms.
Fomd should then start up correctly.

However the solution is for the customer to check the users against the 250+ entry and remove any old names that are not needed and then stop/start SMS.
This will allow explorer to be run in the future without causing this problem again.

References

<NOTE:1010533.1> - SunFire[TM] 12K/15K/E20K/E25K servers: System Management Software (SMS) Daemons Overview
<NOTE:1002075.1> - Sun Fire[TM] 12K/15K/E20K/E25K: System Management Services(SMS) software: Upgrade Requirements

Attachments

This solution has no attachment