VSM6: ASR Hangs or fmservice in Maintenance State or CAM Reported 'out of memory' Errors

Asset ID:	1-72-2118011.1
Update Date:	2017-10-09
Keywords:

Solution Type Problem Resolution Sure

Solution 2118011.1 : VSM6: ASR Hangs or fmservice in Maintenance State or CAM Reported 'out of memory' Errors

Applies to:

StorageTek Virtual Storage Manager System 6 (VSM6) - Version All Versions and later
Auto Service Request (ASR) - Version All Versions and later
Sun StorageTek Multi-Drive Virtual Op Panel - Version 2.1 and later
Sun SPARC Sun OS

Symptoms

Eg. After a new code was installed a few weeks ago, ASR_ENABLE/ASR_QUERY/ASRDISABLE was tested and working. This system now hangs on command "sudo /opt/vsm/bin/asr_sfb_cfg.pl ASR_DISABLE"

or if fmservice hangs ("sudo /opt/vsm/bin/asr_sfb_cfg.pl ASR_DISABLE/ASR_ENABLE/ASR_QUERY" does not come back.

Cause

fmservice failed during transfer of sfb to transport.oracle.com due to customer firewall problems

faultmanager service can not be brought back online.

fmservice cannot be restarted because it was still running but faulty

Solution

1. Find out the PID of fmservice

2. kill fmservice

3. restart fmservice

Example shown below:

vsmadm@vsmpriv1:~$ svcs -xv

svc:/system/fmservice:default (Sun Storage Fault Management Service)
State: maintenance since Tue Mar 15 09:41:28 2016
Reason: Start method failed repeatedly, last exited with status 1.
See: http://support.oracle.com/msg/SMF-8000-KS
See: man -M /opt/se6x20/cli/man -s 1M sscs
See: /var/svc/log/system-fmservice:default.log
Impact: This service is not running.

vsmadm@vsmpriv1:~$ ps -ef | grep SUNWsefms | grep java
root 1126 1 0 Feb 13 ? 539:41 /opt/se6000/java_1.6.0/usr/java/bin/java -Xmx256m -DSUNWsefms=/opt/SUNWsefms -D

vsmadm@vsmpriv1:~$ sudo kill -9 1126

vsmadm@vsmpriv1:~$ ps -ef | grep SUNWsefms | grep java

vsmadm@vsmpriv1:~$ sudo /opt/vsm/bin/asr_sfb_cfg.pl ASR_QUERY

An Internal Error occurred. The Fault Management Service may not be responding.
An Internal Error occurred. The Fault Management Service may not be responding.
Device ASR
------------ ---
vsmpriv1: Unable to query device ASR state.
vsmpriv2: ASR is enabled, ASR notifications will be sent to Oracle.
==============================================================
Thu Mar 17 11:39:44 CET 2016
/opt/vsm/bin/asr_sfb_cfg.pl completed SUCCESSFUL
Logging at /var/opt/StatusService/output/asr_sfb_cfg.log
==============================================================
==============================================================
Thu Mar 17 11:39:44 CET 2016
/opt/vsm/bin/asr_sfb_cfg.pl completed SUCCESSFUL
Logging at /var/opt/StatusService/output/asr_sfb_cfg.log
==============================================================

vsmadm@vsmpriv1:~$ /opt/VSM6cam/ss_cam/restart_fmservice
Clearing fmservice

vsmadm@vsmpriv1:~$ sudo /opt/vsm/bin/asr_sfb_cfg.pl ASR_QUERY

vsmpriv1: ASR is enabled, ASR notifications will be sent to Oracle.

vsmpriv2: ASR is enabled, ASR notifications will be sent to Oracle.
==============================================================
Thu Mar 17 11:40:33 CET 2016
/opt/vsm/bin/asr_sfb_cfg.pl completed SUCCESSFUL
Logging at /var/opt/StatusService/output/asr_sfb_cfg.log
==============================================================
==============================================================
Thu Mar 17 11:40:33 CET 2016
/opt/vsm/bin/asr_sfb_cfg.pl completed SUCCESSFUL
Logging at /var/opt/StatusService/output/asr_sfb_cfg.log
==============================================================

References

<NOTE:1285574.1> - Oracle Auto Service Request (ASR) Frequently Asked Questions
<NOTE:1471530.1> - VLE/VSM - Where to find Product Manuals, User Guides and Documentation
<BUG:22930794> - ASR_QUERY/DISABLE/ENABLE DOES NOT COME BACK

Attachments

This solution has no attachment