CELL-01519: Cannot talk to the Management Server (MS). Error ; Unable to start MS

Asset ID:	1-72-1517395.1
Update Date:	2013-01-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1517395.1 : CELL-01519: Cannot talk to the Management Server (MS). Error ; Unable to start MS

Applies to:

Exadata Database Machine X2-2 Full Rack - Version All Versions to All Versions [Release All Releases]
Oracle Exadata Storage Server Software - Version 11.2.3.1.0 to 11.2.3.2.0 [Release 11.2]
Information in this document applies to any platform.

Symptoms

CellCLI> list griddisk

CELL-01519: Cannot talk to the Management Server (MS). Error: ; nested exception is:
HTTP transport error: javax.xml.soap.SOAPException: java.security.PrivilegedActionException: javax.xml.soap.SOAPException: Bad response: 404 Not Found

CellCLI> alter cell shutdown services all

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of ALL services was not successful. Error: Stop Failed

CellCLI> alter cell restart services all

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of ALL services was not successful. Error: Stop Failed
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...
running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
CELL-01513: Remote host timed out during read operation.
Thread 6 (Thread 0x411cc940 (LWP 9073)):
#0 0x00002ae83e817b3b in accept () from /lib64/libpthread.so.0
#1 0x00002ae83e075b86 in slputcpaccept () from /opt/oracle/cell/cellsrv/lib/libclntsh.so.11.1
#2 0x00000000004208ad in ossrsutl_listener ()
#3 0x0000000000407ea3 in ossrssrm_listener_thd ()
#4 0x00002ae83e81073d in start_thread () from /lib64/libpthread.so.0
#5 0x000000377c6d44bd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x41bcd940 (LWP 9074)):
#0 0x00002ae83e81868f in waitpid () from /lib64/libpthread.so.0
#1 0x000000000040b98a in sossrs_waitpid ()
#2 0x0000000000415711 in ossrsutl_monitor_monpr_thd ()
#3 0x00002ae83e81073d in start_thread () from /lib64/libpthread.so.0
#4 0x000000377c6d44bd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x425ce940 (LWP 9075)):
#0 0x00002ae83e81868f in waitpid () from /lib64/libpthread.so.0
#1 0x000000000040b98a in sossrs_waitpid ()
#2 0x0000000000415711 in ossrsutl_monitor_monpr_thd ()
#3 0x00002

Cause

ms-odl.trc shows No space left on device

2013-01-03T14:00:00.709+00:00] [ossmgmt] [ERROR] [] [ms.core.MSMetricPersistTimerTask] [tid: 15] [ecid: 10.65.222.86:47339:1329939028817:5,0] java.io.IOException: No space left on device[[
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at java.util.zip.GZIPOutputStream.writeHeader(GZIPOutputStream.java:123)
at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:48)
at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:58)
at oracle.ossmgmt.common.util.xml.XMLUtility.printToFile(XMLUtility.java:898)
at oracle.ossmgmt.ms.core.MSMetricPersistTimerTask.run(MSMetricPersistTimerTask.java:143)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

]]

Check df -h output from cell

/dev/md7 2.0G 2.0G 0 100% /opt/oracle

Solution

Cleanup old unwanted large files from /opt/oracle

Inactivate griddisks

cellcli -e 'ALTER GRIDDISK ALL INACTIVE'

Restart Cell Services

cellcli -e 'ALTER CELL RESTART SERVICES ALL'

Activate griddisks

cellcli -e 'ALTER GRIDDISK ALL ACTIVE'

References

<BUG:13709845> - CELLCLI FAILED TO COMMUNICATE WITH MS AFTER /SYS/VPS ALERTS

Attachments

This solution has no attachment