Pillar Axiom: Login to GUI Fails With Any User Account

Asset ID:	1-72-1629755.1
Update Date:	2014-03-19
Keywords:

Solution Type Problem Resolution Sure

Solution 1629755.1 : Pillar Axiom: Login to GUI Fails With Any User Account

Applies to:

Pillar Axiom 600 Storage System - Version All Versions to All Versions [Release All Releases]
Pillar Axiom 500 Storage System - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

You are unable to log into the GUI with any user account that you have defined.

When attempting to log in, the GUI gives the error message:

"Cannot connect to the system. Try again later or contact Technical Support. "

Attempting to connect to the pilot with a keyboard and monitor may produce no output on one or both pilots.

Cause

MCC Core has a known memory leak that seems to manifest itself particularly with SNMP queries. This memory leak can cause MCC memory consumption to get to the point where other pilot processes are starting to be impacted and malfunction, such as the ability to log into the GUI. Take note of this hog log entry from when the problem was occurring (a TSC engineer will need to find this in the logs for you, after the problem is corrected and you are able to collect logs):

top - 06:45:29 up 394 days, 18:52, 0 users, load average: 0.29, 0.42, 0.48
Tasks: 53 total,   1 running, 52 sleeping,   0 stopped,   0 zombie
Cpu(s): 2.0% us, 1.0% sy, 0.0% ni, 94.1% id, 3.0% wa, 0.0% hi, 0.0% si, 0

PID USER      PR   NI   VIRT   RES   SHR S %CPU %MEM    TIME+ COMMAND
6578 root       6 -10   2154m 1.1g 13m S 3.0   57.5   8752:45 mcc-core

Compare the "VIRT" and "RES" memory usages above to those from when MCC had just restarted (by rebooting the pilot):

top - 14:24:55 up 18 min, 2 users, load average: 1.93, 1.03, 0.57
Tasks: 59 total,   2 running, 57 sleeping,   0 stopped,   0 zombie
Cpu(s): 32.0% us, 13.0% sy, 11.0% ni, 23.0% id, 21.0% wa, 0.0% hi, 0.0% si, 0

PID   USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
6670 root       5 -10 787m 31m 14m S   1.0   1.6   0:36.07 mcc-core

Note in the example above that it took 394 days of MCC uptime for it to catastrophically manifest itself.

There is no known solution for the MCC Core memory leak, the source of it was never found.
Reducing the frequency of SNMP polling would probably reduce the memory leakage and so increase the anticipated time to the next such failure.

Solution

Reboot the pilot(s). Note that if you connect a keyboard and monitor to the pilots, you only need to reboot the one that produced no output on the monitor.

The permanent solution to prevent this from occurring again is to upgrade the Axiom software to release 5.x if possible, because MCC Core does not exist in 5.x software, and so this issue does not exist.

References

<BUG:18302517> - UNABLE TO LOG INTO GUI - LOGINFAILEDINVALIDACCOUNT

Attachments

This solution has no attachment