Sun Storage 7000 Unified Storage System: SP console access always gives "in use" message even after 'stop/(re)start'

Asset ID:	1-72-1495018.1
Update Date:	2013-07-24
Keywords:

Solution Type Problem Resolution Sure

Solution 1495018.1 : Sun Storage 7000 Unified Storage System: SP console access always gives "in use" message even after 'stop/(re)start'

Applies to:

Sun ZFS Storage 7420 - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7320 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7410 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7310 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

After logging in to the Service Processor (SP) and attempting to start a console session the message that the console is in use is returned.

Attempting to stop and restart the console session does not free it up:

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y
Serial console is in use.

-> stop /SP/console
Are you sure you want to stop /SP/console (y/n)? y

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y
Serial console is in use.

Changes

A Console Management program has been installed on one of the systems used to access the Service Processor.

Cause

A Console Management program, such as ConMan, has been installed on one of the systems used to access the Service Processor.

The Console Management program is re-establishing the console connection after each time that the console session is stopped.

This happens so quickly that any other sessions cannot take hold of the seriallink process to take ownership of the console session.

Solution

Either only use the Console Management program to access a console session from the host that it is installed on, or uninstall the Console Management software entirely.

If you are not sure whether you have a host with a Console Management program installed but are seeing the symptoms described then please open a call with Oracle Support

who will be able to use a procedure to identify the IP address of any system that is continuously grabbing the lock on the serial link process, provided a remote support session can be provided via Shared Shell or Webex.

Use the following procedure to identify which host is reestablishing the console:
1) Log in as sunservice (passwd same as root) and check for the serial link lock file /var/lock/LCK..ttyS0, note the timestamp on this file if it exists.
2) Check "ps -ef" output for sshd processes to check if other sessions are logged in at the same time.
3) Check "netstat -an | grep :22" to find out where other sshd sessions have come from.
4) kill any unknown sshd sessions (not the one you logged in on), and repeat steps 1-3 again.
5) Identify IP address that is reestablishing the connection from the "netstat -an | grep 22" command.

Whilst logged in a sunservice open another session, or have the customer do so, log in as root and attempt to start the console to check problem is still happening:

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y
Serial console is in use.

-> stop /SP/console
Are you sure you want to stop /SP/console (y/n)? y

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y
Serial console is in use.

Back in the sunservice session check for existence and timestamp on lock file, along with current date/time:

[(flash)root@s7000-sp:/]# ls -l /var/lock/LCK..ttyS0
-rwSr-x--t 1 root root 0 Sep 26 16:19 /var/lock/LCK..ttyS0
[(flash)root@s7000-sp:/]# date
Wed Sep 26 16:21:02 GMT 2012

Check for sshd sessions:

[(flash)root@s7000-sp:/]# ps -ef | grep sshd
373 root 752 S /usr/sbin/sshd -q
31038 root 1820 S sshd: root@pts/0
21550 root 1832 S sshd: root@pts/1
21843 root 1832 S sshd: sunservice@pts/2
24062 root 492 S grep sshd

Check where sessions are coming from:

[(flash)root@s7000-sp:/]# netstat -an | grep :22
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 192.168.0.2:22    192.168.0.150:17007 ESTABLISHED
tcp 0 0 192.168.0.2:22    192.168.0.100:38923 ESTABLISHED
tcp 0 0 192.168.0.2:22    192.168.0.100:39265 ESTABLISHED

Kill unwanted sessions, ensure you know the IP address you are connected in from - so that you do not kill that session:

[(flash)root@s7000-sp:/]# kill 31038

[(flash)root@s7000-sp:/]# ps -ef | grep sshd
373 root 752 S /usr/sbin/sshd -q
21843 root 1832 S sshd: sunservice@pts/2
28229 root 1820 S sshd: root@pts/0
28556 root 492 S grep sshd

[(flash)root@s7000-sp:/]# kill 28229

[(flash)root@s7000-sp:/]# ps -ef | grep sshd
373 root 752 S /usr/sbin/sshd -q
21843 root 1832 S sshd: sunservice@pts/2
29186 root 492 S grep sshd

[(flash)root@s7000-sp:/]# ps -ef | grep sshd
373 root 752 S /usr/sbin/sshd -q
21843 root 1832 S sshd: sunservice@pts/2
29867 root 1820 S sshd: root@pts/0 <----- "this one came back"
30862 root 492 S grep sshd

Identify where the session is coming from that is coming back:

[(flash)root@s7000-sp:/]# netstat -an | grep :22
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 192.168.0.2:22    192.168.0.150:25953 ESTABLISHED <----"This is the client that is continususly re-establishing the console connection"
tcp 0 0 192.168.0.2:22    192.168.0.100:39265 ESTABLISHED
tcp 0 0 192.168.0.2:22    192.168.0.100:44409 ESTABLISHED

Attachments

This solution has no attachment