FS System: SNMP Diagnosis and Usage

Asset ID:	1-75-2360204.1
Update Date:	2018-03-05
Keywords:

Solution Type Troubleshooting Sure

Solution 2360204.1 : FS System: SNMP Diagnosis and Usage

Applies to:

Oracle FS1-2 Flash Storage System - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Purpose

SNMP operation on the FS1-2

Troubleshooting Steps

Background Information:

SNMP is a standard network monitoring and device management protocol. Many organizations use a SNMP management system such as Nagios to monitor devices in their environment. Many well known devices will have information that is exposed via SNMP. Many monitoring services systems will have a database of these devices and can monitor these systems out of the box. Systems that are less common will include a text file called a MIB or Management Information Base. Some management systems can import MIBs and others can't. If the system can't import a MIB you can usually add a customer monitor using locations defined in the MIB files. These locations are called OIDs, or Object Identifier.

There is an industry standard structure to MIBs. This document explains in detail how a MIB is structured. OIDs are defined by a location systems using numbers and periods. Each number represents a tree location in a monitored device's SNMP service. One has to extract this value from the MIB and present it to the management server so it can query the information you intend to monitor on the end device.

There are translation tools on Windows and Linux that make the OID mapping process less daunting. On Linux, the net-snmp-utils package has two command, snmptranslate and snmpwalk On Windows there are separate stand alone tools such as Syslog Watcher's snmpwalk and Paessler's MIB Importer

Working with MIBs in Linux:

Install net-snmp-utils via your platforms package manager. The command below is for Oracle Linux 7 running as root

yum install net-snmp-utils

Copy your device's MIB file to the directory /usr/share/snmp/mib/
Open the MIB file in a text editor. In this document the Oracle FS1-2 MIB will be used as an example.
Page through the MIB and look for items the manufacture has exposed. Below are a few examples from the Oracle FS1-2:

FS1-2 MIB Section: cStorageDetailsInformation

cStorageDetailsInformationSystemCapacitySummaryPhysicalTotalCapacity
cStorageDetailsInformationSystemCapacitySummaryPhysicalUsedCapacity
cStorageDetailsInformationSystemCapacitySummaryPhysicalFreeCapacity
systemHealthStatus
sControllerInformationOverallControllerStatus
sControllerInformationRestartCounter
sEnclosureInformationTable
sEnclosureInformationOverallEnclosureStatus
sPilotInformationTable
sPilotInformationControlUnitHardwareComponentStatus

Management tools use the DOT address of the MIB stanza you want to set/get. The first step in the process is to translate the items found in the MIB to an OID. Each translation below starts with the command:

snmptranslate -m +ORACLE-FS-MIB -IR -On

followed by the section from the MIB

Retrieving OID used in this document

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On cStorageDetailsInformationSystemCapacitySummaryPhysicalTotalCapacity
.1.3.6.1.4.1.15548.3.1.2.3.1.1.2
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On cStorageDetailsInformationSystemCapacitySummaryPhysicalUsedCapacity
.1.3.6.1.4.1.15548.3.1.2.3.1.1.3
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On cStorageDetailsInformationSystemCapacitySummaryPhysicalFreeCapacity
.1.3.6.1.4.1.15548.3.1.2.3.1.1.4
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On sControllerInformationOverallControllerStatus
.1.3.6.1.4.1.15548.3.2.1.1.1.13
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On sControllerInformationRestartCounter
.1.3.6.1.4.1.15548.3.2.1.1.1.18
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On sEnclosureInformationOverallEnclosureStatus
.1.3.6.1.4.1.15548.3.2.2.1.1.8
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmptranslate -m +ORACLE-FS-MIB -IR -On sPilotInformationControlUnitHardwareComponentStatus
.1.3.6.1.4.1.15548.3.2.3.2.1.9
[adminobi@nagios1 ~]$

Using the command snmpwalk with the OIDs against a system configured for SNMP will return a value or state. Each value retrieval starts with the command:

snmpwalk -v 2c -c <community> <host FQDN or IP address> <OID>

Information from a live FS1-2

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.1.2.3.1.1.2
SNMPv2-SMI::enterprises.15548.3.1.2.3.1.1.2.1 = INTEGER: 475943
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.1.2.3.1.1.3
SNMPv2-SMI::enterprises.15548.3.1.2.3.1.1.3.1 = INTEGER: 3894
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.1.2.3.1.1.4
SNMPv2-SMI::enterprises.15548.3.1.2.3.1.1.4.1 = INTEGER: 387725
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.2.1.1.1.13
SNMPv2-SMI::enterprises.15548.3.2.1.1.1.13.1 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.1.1.1.13.2 = STRING: "NORMAL"
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.2.1.1.1.18
SNMPv2-SMI::enterprises.15548.3.2.1.1.1.18.1 = STRING: "0"
SNMPv2-SMI::enterprises.15548.3.2.1.1.1.18.2 = STRING: "0"
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.2.2.1.1.8
SNMPv2-SMI::enterprises.15548.3.2.2.1.1.8.1 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.2.1.1.8.2 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.2.1.1.8.3 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.2.1.1.8.4 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.2.1.1.8.5 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.2.1.1.8.6 = STRING: "NORMAL"
[adminobi@nagios1 ~]$

[adminobi@nagios1 ~]$ snmpwalk -v 2c -c public fs1-sca11-a.us.oracle.com .1.3.6.1.4.1.15548.3.2.3.2.1.9
SNMPv2-SMI::enterprises.15548.3.2.3.2.1.9.1.1 = STRING: "NORMAL"
SNMPv2-SMI::enterprises.15548.3.2.3.2.1.9.1.2 = STRING: "NORMAL"
[adminobi@nagios1 ~]$

Below is an example setup of the Open-Source Monitoring tool Nagios to monitor the FS1-2

NOTE: If you have issues with Nagios see the online documentation for the product. Always remember google is your friend..

Setup Nagios using MIB OIDs:

Defining a host:

create a nagios host definition file for FS1-2's. You can add many hosts that use the same monitors. In this example only one array is monitored.

touch /etc/nagios/conf.d/fs1-sca11-a.cfg

The first section of the file is for host definitions.
The middle sections defines the services that will be monitored plus the command and any variables that need to be passed.
The last section defines the OID query command or script pointer plus passed variables.
Finally a bash script was created to return one of two values to the Nagios Server. A (0) zero means things are good, a (2) means there is an issue. Nagios will also display the value from Standard Out if presented. This script prints out the returns from the fetch and shows up in the Nagios Status Information Column.

#fs1-sca11-a.cfg

define host {
host_name fs1-sca11-a
alias fs1-lab
address fs1-sca11-a.us.oracle.com
max_check_attempts 3
check_period 24x7
check_command check-host-alive
# contacts root
notification_interval 60
notification_period 24x7
}

define service{

use generic-service
host_name fs1-sca11-a
service_description array percent physical disk used
check_command physical_used_capacity!used!public!$HOSTADDRESS$

}
define service{

use generic-service
host_name fs1-sca11-a
service_description controller 1 status
check_command controller_status!controller!public!$HOSTADDRESS$!1

}
define service{

use generic-service
host_name fs1-sca11-a
service_description controller 2 status
check_command controller_status!controller!public!$HOSTADDRESS$!2

}
define service{

use generic-service
host_name fs1-sca11-a
service_description pilot 1 status
check_command pilot_status!pilot!public!$HOSTADDRESS$!1

}
define service{

use generic-service
host_name fs1-sca11-a
service_description pilot 2 status
check_command pilot_status!pilot!public!$HOSTADDRESS$!2

}
define service{

use generic-service
host_name fs1-sca11-a
service_description enclosure 1 status
check_command enclosure_status!enclosure!public!$HOSTADDRESS$!1

}
define service{

use generic-service
host_name fs1-sca11-a
service_description enclosure 2 status
check_command enclosure_status!enclosure!public!$HOSTADDRESS$!2

}
define service{

use generic-service
host_name fs1-sca11-a
service_description enclosure 3 status
check_command enclosure_status!enclosure!public!$HOSTADDRESS$!3

}
define service{

use generic-service
host_name fs1-sca11-a
service_description enclosure 4 status
check_command enclosure_status!enclosure!public!$HOSTADDRESS$!4

}
define service{

use generic-service
host_name fs1-sca11-a
service_description enclosure 5 status
check_command enclosure_status!enclosure!public!$HOSTADDRESS$!5

}
define service{

use generic-service
host_name fs1-sca11-a
service_description enclosure 6 status
check_command enclosure_status!enclosure!public!$HOSTADDRESS$!6

}

################################################################################
#
# Sample FS1 monitors
#
#
################################################################################

define command{

command_name physical_used_capacity
command_line /etc/nagios/conf.d/fs1-snmp.sh $ARG1$ $ARG2$ $ARG3$

}
define command{

command_name controller_status
command_line /etc/nagios/conf.d/fs1-snmp.sh $ARG1$ $ARG2$ $ARG3$ $ARG4$

}
define command{

command_name pilot_status
command_line /etc/nagios/conf.d/fs1-snmp.sh $ARG1$ $ARG2$ $ARG3$ $ARG4$

}
define command{

command_name enclosure_status
command_line /etc/nagios/conf.d/fs1-snmp.sh $ARG1$ $ARG2$ $ARG3$ $ARG4$

}

Finally the bash script that does the FS1 SNMP access, query OIDs, send back status codes and status information.

#!/bin/bash

#################################################
#/etc/nagios/conf.d/fs1-snmp.sh
#
# Catchall script for FS1 SNMP trap definitions
#
#
##################################################

COMP=$1
COMMUNITY=$2
HOST=$3
COMPNU=$4

case $COMP in
pilot)

PSTATE=$(snmpwalk -v 2c -c $COMMUNITY $HOST enterprises.15548.3.2.3.2.1.9.1.$COMPNU | awk '{print $4}' |tr -d '"')
printf "pilot $COMPNU state is $PSTATE\n"

if [ $PSTATE == "NORMAL" ]
then
exit 0

else
exit 2
fi
;;

controller)

CSTATE=$(snmpwalk -v 2c -c $COMMUNITY $HOST .1.3.6.1.4.1.15548.3.2.1.1.1.13.$COMPNU | awk '{print $4}' |tr -d '"')
printf "controller $COMPNU state is $CSTATE\n"

if [ $CSTATE == "NORMAL" ]
then
exit 0

else
exit 2

fi
;;

enclosure)

ESTATE=$(snmpwalk -v 2c -c $COMMUNITY $HOST enterprises.15548.3.2.2.1.1.8.$COMPNU | awk '{print $4}' |tr -d '"')
printf "enclosure $COMPNU state is $ESTATE\n"

if [ $ESTATE == "NORMAL" ]
then

exit 0

else

exit 2

fi
;;

used)

USED=$(snmpwalk -v 2c -c $COMMUNITY $HOST .1.3.6.1.4.1.15548.3.1.2.3.1.1.3 | awk '{print $4}' |tr -d '"')
TOTAL=$(snmpwalk -v 2c -c $COMMUNITY $HOST .1.3.6.1.4.1.15548.3.1.2.3.1.1.4 | awk '{print $4}' |tr -d '"')
PERCENT=$(awk "BEGIN { pc=100*${USED}/${TOTAL}; i=int(pc); print (pc-i<0.5)?i:i+1 }")

if [[ $PERCENT -le 80 ]]
then
printf "Array physical disk percentage used = $PERCENT%%\n"
exit 0

else
printf "Array is close to full, delete vols or add disk space\n"
exit 2

fi
;;

echo $"Usage: fs1-snmp.sh <component community host component-number>"
;;

esac

Screen Shot of Nagios 4.3.4 Services Link with FS1-2 Monitors

Nagios FS1 Sensors Screen Shot

Attachments

This solution has no attachment