Pillar Axiom 300 Storage System
 
Pillar Axiom 600 Storage System
 
Pillar Axiom 500 Storage System
 

PLA-Support>Sun Systems>DISK>Axiom>SN-DK: Ax600
 

In this Document

Goal

Solution

Applies to:

Goal

1. Introduction

1.1 What's it all about?
AxiomOne Path Manager (APM) is a software package which runs on SAN Host systems to manage multipathing of SAN connections between the host and the Pillar Axiom system. It also provides integration into the management system on the Axiom. There are several different implementations of APM for different host Operating Systems.

1.2 This Document

This document explains what APM is, and describes how and when diagnostic logs should be collected for APM.

1.3 Glossary

This is the repository of terms that should be used in reference to this development on all OSes:

2. APM Overview

Each implementation of APM consists of either one or two major components:

• A user-mode component generally referred to as the Daemon. Its job is to manage and monitor the SAN multipathing on the host, and to integrate with the AxiomONE Storage Services Manager on all Pillar Axiom systems to which the host is attached. All implementations of APM include this component. It runs as a Service on Windows and as a background task on UNIX-alikes.
• A kernel-mode component generally referred to as the Driver. Its job is to implement the multipathing of the SAN connections, grouping paths to a LUN together, monitoring for failure and automating failover, and so on. Some implementations of APM provide this component in its entirety; some provide a module which gives Axiom-specific hints and recommendations to an OS-supplied multipathing module; some do not provide this component at all, relying entirely on multipathing functionality provided by the Operating System.

There are also a number of minor supporting acts which vary between implementations.

Solution

3. APM Logs

3.1 Log Generation

The Daemon and the Driver each maintain an in-memory trace buffer. These traces record significant events and information which is useful for diagnosing problems. The buffers are of limited size and written in a circular fashion; once full, newer entries overwrite older ones. It is important to note that these logs are kept as in-memory buffers; this has a number of consequences:

• The trace logs will be lost if the component is stopped or restarted. If you restart the Daemon, then all Daemon log entries from before the restart will be lost. Similarly, Driver log entries will be lost if the driver is unloaded or reloaded. If you plan to stop or restart either APM component (or reboot the host) as part of a problem investigation or correction, it is important that you collect APM logs beforehand.
• Some extreme failures can result in memory dumps: a core dump of the Daemon process after problems in the daemon, or a kernel memory dump (or full system memory dump) if the system crashes or blue-screens. We hope that APM itself will never be the cause of this sort of failure, but if they happen then the memory dump should contain the Daemon or Driver trace logs up to the moment of failure. The logs can be extracted from these dumps (along with a lot of other useful information with a bit of luck) so any memory dumps should always be provided as part of an incident report.

3.2 Log Collection

3.2.1 Collect System Information at the Axiom

The daemon attempts to have a continuously open connection to the management subsystem on every Axiom to which the host has SAN connections. The state of a host’s Daemon’s connection to an Axiom can be seen in the Axiom GUI or CLI. The Hosts page in the GUI has a column headed AxiomONE Path Manager which shows this state. The entry for a host will be one of:

• Not registered - this indicates that an APM Daemon on the host has never successfully logged in to the Pilot
• Not Communicating - the Daemon has logged in the past, but does not have a connection at the moment
• Communicating - the Daemon is successfully connected to the Pilot.

When you do the Collect System Information action at the Axiom GUI, and check Debug Logs, then by default the Axiom will send a request out to the APM Daemon on all hosts whose Daemon is Communicating. The sending of these messages is controlled by the All SAN Hosts option on Debug Logs, which is checked by default. This message asks the Daemon to collect logs and transfer them to the Axiom.

The Daemon transfers its own trace buffer, the Driver’s trace buffer (if the implementation has a Driver), and various other files of information which vary with the OS. The additional files will typically include the outputs of various diagnostic commands, and the contents of various standard OS log files and configuration files. MCC on the pilot then writes out these files as it receives them and incorporates them in the log download bundle.

This is the preferred way to collect APM logs. To thoroughly understand events seen by APM on the host, it is often necessary to correlate the APM logs with what was happening at the same time in components on the Axiom, such as SAN and MCC. Collecting logs in this way at the Axiom (selecting Debug Logs and leaving all options as set by default) ensures we get a coherent set of logs for the host and the Axiom (and also for other hosts which can often be helpful).

However, this method of collecting logs depends on the Daemon on the host successfully communicating with the Pilot. It must be in the Communicating state when the request to collect logs is made at the Axiom, and it must remain continuously in this state until the logs have been collected. If you have any doubts about the status of the connection at any time during the log collection, you should collect logs locally on the host as described next.

3.2.2 Local Log Collection at the Host

If the Daemon is not communicating reliably with the Axiom then you can manually send a request to the Daemon asking it to collect the same set of logs, but dump them in a directory on the host. To send this request, you need to log in as Administrator (on Windows) or root (on UNIX-alikes) then run the Daemon executable file with the -d option.

On Windows: log in as Administrator and open a command shell. One way to find the path to the Daemon executable is to run the Services applet and look at the Properties of the AxiomONE Path Manager Service; the Path to executable field gives the information you need. You then run this program in the command shell and give it the -d option; a typical example would be:

On a UNIX-alike: log in as root and run the command

If the Daemon is running, the output should include the path to its executable. For example, on an AIX system this might produce output like:

Ignore any options shown in the output, and run the program with the -d option; in this case you would run

Running the command with the -d option simply sends a request to the current Daemon image, so this command will usually return very quickly. The Daemon will usually start work on collecting the logs immediately, though the request may sometimes get queued up behind other activity. The Daemon will create a new directory on the host to contain the log set. The location of the new directory is as follows: 

• Windows: %windir%\Debug\axiompmd
• UNIX-alike: /var/log/axiompmd

The Daemon will create these directories if necessary. In here it will create a directory to contain the new log set, with the directory name derived from the date and time it started collecting; for example, Fri_Apr_18_17_30_23_2008. It will then start to collect the logs and place them in this directory. The last action of creating the log set is to create an empty log file with a name consisting of the host name followed by COLLECTION_FINISHED; for example dopey_COLLECTION_FINISHED. This indicates what its name suggests.

The steps to do a local log collection are therefore:

1. find the path to the Daemon executable and run it with the –d option
2. wait for the axiompmd log directory to be created
3. wait for a directory to be created in the axiompmd log directory with a name indicating the current date and time
4. wait for an empty file to be created in the date-and-time directory with COLLECTION_FINISHED on the end of its name
5. collect all the files in the date-and-time directory
6. delete the date-and-time directory if you don’t want to leave clutter on the host.

As explained earlier, it is often vital to be able to correlate APM logs with logs of the same period from the Axiom. This is often the case even if the problem you are investigating is that APM cannot communicate with the Axiom. If you have collected APM logs locally on the host, you should usually also collect logs at the Axiom so we get both ends of the story.

3.3 Other APM Diagnostics

The trace logging system in APM is flexible and highly configurable. It can be controlled for both the Driver and the Daemon by options to the Daemon when it is started. The default options are tuned to collect the most useful information without wrapping the logs around too quickly; they are adequate in almost all situations.

In rare cases it may be necessary to set non-default logging options to collect specific information. This should only be done at the request of the APM development team, using options and methods which will be specified for each case.

This solution has no attachment