Oracle Exadata Hardware
 
Exadata X3-2 Hardware
 
Exadata X4-2 Hardware
 
Exadata Database Machine X2-2 Hardware
 
Exadata X4-2 Half Rack
 
Exadata X5-8 Hardware
 
Exadata X5-2 Hardware
 
Exadata Database Machine X2-8
 
Oracle Exadata Storage Server Software
 
Exadata X4-8 Hardware
 
Exadata X3-2 Hardware
 
Exadata Database Machine V2
 
Exadata X3-8 Hardware
 
Exadata Database Machine X2-2 Hardware
 
Exadata Database Machine V2
 
Exadata X3-8b Hardware
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 
_Old GCS Categories>ST>Server>Engineered Systems>Exadata>Hardware
 

In this Document

Purpose

Troubleshooting Steps

--- Software Requirements/Prerequisites ---

--- Configuring the Script ---

--- Running the Script ---

For Sun Oracle Exadata Environments:

Community Discussions

References

Applies to:

Purpose

This document provides the set of commands required to collect diagnostic information required for disk failures on the Exadata Storage Servers and could be used on Db nodes or Storage Servers for some other hardware issues.

Troubleshooting Steps

--- Software Requirements/Prerequisites ---

The script needs to be executed as root on the Exadata Storage server having disk problems and sometimes also on Db nodes or Storage Servers for some other hardware issues.

The script collects diagnostic information required by Sun or HP.

--- Configuring the Script ---

Executed as root user.

--- Running the Script ---

For Sun Oracle Exadata Environments:

sundiag.sh is included in the Exadata base image in /opt/oracle.SupportTools:

For image versions 12.1.2.2.0 or later, use the sundiag.sh already included in /opt/oracle.SupportTools/sundiag.sh.

For all systems using any image version prior to 12.1.2.2.0 then update the existing sundiag.sh to the latest v12.1.2.2.0_150917 version.

Note: as of 12.1.2.2.0 image, the version number scheme changed from v1.5.1 as the prior release, to version numbers matching the image release with date.

Updating sundiag.sh to the latest on older image systems:

1. Download file sundiag.zip attached to this Note and copy to the first compute node under /tmp.
2. Using dcli, copy the file to all the nodes and unzip it
      #dcli -l root -g /opt/oracle.SupportTools/onecommand/all_group -f /tmp/sundiag.zip -d /tmp
      #dcli -l root -g /opt/oracle.SupportTools/onecommand/all_group "cd /tmp;unzip sundiag.zip;ls -l sundiag_12.1.2.2.0_150917.sh;md5sum sundiag_12.1.2.2.0_150917.sh"

Output should be like this, for all the nodes referenced in file all_group

3. Copy the new version of sundiag.sh to the default location
     #dcli -l root -g /opt/oracle.SupportTools/onecommand/all_group "cd /opt/oracle.SupportTools;mv sundiag.sh sundiag.sh.orig"
     #dcli -l root -g /opt/oracle.SupportTools/onecommand/all_group "cd /tmp;mv sundiag_12.1.2.2.0_150917.sh /opt/oracle.SupportTools/sundiag.sh;md5sum /opt/oracle.SupportTools/sundiag.sh;ls -l /opt/oracle.SupportTools/*sundiag*"

4. Remove temporary files
 
    #dcli -l root -g /opt/oracle.SupportTools/onecommand/all_group "cd /tmp;rm -fr sundiag.zip;rm -fr sundiag_12.1.2.2.0_150917.sh"

Usage of sundiag.sh:

Execution will create a date stamped tar.bz2 file in /var/log/exadatatmp/sundiag_<hostname>_<serial#>_<date/time>.tar.bz2 (in /tmp on v1.4-1.5.1).
Upload this file to the Service Request.

For gathering OS Watcher data alongside sundiag:

Usage:

# /opt/oracle.SupportTools/sundiag.sh osw <time ranges>

By default sundiag will collect OSWatcher/ExaWatcher, Cell Metrics and traces, if there was an alert in the last 7 days. If there is more than one alert, the latest alert is chosen to set the time range for data collection. Time range is 8hrs prior to and 1hr after the latest alert, for the total of 9 hrs e.g: latest alert timestamp =  2014-03-29T01:20:04-05:00
      echo  Time range = 2014-03-28_16:00:00 and 2014-03-29_01:00:00

User can also specify time ranges, which takes precedence over default behavior of checking for alerts. This argument when used expects a value of one or more comma separated time ranges. OSWatcher/ExaWatcher, cell metrics and traces will be gathered in those time ranges.

The format for time range(s) is <from>-<to>,<from>-<to> and so on without spaces  where <from> and <to> format is <date>_<time>.
<date> and <time> format should be any valid format that can be recognized by 'date' command. The command 'date -d <date>' or 'date -d <time>' should be valid.

For Example: /opt/oracle.SupportTools/sundiag.sh osw 2014/03/31_15:00:00-2014/03/31_18:00:00

Note: Total time range should not exceed 9 hrs. Only the time ranges that fall within this limit are considered for the collection of above data. This is to limit the amount of data being gathered to be appropriate for the problem being analysed.

Execution will create a date stamped tar.bz2 file in /var/log/exadatatmp/sundiag_<hostname>_<serial#>_<date/time>.tar.bz2 (in /tmp on v1.4-1.5.1) including OS Watcher archive logs. These logs may be very large.

Upload this file to the Service Request.

For gathering ILOM  data alongside sundiag:

Usage:

# /opt/oracle.SupportTools/sundiag.sh snapshot

Execution will create a date stamped tar.bz2 file in /var/log/exadatatmp/sundiag_<hostname>_<serial#>_<date/time>.tar.bz2 (in /tmp on v1.4-1.5.1) which includes running an ILOM snapshot. In order to collect a snapshot, the host (not ILOM) 'root' password is required to facilitate network transfer of the snapshot into the /tmp directory.  This is the preferred method of ILOM data collection.

If there are concerns about entering the host 'root' password, then an alternative option is provided using the "# /opt/oracle.SupportTools/sundiag.sh ilom" which will use IPMI to gather user-level ILOM outputs. This is usually good but the ILOM snapshot level can provide more underlying ILOM outputs for troubleshooting issues with ILOM and system faults that the user-level data may not provide.

Upload this file to the Service Request.

For gathering sundiag data across a whole rack:

For gathering sundiag.sh outputs on versions of sundiag.sh, where the filename is unique for each node (v1.4 and later), use the following from DB01:

1. [root@exadb01 ~]# cd /opt/oracle.SupportTools/onecommand (or wherever the all_group file is with the list of the rack hostnames)

2. [root@exadb01 onecommand]# dcli -g all_group -l root /opt/oracle.SupportTools/sundiag.sh 2>&1
<this will take up to several minutes while each node runs sundiag.sh>

3. Verify there is output in /tmp or  /var/log/exadatatmp/ on each node:
[root@exadb01 onecommand]# dcli -g all_group -l root --serial 'ls -l /tmp/sundiag* '   (v.1.4-1.5.1)

[root@exadb01 onecommand]# dcli -g all_group -l root --serial 'ls -l /var/log/exadatatmp/sundiag* '   (v12.1.2.2.0_150917)

4. Make a temporary directory to copy for zipping:
[root@exadb01 onecommand]# mkdir dbm01_sundiags_date  

It is recommended the date be of the format YYMMDD year, month, day for SR's where multiple days of analysis may be required.

5. Copy the generated sundiag files from the nodes to the temporary directory (/tmp on v1.4-1.5.1, /var/log/exadatatmp on v12.1.2.2.0_150917):

[root@exadb01 onecommand]# for H in `cat all_group`; do  scp -p $H:/tmp/sundiag*.tar.bz2 dbm01_sundiags_date ; done  

[root@exadb01 onecommand]# for H in `cat all_group`; do  scp -p $H:/var/log/exadatatmp/sundiag*.tar.bz2 dbm01_sundiags_date ; done

6. Bundle them into a single file for upload to Oracle:

[root@exadb01 ~]# tar jcvf exa_rack_sundiag_date.tar.bz2 dbm01_sundiags_date

For gathering sundiag.sh outputs on older versions of sundiag.sh (prior to v1.4) where the filename generated does not include hostname.
When gathering sundiag.sh outputs across a whole rack using dcli, the outputs may end up with the same tarball name which will overwrite each other upon unzipping.  To avoid this, use the following from DB01:

1. [root@exadb01 ~]# cd /opt/oracle.SupportTools/onecommand (or wherever the all_group file is with the list of the rack hostnames)

2. [root@exadb01 onecommand]# dcli -g all_group -l root /opt/oracle.SupportTools/sundiag.sh 2>&1
<this will take up to about 2 minutes>

3. Verify there is output in /tmp on each node:
[root@exadb01 onecommand]# dcli -g all_group -l root --serial 'ls -l /tmp/sundiag* '

4. Sort them by hostname into directories, as they will likely mostly have the same filename with the same date stamp:
[root@exadb01 onecommand]# for H in `cat all_group`; do mkdir /root/rack-sundiag/$H ; scp -p $H:/tmp/sundiag*.tar.bz2 /root/rack-sundiag/$H ; done

5. [root@exadb01 onecommand]# cd /root/rack-sundiag

6. [root@exadb01 ~]# ls exa*
exacel01:
sundiag_2011_05_24_10_11.tar.bz2

exacel02:
sundiag_2011_05_24_10_11.tar.bz2
...
exadb08:
sundiag_2011_05_24_10_11.tar.bz2

7. [root@exadb01 ~]# tar jcvf exa_rack_sundiag_oracle.tar.bz2 exa*
exacel01/
exacel01/sundiag_2011_05_24_10_11.tar.bz2
exacel02/
exacel02/sundiag_2011_05_24_10_11.tar.bz2
...
exadb08/
exadb08/sundiag_2011_05_24_10_11.tar.bz2

8. [root@exadb01 ~]# ls -l exa_rack_sundiag_oracle.tar.bz2
-rw-r--r-- 1 root root 3636112 May 24 10:21 exa_rack_oracle.tar.bz2

Upload this file to the Service Request.


For HP Oracle Exadata Environments:

1. Download to the Exadata Storage Server with the disks failures,  file deaddisk.zip
2. Unzip into any directory
3. Change permissions (700) to file deaddisk.sh and execute:
    # chmod 700 deaddisk.sh
    #./deaddisk.sh
4. Upload the zip file generated. Under /tmp, will be a zip file with the content of directory created by the     
    execution of the script, format info_deaddisk_.zip,
    Example: info_deaddisk_2009-01-18-19:52:14.zip


If link for deaddisk.zip is broken, please copy this code:

Community Discussions

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

References

This solution has no attachment