FS System: How to Verify Single Tiered Extents Larger than 2 TB in Size.

Asset ID:	1-71-2129692.1
Update Date:	2016-05-09
Keywords:

Solution Type Technical Instruction Sure

Solution 2129692.1 : FS System: How to Verify Single Tiered Extents Larger than 2 TB in Size.

Applies to:

Oracle FS1-2 Flash Storage System - Version 6.1 to 6.2 [Release 6.1 to 6.2]
Information in this document applies to any platform.

Goal

This document describes how to use the Release 6.3 version of the codAudit utility on releases at or below R6.2.3-0280.01 to identify Extents larger than 2TB in size. Systems running R6.2.3-0280.02 or higher will not see this problem.

Solution

Whenever possible, customers should create and upload a MANUAL log bundle to the Oracle Callhome server for analysis by Oracle Support. See Document 1943289.1 FS System: How to Attach a Manual Log Bundle to an Existing MOS Service Request. Be sure to create a MANUAL log bundle that includes the System Configuration, Pilot and Controller logs.

For customers that cannot provide a log bundle, the latest version (350) of the codAudit utility is attached to this document. It has a MD5 checksum of 9b3060fa6b2d553ee05a5b3e8cc37997. Please contact Oracle Support for assistance in the execution of the codAudit utility and analysis of the results.

NOTE: An earlier version was in the 2of2 part of the R6.2.3 patch but it produces spurious errors and should not be used.

In addition to the utility itself, on site execution of the utility directly on the Pilot requires that a cod file be extracted from a log bundle. The syntax for the utility can be seen using the -h option (and shows the 350 version as 035000):

The latest version of the utility is on the ISDE/COREs under: /cores_data/local/tools/pillar/FlashStorage/codAudit

% codAudit -h
codAudit: build 060300-035000 on 04/29/16 13:52:46
usage: codAudit [-q] <cod filename>
-q for quick mode

NOTE: Always run this command with the -q option.

Set Up for Using the Utility

The following steps are for executing the codAudit utility on the CORES/ISDE servers:

NOTE: There is also a perl script (Bug_23110513_Checker.pl) that is easier to use and produces a cleaner output that can be used. It is located in the same directory as the utility and must be run in the same directory as the log bundle's *chsh.xml and A1*.cod files. See the help option (-h) for more details. It will NOT work in customer environments. If an error is seen in the output, run the codAudit utility for additional information on the failure.

Look for a current (today) SECONDARY_PERIODIC or MANUAL log bundle on ISDE/COREs:
amonv0025:% cfmagent -s "source_type=Axiom,serial_number=<FS1-2_SSN>" -o "m2id,tid,container_name"
....
58721399|2016-04-15T10:31:38.000Z|AK00126934-160415092855-160415103437-MANUAL_COLLECTION-m-01-01.tar
58743543|2016-04-15T12:33:17.000Z|AK00126934-160415103441-160415123618-MANUAL_COLLECTION-m-01-01.tar
58748671|2016-04-15T14:31:53.000Z|AK00126934-160415123621-160415143447-MANUAL_COLLECTION-m-01-01.tar
59010029|2016-04-20T12:32:00.000Z|AK00129634-160420043153-160420123556-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
59010064|2016-04-22T12:36:00.000Z|AK00126934-160421202731-160422014243-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
Using the mid2 file, reconstitute the log bundle in the SR directory:
amomv0025:/sr/SR#######% m2recon -m 59010064
Connecting to https://tds.us.oracle.com
Updating token
Number of instances to reconstitute = 1
Archive type = TAR
Compression type = null
Retrieving metadata - 10 nodes
Creating tar archive ./AK00126934-160421202731-160422014243-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
A13674D9582C024B.bbm.tar.gz
A13674D9582C024B.chsh.xml.tar.gz
A13674D9582C024B.cod.tar.gz
A13674D9582C024B.drivescrub.tar.gz
A13674D9582C024B.events.xml.tar.gz
A13674D9582C024B.manifest.xml.tar.gz
A13674D9582C024B.npperfdata.tar.gz
A13674D9582C024B.rmap.brmap.tar.gz
A13674D9582C024B.rmap.crmap.tar.gz
A13674D9582C024B.smartdata.tar.gz
M2ID 59010064 reconstituted from database
Extract the cod file from the log bundle:
amomv0025:/sr/SR#######% tar xvf AK00126934-160421202731-160422014243-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
A13674D9582C024B.bbm.tar.gz
A13674D9582C024B.chsh.xml.tar.gz
A13674D9582C024B.cod.tar.gz
A13674D9582C024B.drivescrub.tar.gz
A13674D9582C024B.events.xml.tar.gz
A13674D9582C024B.manifest.xml.tar.gz
A13674D9582C024B.npperfdata.tar.gz
A13674D9582C024B.rmap.brmap.tar.gz
A13674D9582C024B.rmap.crmap.tar.gz
A13674D9582C024B.smartdata.tar.gz
Extract the cod file:
amomv0025:/sr/SR#######% tar xvzf A13674D9582C024B.cod.tar.gz
A13674D9582C024B.cod

ssh to the active Pilot. For details, see Document 2029847.1 FS System: How to Enable SSH Access to the Pilot.
Create a directory to work from and cd to that directory:
[root@pilot1 ~]# mkdir -p /var/tmp/AUDIT
[root@pilot1 ~]# cd /var/tmp/AUDIT
[root@pilot1 AUDIT]#
Use one of the following methods to obtain a current (same day) log bundle:
1. Check existing log bundles for a current SECONDARY_PERIODIC in the /var/PillarCallHome directory and copy it to /var/tmp/AUDIT:
  
  [root@pilot1 AUDIT]# ls -ltr /var/PillarCallHome | grep SECONDARY_PERIODIC
  -rw------- 1 root root 16520192 Dec 4 01:40 AK00xxxxxx-151204004617-151204014059-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  -rw------- 1 root root 127974912 Dec 11 01:41 AK00xxxxxx-151210192033-151211014149-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  -rw------- 1 root root 111535616 Dec 18 01:41 AK00xxxxxx-151217192034-151218014136-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  -rw------- 1 root root 103431168 Dec 25 01:41 AK00xxxxxx-151224192034-151225014134-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  ...
  -rw------- 1 root root 15198208 Apr 1 01:41 AK00xxxxxx-160401013922-160401014113-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  -rw------- 1 root root 75181568 Apr 8 01:41 AK00xxxxxx-160407192040-160408014141-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  -rw------- 1 root root 126627328 Apr 15 01:42 AK00xxxxxx-160414192042-160415014203-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  -rw------- 1 root root 110319616 Apr 22 01:42 AK00xxxxxx-160421202731-160422014243-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
  [root@pilot1 AUDIT]# cp /var/PillarCallHome/AK00xxxxxx-160421202731-160422014243-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
2. If no current SECONDARY_PERIODIC bundle is available, use the initial steps in Document 1943289.1 FS System: How to Attach a Manual Log Bundle to an Existing Service Request to create a MANUAL log bundle that includes the System Configuration, Pilot and Controller logs and copy from /var/PillarCallHome to /var/tmp/AUDIT.
Extract the log bundle to obtain the cod file:
[root@pilot1 AUDIT]# tar xvf AK00xxxxxx-160421202731-160422014243-SECONDARY_PERIODIC_COLLECTION-p-02-02.tar
A1362260C5AAD400.chsh.xml.tar.gz
A1362260C5AAD400.events.xml.tar.gz
A1362260C5AAD400.cod.tar.gz <================
A1362260C5AAD400.rmap.crmap.tar.gz
A1362260C5AAD400.rmap.brmap.tar.gz
A1362260C5AAD400.bbm.tar.gz
A1362260C5AAD400.drivescrub.tar.gz
A1362260C5AAD400.stats.tar.gz
A1362260C5AAD400.tds.tar.gz
A1362260C5AAD400.smartdata.tar.gz
A1362260C5AAD400.npperfdata.tar.gz
A1362260C5AAD400.manifest.xml.tar.gz
Extract the cod file:
[root@pilot1 AUDIT]# tar xvzf A1362260C5AAD400.cod.tar.gz
A1362260C5AAD400.cod
Download the utility from this document.
Copy the utility to the active Pilot:
1. Using WinSCP (or similar program) copy the compressed utility to the active Pilot:
  
  C:\Program Files (x86)\WinSCP>WinSCP root@<FS1_IP_ADDR>:/var/tmp/AUDIT
  Opening session using command-line parameter in scripting is deprecated. Use 'open' command instead.
  Searching for host...
  Connecting to host...
  Authenticating...
  Using username "root".
  Password:
  Authenticated.
  Starting the session...
  Session started.
  Active session: [1] root@<FS1_IP_ADDR>
  winscp> pwd
  /var/tmp/AUDIT
  winscp> put C:\temp\codAudit.tar.gz
  C:\temp\codAudit.tar.gz | 6592 KB | 7411.2 KB/s | binary | 100%
  winscp>exit
  C:\Program Files (x86)\WinSCP>
2. Uncompress the utility and confirm it is executable and it's integrity:
  
  [root@pilot1 AUDIT]# tar xvzf codAudit.tar.gz
  codAudit
  [root@pilot1 AUDIT]# ls -l
  total 26500
  -rwxr-xr-x 1 559332 999 20338405 Apr 21 16:05 codAudit
  -rw-r--r-- 1 root root 6750744 Apr 26 15:16 codAudit.tar.gz
  [root@pilot1 AUDIT]# sum codAudit
  15047 19862
  [root@pilot1 AUDIT]#

Execution and Analysis of codAudit

With the cod file as an argument, run the codAudit utility with the -q option and redirect stderr and stdout to a file:

[root@pilot1 AUDIT]# ./codAudit -q A1362260C5AAD400.cod > audit.out 2>&1
Examine the resultant text file for "MAUs may be too large":

The following two lines are symptoms of the issue (See Service Alert 2132072.1):
07:03:22.421605 322262784 CodValidation.cpp 831 ERROR: extent 0xa2110cfb5f39b6c5 capacity 0x299d MAUs may be too large to safely extent migrate (extent mi
07:03:22.421655 322262784 CodValidation.cpp 831 ERROR: extent 0xa2110cfb5f5ed010 capacity 0x206b MAUs may be too large to safely extent migrate (extent mi

Errors and Warnings like these can be ignored:

07:02:40.893306 322262784 TaskFactorySwitch.cpp 3378 ERROR: Failed to create task for unknown command type 0x1c8027
07:02:40.893425 322262784 TaskExecutionThread.cpp 345 WARNING: task 0x82310006 goes to run queue because launchSubtask failed
07:02:40.893493 322262784 TaskFactorySwitch.cpp 3378 ERROR: Failed to create task for unknown command type 0x1c8028
07:02:40.893533 322262784 TaskExecutionThread.cpp 345 WARNING: task 0x82310006 goes to run queue because launchSubtask failed

If COD Audit failure Errors like these are observed:

07:03:22.422554 322262784 CodValidation.cpp 100 ERROR: validation problem for suid 0xa2110cfb5f39b6c5
07:03:22.422583 322262784 CodValidation.cpp 100 ERROR: validation problem for suid 0xa2110cfb5f5ed010
07:03:22.422728 322262784 ColdStartAllCodTask.cpp 165 ERROR: CHECK_RESULT_FOR_CONFIG_COD, ColdStartCod failed. ColdStartAllCodTask(0x81960001), errorCod 0x001ce
07:03:22.422780 322262784 ColdStartAllCodTask.cpp 258 ERROR: ColdStartAllCodTask (0x81960001) FAILED, error code (0x001ce023)
COD audit failed 0x1ce023

please send the audit.out and the ut-trace.out files it produces and the original log bundle and send them to Oracle Support. Do not attempt a software upgrade until these errors are understood.

Oracle Support: add any COD audit failures to bug 23205625. Do not upgrade until the COD Audit failure is better understood by Engineering.

If the only "MAUs may be too large to safely extent migrate (extent mi" error is found, that is the signature of Service Alert 2132072.1 and the fix is in Patch 23132401 and higher.
If the codAudit utility was run on a customer Pilot, remove all files created during this process:

[root@pilot1 AUDIT]# cd /var/tmp
[root@pilot1 tmp]# rm -Rf AUDIT
[root@pilot1 tmp]#

References

<BUG:23110513> - CONTROLLER-02 FAILED AND MANY DGS FAILED

Attachments

This solution has no attachment