| Asset ID: |
1-75-1397577.1 |
| Update Date: | 2018-03-20 |
| Keywords: | |
Solution Type
Troubleshooting Sure
Solution
1397577.1
:
Sun Storage 7000 Unified Storage System: How to Troubleshoot Shadow Migration
| Related Items |
- Sun ZFS Storage 7420
- Sun Storage 7110 Unified Storage System
- Oracle ZFS Storage ZS3-2
- Oracle ZFS Storage ZS4-4
- Sun Storage 7210 Unified Storage System
- Sun Storage 7410 Unified Storage System
- Sun ZFS Storage 7120
- Oracle ZFS Storage ZS3-4
- Sun Storage 7310 Unified Storage System
- Sun ZFS Storage 7320
- Oracle ZFS Storage Appliance Racked System ZS4-4
- Oracle ZFS Storage ZS3-BA
|
| Related Categories |
- PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
- _Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
|
In this Document
Applies to:
Sun ZFS Storage 7120 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
7000 Appliance OS (Fishworks)
NAS head revision : [not dependent]
BIOS revision : [not dependent]
ILOM revision : [not dependent]
JBODs Model : [not dependent]
CLUSTER related : [not dependent]
Purpose
This document is provided to assist in troubleshooting shadow migration issues in a Sun Storage 7000 Unified Storage System.
It will help to frame the problem and to drive the resolution path for all known issues.
Finally, it provides some information to collect analytics in order to isolate the issue and provide required information to ORACLE support team, if needed.
Troubleshooting Steps
Shadow Migration Does Not Support Following File Types
The following file types will not be migrated from the source and result in error.
This is by design to avoid problems in the target.
- Files in /proc directory
- Door files
- Sockets
- Ports
- Pipes
- Locked Files
Below is the 'ls -l' output of each of these file types - except locked files.
Files and directories in the /proc directory look like regular files, however, you cannot migrate them.
osklabss90# pwd
/proc
osklabss90# ls -l |head -5
total 158
dr-x--x--x 5 root root 864 Nov 2 04:13 0
dr-x--x--x 5 root root 864 Nov 2 04:13 1
dr-x--x--x 5 root root 864 Nov 2 04:14 1013
dr-x--x--x 5 root root 864 Nov 2 04:14 102
Door files -- permission starts with a capitalized 'D'
root@ht3:/proc/209/fd# ls -l
total 0
D--------- 3 root root 0 Jan 1 1970 4
D--------- 2 root root 0 Jan 1 1970 6
D--------- 2 root root 0 Jan 1 1970 7
D--------- 2 root root 0 Jan 1 1970 8
Socket Files -- permission starts with an 's'
root@ht3:/tmp# ls -l
total 24
srwxrwxrwx 1 mysql mysql 0 Nov 25 16:56 mysql.sock
Port Files -- permission starts with a capitalized 'P'
root@ht3:/proc/20615/fd# ls -l
total 0
c--------- 1 tsuhi02 tty 133, 1 Dec 8 09:41 0
c--------- 1 tsuhi02 tty 133, 1 Dec 8 09:41 1
c--------- 1 tsuhi02 tty 133, 1 Dec 8 09:41 2
P--------- 2 root root 0 Dec 8 09:41 3
Pipes -- permission starts with a lower-case 'p'
root@ht3:~/samples# ls -l
total 0
prw-r--r-- 1 root root 0 Dec 8 08:51 named_pipe
Note: Source Files Must Not Be Locked
Shadow migration cannot migrate locked files.
Before making the source file system read-only, remove all mandatory locks and other types of file locking on both files and directories.
Framing the problem
Shadow migration is the online process used by Sun Storage 7000 Appliance for migrating data from local or external sources preserving the access to data at any time.
The architecture of Shadow Migration is based on 3 main components: source, target and shadow migration service.
"Source" could be an existing local Sun Storage 7000 Appliance share, or and existing external NFS share.
"Target" is a Sun Storage 7000 Appliance share added for shadow migration purpose.
"Shadow Migration service" is the Sun Storage 7000 Appliance Data Service responsible for data migration. It handles clients requests to data which are migrating.
Possible shadow migration issues are :
- shadow migration setup issue
- shadow migration service issue
- shadow migration errors
- shadow migration hangs
- shadow migration performance
- clients got an error accessing migrating data
- clients hangs when access migrating data
- ACL issue on migrated data
Troubleshooting
Shadow migration setup issue
Symptoms
Setup of shadow migration share fails with following error on BUI:
Data migration source: Failed to mount filesystem
or following error on CLI
error: An unanticipated system error occurred: cannot mount
'<sharename>': Invalid argument
This may be due to transient failure, or a software defect. If this problem
persists, contact your service provider
Shadow migration target system can't access shadow migration source path
- Check whether shadow migration source filesystem exists
- Check whether shadow migration source filesystem is shared with NFS and fix any problem
- Check whether shadow migration source filesystem has some wrong Protocol NFS exceptions configure and fix any problem
Shutdown of Shadow migration source
BUI update for the target share stalled and shows different amount of space used than the CLI does.
Cause
The source of the shadow migration was shutdown and /or NFS connection was no longer available.
Solution
Stop shadow migration before you need to shutdown or disconnect the NFS source for the migration.
Please see some of the bugs coming off BUG 15702398 - SUNBT7026945-12.0 "Handle loss of shadow source gracefully."
Shadow migration service issue
Shadow migration service is offline or in maintenance
Cause
Shadow migration service has been disabled or got a problem and entered maintenance status
Solution
re-enable or restart service
Shadow migration status reports migration has completed with some errors
Shadow migration failed to access some files
Check whether shadow migration source has been mounted in read-only
Shadow migration status isn't completed yet and doesn't show any progress
- Shadow migration service is in hung
- Shadow migration performance are slow and looks like hung
See Doc ID
1377069.1 "Sun Storage 7000 Unified Storage System: Shadow Migration Copy Performance Is Slow"
Shadow migration performance
Shadow migration progress is slow
- Source filesystem is huge
- Source filesystem contains lots of little files and/or has lots of sub-directories
Solution
See Doc ID
1377069.1 "Sun Storage 7000 Unified Storage System: Shadow Migration Copy Performance Is Slow"
Clients got an error accessing migrating data
Clients got NFS error accessing files
- NFS share is unavailable
- File is unavailable
- File ACL are wrong
Solution
- Check NFS filesystem availability and fix any issue on it
- Check file availability and fix any issue on it
- Check file permission and fix any issue on it
- Check if you can read file with some trivial command
Conclusion
If after troubleshooting steps above shadow migration issue is still not fully understood or fixed, engage Oracle Support by opening a Service Request for assistance.
Please include all the relevant details and information - including examples of any errors that you see - along with an accurate problem description in the SR notes.
If possible, a current supportbundle (from both heads, if this a cluster system) should also be obtained and uploaded to Oracle.
The following links will provide more information:
Doc ID 1019887.1 "Sun Storage 7000 Unified Storage System: How to collect support bundle using the BUI or CLI"
Doc ID 1345655.1 "Sun Storage 7000 Unified Storage System: How to provide the correct Serial Number when opening an Oracle Service Request on a ZFS Storage Appliance or S7000 series NAS"
It may be necessary for the Oracle Support Engineer to remotely run some 'emergency shell' commands. To accomplish this, the Oracle Support Engineer may request that you initiate an Oracle Shared Shell session. It would be useful if you are already familiar with this remote access tool - please see:
https://www.oracle.com/us/support/systems/premier/shared-shell-sun-systems-163755.html
Other useful information:
1. The Online Appliance Wiki documentation can be found at
https://<appliance-ip-address>:215/wiki/index.php
2. To upgrade to the latest Appliance Firmware Release:
There are many improvements in later Appliance Firmware releases, please check the current Appliance Firmware revision and, if required, upgrade to the latest release please see Doc ID 2021771.1 "Oracle ZFS Storage Appliance: Software Updates".
Known Issue
An unanticipated system error occurred: could not change user: I/O error
An issue was discovered where Shadow Migration would fail to setup on a particular NFS share on solaris 10 and above.
The NFS share's com.sun.sk:default_user and sun.com.ak:default_group properties has been changed to map specific UIDs/GUIs to the default user/group for the share.
The properties can be changed by the administrator, but we believe that it is possible to make a combination that may prevent Shadow Migration from working correctly.
Please see SR 3-8614525741 for more details.
References
<BUG:15702398> - SUNBT7026945 HANDLE LOSS OF SHADOW SOURCE GRACEFULLY
Attachments
This solution has no attachment
