Oracle ZFS Storage Appliance: NDMP error "Processing directories aborted"

Asset ID:	1-72-2278465.1
Update Date:	2018-03-20
Keywords:

Solution Type Problem Resolution Sure

Solution 2278465.1 : Oracle ZFS Storage Appliance: NDMP error "Processing directories aborted"

Applies to:

Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS3-BA - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

Customer report an issue with NDMP backups not completing.

From the NDMP logs :

017-05-26T03:50:41+0000 [155616] -------- backup initiated --------
2017-05-26T03:50:41+0000 [155616] env(TYPE): "dump"
2017-05-26T03:50:41+0000 [155616] env(FILESYSTEM): "/export/usershome/u057"
2017-05-26T03:50:41+0000 [155616] env(PREFIX): "/export/usershome/u057"
2017-05-26T03:50:41+0000 [155616] env(LEVEL): "1"
2017-05-26T03:50:41+0000 [155616] env(HIST): "y"
2017-05-26T03:50:41+0000 [155616] env(UPDATE): "y"
2017-05-26T03:50:41+0000 [155616] env(SNAPSURE): "Y"
2017-05-26T03:50:41+0000 [155616] mode(NDMP V4)
2017-05-26T03:50:41+0000 [155616] env(HIST): "y"
2017-05-26T03:50:41+0000 [155616] env(EXCLUDE) not defined
2017-05-26T03:50:41+0000 [155616] env(FILES) not defined
2017-05-26T03:50:41+0000 [155616] backup default env(DIRECT): YES
2017-05-26T03:50:41+0000 [155616] env(UPDATE): "y"
2017-05-26T03:50:41+0000 [155616] TYPE: dump
2017-05-26T03:50:41+0000 [155616] Local operation: 65536
2017-05-26T03:50:41+0000 [155616] Adjusted read size: 524288
2017-05-26T09:48:19+0000 [155616] bksize 3795027605922, 3706081646KB, 3619220MB
2017-05-26T11:51:11+0000 [155616] INCOMPLETE
2017-05-26T11:51:11+0000 [155616] *** E R R O R *** [mark_cb:203]:Processing directories aborted.

Cause

To analyze the issue, the customer split the u057 directory into smaller parts.

The issue was narrowed the issue in one directory : /export/usershome/u057/liuh

The backup of the liuh directory does not finish - but also does not report an issue.

Checking previous errors

ndmplog.0:2017-05-23T10:17:51+0000 [101306] *** E R R O R *** [ndmpd_fhnode_v3_cb:688]:"/export/usershome/u057/liuh/research/manuscript/nature reserves/habitats/p0=40/q00=4080/size_18+18/times=10273.csv" -1

After checking, determined that liuh is a huge directory with a lot of files - around 147685518. So, the NDMP backup is timing out, trying to backup this file directory.

From document https://www.veritas.com/support/en_US/article.TECH21246 we see the NDMP backup has a timeout value of NDMP_PROGRESS_TIMEOUT

Solution

1. Stop the backup on liuh

2. Remove files from the liuh directory (reduce usage)

3. Restart NDMP backup

Alternatively, engage Veritas to increase the NDMP progress timeout.

Attachments

This solution has no attachment