ZFS NDMP Backups Failing after applying 2013.06.05.4.2,1-1.1 (2013.1.4.2)

Asset ID:	1-72-2057359.1
Update Date:	2016-07-19
Keywords:

Solution Type Problem Resolution Sure

Solution 2057359.1 : ZFS NDMP Backups Failing after applying 2013.06.05.4.2,1-1.1 (2013.1.4.2)

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.6.2.2 to 2.0.6.2.3
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on x86-64 (64-bit)
Linux x86-64
Oracle Virtual Server x86-64
Issue is not specific to Exalogic environment. It can happen on standalone ZFS appliance running with the specified akd version

Symptoms

In Exalogic racks with July 2015 PSU (both physical & virtual) issue of NDMP backups failing is seen. Below is the error logged on the ZFS storage appliance at the time of the issue.

Sep 7 20:31:15 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [0] No device attached.
Sep 8 22:00:03 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1222] /export/ACSService/gaxvelcxt01 is contained in a pool that is unusable by this NDMP server
Sep 8 22:00:03 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1222] err: 9
Sep 8 22:00:10 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1223] /export/app/cbisd is contained in a pool that is unusable by this NDMP server
Sep 8 22:00:10 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1223] err: 9
Sep 8 22:00:10 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1224] /export/app/demd is contained in a pool that is unusable by this NDMP server
Sep 8 22:00:10 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1224] err: 9
Sep 8 22:00:11 Testsn01 /usr/lib/ndmp/ndmpd[1966]: [ID 828132 daemon.error] [1225] /export/app/dems is contained in a pool that is unusable by this NDMP server

Cause

The cause for this issue is "elig-resources/zpools" NDMP property being empty.

There is an in-memory resource database which gets populated from the "elig-resources/zpools" NDMP property.

"elig-resources/zpools" NDMP property can have three possible values as follows:

not set - all pools are eligible for NDMP backups/restores
empty string ("") - no pool is eligible
<list of strings> - every pool in the list is eligible for NDMP

On an appliance, this property is set by akd during pool import/export, which usually happen when: (1) akd comes up or goes down, (2) a cluster takeover or failback occurs.

Possible reasons why the value of this property is empty are as follows:

After the upgrade the akd restart was not done correctly.
A system configuration backup ( (Maintenance > system > backup)) was performed on the storage head before doing the NDMP backup. Each time config backups is run, akd needs to be restarted to avoid ndmp backup failures.

INTERNAL UPDATE

There is a bug in firmware which will clear the above property when backup is ran. This is being worked via bug 21219986.

There is existed OOB bug 23488087 from 21219986. OOB 23488087 is fixed in version ZFS 2013.1.6.0 which will be fixed in Oct PSU 2016.

Solution

To resolve this issue restart the appliance management interface from the CLI using below command.

maintenance system restart

Above command will only restart akd, not the entire ZFS head.

References

<BUG:21219986> - NDMP BACKUP FAILURE AFTER CONFIG BACKUPS IS RUN
<NOTE:1268557.1> - Exalogic Elastic Cloud Software Known Issues

Attachments

This solution has no attachment