Oracle ZFS Storage Appliance: How To Recover From A Non-Booting ZFS-SA with "Failed To Write To /etc/svc/repository.db At Offset XX: No Space" Error

Asset ID:	1-72-2196434.1
Update Date:	2016-11-02
Keywords:

Solution Type Problem Resolution Sure

Solution 2196434.1 : Oracle ZFS Storage Appliance: How To Recover From A Non-Booting ZFS-SA with "Failed To Write To /etc/svc/repository.db At Offset XX: No Space" Error

Related Items


Sun ZFS Storage 7320
 Sun Storage 7210 Unified Storage System
 Oracle ZFS Storage ZS3-BA
 Oracle ZFS Storage ZS5-4
 Oracle ZFS Storage ZS3-2
 Sun Storage 7410 Unified Storage System
 Oracle ZFS Storage ZS3-4
 Sun ZFS Storage 7420
 Oracle ZFS Storage ZS5-2
 Sun Storage 7310 Unified Storage System
 Oracle ZFS Storage ZS4-4
 Sun Storage 7110 Unified Storage System
 Sun ZFS Storage 7120

Related Categories


PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS

In this Document

Symptoms

Cause

Solution

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: contains command line references
Created from <SR 3-13479167561>

Applies to:

Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

Error message: svc.configd: Fatal error: Backend copy failed: fails to write to /etc/svc/repository.db at offset 917504: No space left on device

Cause

The meaning of the error is that the system pool has no space left . Normally this means the /tmp filesystem is full.

Solution

If this is a temporary situation, then a reboot will clear /tmp and /swap.

If it does not, then a shared shell will be needed to boot to 'milestone none' (if the appliance does not drop to the Emergency shell first), so we can manually remove old data.

With this example, I found that the system pool had little free space and dropped to the emergency shell on logging in :

$ df -hal Filesystem size used avail capacity

Mounted on system/ak-nas-2011.04.24.9.2_1-1.50/root 457G 1.3G 765K 100% / <<<<<<< does not add up !

I cleared some old updates - as they had a large amount of space taken up, presumably by old Analytics - and this allowed AKD to start.

I then cleared up the Analytics in the current version

        # df -h | sed -ne 1p -e /stash/p
        Filesystem                                                                                size      used    avail capacity Mounted on
        system/ak-nas-2011.04.24.9.2_1-1.50/running/stash    457G    134G    26G     84%     /var/ak/stash

This then left enough space to get AKD running more reliable

# zfs list

NAME USED AVAIL REFER MOUNTPOINT
system 368G 88.8G 77.5K legacy

So basically old updates with unusually large amounts of analytics was the issue here.

Attachments

This solution has no attachment