Sun Storage 7000 Unified Storage System: Database Error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION

Asset ID:	1-72-1900634.1
Update Date:	2018-05-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1900634.1 : Sun Storage 7000 Unified Storage System: Database Error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION

Applies to:

Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

7x20 zfs appliance errors:

May 14 21:01:46 NAS-zfs tmpfs: [ID 518458 kern.warning] WARNING: /system/volatile: File system full, swap space limit exceeded
May 14 21:01:46 NAS-zfs idmap[19046]: [ID 469902 daemon.error] Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION; (database is full)
May 14 21:01:46 NAS-zfs idmap[19046]: [ID 469902 daemon.error] Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION; (cannot start a transaction within a transaction)
May 14 21:07:40 NAS-zfs last message repeated 44 times
May 14 21:08:58 NAS-zfs idmap[19046]: [ID 469902 daemon.error] Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION; (cannot start a transaction within a transaction)

May 14 23:26:26 NAS-zfs svc.startd[9241]: [ID 462725 daemon.warning] svc:/network/nfs/status:default: Couldn't fork to execute method exec /usr/lib/ak/svc/method/nfs-status start: Not enough space
May 14 23:26:26 NAS-zfs svc.startd[9241]: [ID 748625 daemon.error] network/nfs/status:default failed: transitioned to maintenance (see 'svcs-xv' for details)
May 14 23:27:07 NAS-zfs idmap[19046]: [ID 469902 daemon.error] Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION; (cannot start a transaction within a transaction)
May 14 23:27:43 NAS-zfs last message repeated 5 times
May 14 23:27:45 NAS-zfs genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 19401 (devchassisd)
May 14 23:27:45 NAS-zfs genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 4331 (svc-fmd)
May 14 23:27:48 NAS-zfs idmap[19046]: [ID 469902 daemon.error] Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION; (cannot start a transaction within a transaction)
May 14 23:28:24 NAS-zfs svc.startd[9241]: [ID 636263 daemon.warning] svc:/system/fmd:default: Method "exec /lib/svc/method/svc-fmd" failed due to signal SEGV.

Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION;

The first error the array reports in this situation is system/volatile is full. Subsequent errors state the array cannot start a transaction within a transaction.

The BUI can seem slow or hung, and the system generally is very slow. This can also lead to problems with the NFS services.

NOTE: See also Doc ID 2188749.1 (Oracle ZFS Storage Appliance: Idmap database errors and smb_kdoor failures reported in the debug logs)

Cause

This is the critical error here...

May 14 21:26:11 zfsapp-a1 tmpfs: [ID 518458 kern.warning] WARNING: /system/volatile: File system full, swap space limit exceeded
May 14 21:26:16 zfsapp-a1 idmap[19046]: [ID 469902 daemon.error] Database error on /system/volatile/idmap/idmap.db while executing BEGIN TRANSACTION; (cannot start a transaction within a transaction)

/status$ more df.out
Filesystem                                              Size      Used    Available   Capacity Mounted on
system/ak-nas-2013.06.05.1.9_1-1.1/root 457G     1.5G    319G      1%         /
/devices                                                 0K        0K       0K            0%        /devices
/dev                                                     0K        0K       0K            0%        /dev
ctfs                                                     0K        0K       0K            0%       /system/contract
proc                                                     0K        0K       0K            0%       /proc
mnttab                                                   0K        0K       0K            0%      /etc/mnttab
swap                                                     4.8G      3.0M     4.8G        1%      /system/volatile

While a recent Bundle output does not match the issue - "swap full" issues are often very short lasting, making data collection on this issue difficult

Review the following: Bug 18430463 - /system/volatile: File system full caused akd core dump and idmap issue

... Closed as a Duplicate of Bug 18488534 - swap allocation fails too soon

Please try to collect following data 1) vmstat -S 2 10 > /var/ak/dropbox/vmstat.out 2) echo "::stacks -c swap_putpage" | mdb -k > /var/ak/dropbox/swap_putpage.out 3) echo "::memstat" | mdb -k > /var/ak/dropbox/memstat.out 4) echo "::swapinfo" | mdb -k > /var/ak/dropbox/swapinfo.out 5) generate livecore with `savecore -L`

If the system is inaccessible, then the NMI is the way to collect the data.

Solution

The customer should attempt to 'flush' their idmap configuration there in the first instance.

Go to:
https://<ArrayIP address>:215/#configuration/services=idmap then see 'Mappings' There you will see a button marked 'flush'

((This operation advises you that the identity Mapping and SMB services will restart.) Your Business constraints permitting of course).

No further errors reported.

***Checked for relevance on 25-MAY-2018***

References

<BUG:18430463> - /SYSTEM/VOLATILE: FILE SYSTEM FULL CAUSED AKD CORE DUMP AND IDMAP ISSUE
<NOTE:1481997.1> - Sun Storage 7000 Unified Storage System: svc.configd crashes, /var/run is full
<BUG:18657573> - DATABASE ERROR ON /SYSTEM/VOLATILE/IDMAP/IDMAP.DB WHILE EXECUTING UPDATE IDMAP_C
<BUG:15607918> - SUNBT6907210 SEQUENCE OF MAP/ADD/MAP/REMOVE/MAP... LEAVES CACHE IN BAD STATE
<BUG:18488534> - SWAP ALLOCATION FAILS TOO SOON

Attachments

This solution has no attachment