Zpool Import or Mount of a ZFS File System might Fail with "mountpoint or dataset is busy" Message

Asset ID:	1-71-1020298.1
Update Date:	2018-01-08
Keywords:

Solution Type Technical Instruction Sure

Solution 1020298.1 : Zpool Import or Mount of a ZFS File System might Fail with "mountpoint or dataset is busy" Message

Applies to:

Solaris Operating System - Version 10 6/06 U2 and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
All Platforms

Goal

This document provides information how to check which process(es) blocks zpool import/export or zfs mount/umount commands

Symptoms:

svc:/system/filesystem/local fails to online with "zfs mount -a" errors
cannot import or export zpool

Solution

How to identify which process is using a zpool or zfs filesystem if a "zpool import" or a "zfs mount" fails with "mountpoint or dataset is busy" message:

You can identify the process by fuser command. Once you identified the process and user, you might consider stop or kill it so that you can import the zpool or mount the zfs file system..

In following example the process (pid:2022) is using the mount point.

# zpool import tank
cannot mount 'tank': mountpoint or dataset is busy

Use the fuser command to check which process keeps the mountpoint of 'tank' dataset busy

# fuser -fu /tank
/tank: 2022c(root)

fuser(1M) show to interpret the letter code ( 'c' in the fuser output above )

   c    Indicates that the process is using the file as its current directory.
   m    Indicates that the process is using a file mapped with mmap(2). See mmap(2) for details.
   n    Indicates that the process is holding a nonblocking mandatory lock on the file.
   o    Indicates that the process is using the file as an open file.
   r    Indicates that the process is using the file as its root directory.
   t    Indicates that the process is using the file as its text file.
   y    Indicates that the process is using the file as its controlling terminal.

In the example above, we can see that the process 2022 is using the directory for the dataset 'tank' as its current directory.

How to identify which process is using a zpool or zfs filesystem if a "zpool export" or a "zfs unmount" fails with "device busy" message:

In the following example, a process (pid:2016) is using the zpool.

# zpool export tank
cannot unmount '/tank/home': Device busy

# fuser -f u /tank/home/tom
/tank/home/tom:     2016c(root)

-- or --

# fuser -cu /tank/home
/tank/home:     2016c(root)

Note: It highly depends on the system and application whether certain processes can be killed (terminated) to free a zpool or zfs dataset
So, it's up to the system administrator to decide which of the following actions is most appropriate:
A) kill (terminate) a process
B) gracefully reboot the system to terminate the blocking processes
C) use commands as shown below to force export zpool or zfs unmount ( can be dangerous )

You can export zpool or unmount zfs forcibly as follows.
# zpool export tank
cannot unmount '/tank/home': Device busy

# zpool export -f tank

# zfs unmount tank/home
cannot unmount '/tank/home': Device busy

# zfs unmount -f tank/home
#

Another possible way to kill the processes that hold a certain mountpoint is using "fuser -fn -s term"
The following command sends SIGTERM to any processes that hold a non-blocking mandatory lock on file /export/foo/my_file.

# fuser -fn -s term /export/foo/my_file

For further info please see fuser(1M)

Product
Solaris 10 Operating System
Solaris 11 Operating System

This document contains normalized content and is managed by the the Domain Lead(s) of the respective domains. To notify content owners of a knowledge gap contained in this document, and/or prior to updating this document, please contact the domain engineers that are managing this document via the "Document Feedback" alias(es) listed below: Solaris OS Domain Feedback Alias : Solaris-ZFS-Normalization-SMEs@sun.com

zfs, normalized, zpool mount, zpool unmount, zpool import, zpool export, device is busy

References

<NOTE:1507489.1> - Troubleshooting zpool import failure
<NOTE:1506251.1> - WARNING: /usr/sbin/zfs mount -a failed: one or more file systems failed to mount
<NOTE:1911523.1> - beadm destroy fails with "/.../var: mountpoint or dataset is busy"

Attachments

This solution has no attachment