Oracle ZFS Storage Appliance: Writes to an iSCSI or FC LUN fail with medium errors, filesystem becomes read-only

Asset ID:	1-72-1469814.1
Update Date:	2018-01-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1469814.1 : Oracle ZFS Storage Appliance: Writes to an iSCSI or FC LUN fail with medium errors, filesystem becomes read-only

Applies to:

Oracle ZFS Storage ZS3-BA - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS4-4 - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

On an iSCSI or FC client, a filesystem residing on a ZFSSA LUN has become read-only.

The client system log reports write failures due to "Medium Errors".

Example - on a Linux client:

    end_request: I/O error, dev sdc, sector 23688554850
    sd 11:0:0:0: SCSI error: return code = 0x08000002
    sdc: Current: sense key: Medium Error
    Add. Sense: Write error

Example - on a Solaris client:

    Dec 7 15:48:49 solclient scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g600144f0912b34c0000056659a210007 (ssd2):
    Dec 7 15:48:49 solclient        Error for Command: write(10)               Error Level: Fatal
    Dec 7 15:48:49 solclient scsi: [ID 107833 kern.notice] Requested Block: 532480                    Error Block: 532480
    Dec 7 15:48:49 solclient scsi: [ID 107833 kern.notice] Vendor: SUN                                Serial Number:
    Dec 7 15:48:49 solclient scsi: [ID 107833 kern.notice] Sense Key: Media Error
    Dec 7 15:48:49 solclient scsi: [ID 107833 kern.notice] ASC: 0xc (write error), ASCQ: 0x0, FRU: 0x0

After running a filesystem check (fsck, e2fsck), the filesystem can be mounted read/write. But when resuming write operations, it goes back to read-only.

Cause

The ZFSSA zpool which contains the LUN is full. It has no more space to write new data.

It is possible to completely fill up the underlying zpool of the ZFSSA by over-allocating the space when creating thin-provisioned (sparse) LUNs, and/or by leaving insufficient free space in the pool to account for LUN metadata and structural constraints.

See Doc ID 1914108.1 - LUN uses more Space than the Created Size.

zpool space usage

The left-hand side of the Shares page in the BUI provides the zpool Usage. The Shares->Projects page details the usage for each project.

In the CLI, the zpool usage appears at the top of the "status show" output.

The "Used" and "Available" fields count only blocks which have actually been written to.
The "Free" field truly indicates how much unreserved space the zpool has left for metadata and for writing to thin-provisioned LUNs.

Example:

    cli> status show
    Storage:
       Cesspool:
          Used        10.2G bytes
          Available   9.71G bytes           <== a lot space is "reserved" but not yet used
          Free            0 bytes           <== there is no more free (unreserved) space in the zpool
          State             online
          Compression:   1x
          Dedup:         1x

Project and share space usage

The use of quotas and reservations may result in a different amount of available space for each project or share within the same zpool.

To display how much space is currently available to a specific share for writing data, or to a specific project for creating new shares, use:

shares select <project> select <lun> get space_available
shares select <project> get space_available

Example:

No new space available for the "default" project

cli> shares select default get space_available
space_available = 0

within the "default" project, the shares have used up more or less of their allocated space

    cli> shares select default show
    ..
    LUNs:

    NAME              VOLSIZE ENCRYPTED     GUID
    lun1              7G     off           600144F0912B34C00000566599400005
    lun2              7G     off           600144F0912B34C00000566599EF0006
    lun3              300M   off           600144F0912B34C0000056659A210007

    cli> shares select default select lun1 get space_available
                   space_available = 39.1M

    cli> shares select default select lun2 get space_available
                   space_available = 7.00G

    cli> shares select default select lun3 get space_available
                  space_available = 0

In the bundle or the shell, see the output of "zfs list".

Example:

NAME                                                      USED    AVAIL REFER MOUNTPOINT
Cesspool                                                  14.6G      0 44.9K none
Cesspool/local                                            14.6G      0 44.9K none
Cesspool/local/default                                    14.6G      0 44.9K /export
Cesspool/local/default/fs1                                 396M      0   396M /export/fs1
Cesspool/local/default/lun1                                  7G 39.1M 6.96G -
Cesspool/local/default/lun2                                  7G 7.00G 63.6K -
Cesspool/local/default/lun3                                260M      0   260M -

Solution

The only solution is to free up space in the zpool. This can be achieved by:

1 - Releasing unwritten LUN space by temporarily making one or more LUNs "thin provisioned" (sparse).
    Unwritten LUN space is displayed by the above "get space_available" command on selected LUNs.
    It is unrelated to disk usage reported on the client. Deleting LUN data from the client has no impact on zpool or share space usage.

    In our example, only lun2 has a significant amount of unwritten reserved space that could be temporarily used to remedy the full zpool.

2 - Deleting LUNs or filesystems.
    If no space can otherwise be freed to make the zpool functional, this is the last resort.

If the LUNs cannot be mounted and no recent backups exist, and if the customer has alternate storage space,
it may be necessary to evacuate the data from one LUN before deleting it to rescue the others.
See How to evacuate Data from LUNs on the AmberRoadSupport wiki.

Once the pool has enough free space to function correctly, a longer term solution must be put in place to prevent the pool from filling up again.
When recreating the LUNs, refer to Doc ID 1995759.1 - How much free space to leave when configuring a pool with LUNs.

Attachments

This solution has no attachment