Sun Flash F20 PCIe Card
 
Solaris Operating System
 
F40 Flash Accelerator Card
 
Flash Accelerator F80 PCIe Card
 
Sun Flash F5100 Array
 

PLA-Support>Sun Systems>SAND>Kernel>SN-SND: Sun Kernel ZFS
 

In this Document

Symptoms

Cause

Solution

References

Applies to:

Symptoms

Newer storage devices, particularly Solid State Disks (SSD), Non-Volatile Memory on PCI (NVME), and Flash Module (FMOD) HBAs, are being released with increasing native physical-block-sizes.  Traditional spinning disks have a physical-block-size of 512 bytes.  SSD, NVME, and FMODS typically have 4K or 8K native block sizes with backwards compatability to 512e (512 byte emulated).  For optimal performance the native physical-block-size should be used.  This avoids multiple Read-Modify-Write (RMW) operations within the device itself that can lead to premature wear failures and mis-aligned IOs.  Usually this requires the 'physical-block-size' parameter to be added to the driver configuration file.  Refer to the product documentation for the recommended settings.  For example:

For F40 or F80 flash devices, add the following entry to /kernel/drv/sd.conf. In the entry below, ensure that "ATA " is padded to 8 characters. String length of  "ATA     2E256" is 13, and string length of "ATA     3E128" is also 13. Note that this should be a single line in sd.conf. 

ZFS uses the physical-block-size to calculate the appropriate value for its 'ashift' property.  The 'ashift' property ensures the IO is correctly aligned for optimal performance.

To verify the physical-block-size of 'sd' devices, use:

For ssd devices use: 

To verify the ZFS ashift value, choose one of the Virtual Devices (VDEVs) within the pool, then use zdb to print the label and grep for 'ashift':

Creating a RAIDZ1 (7 Data + 1 Parity) Pool consisting 16 devices (Oracle F80 HBAs in this case) with various physical-block-sizes/ashift values results in the following 'loss' of usable capacity as the physical-block-size is increased from 512e, 4K, to 8K (native):

Using the same devices to create a new pool without RAID while varying the physical-block-size/ashift values results in the following: 

Cause

ZFS is both a volume/pool manager and a filesystem.  ZFS does not pre-allocate anything when the pool and datasets are created.  Instead, ZFS creates the required metadata when write IOs occur.  ZFS also supports variable stripe and raid widths, chosen as data is written out.  Because of this, the initial calculation during pool creation results in the loss of usable space.

The key function is:

static uint64_t
vdev_raidz_asize(vdev_t *vd, uint64_t psize, dva_layout_t layout, int copies)
{
	uint64_t asize;
	uint64_t ashift = vd->vdev_top->vdev_ashift;
	uint64_t cols = vd->vdev_children;
	uint64_t nparity = vd->vdev_nparity;

	asize = ((psize - 1) >> ashift) + 1;

	if (layout == DVA_LAYOUT_RAIDZ_MIRROR) {
		ASSERT(copies > nparity);
		asize *= copies;
	} else {
		ASSERT(layout == DVA_LAYOUT_STANDARD);
		asize += nparity *
		    ((asize + cols - nparity - 1) / (cols - nparity));
	}

	asize = roundup(asize, nparity + 1) << ashift;

	return (asize);
}

 
vdev_raidz_asize() is used from vdev_metaslab_init(vdev_t *vd, uint64_t txg)

To compute the vdev_deflate_ratio

	vd->vdev_deflate_ratio = (1 << 17) /
	    (vdev_psize_to_asize(vd, 1 << 17, DVA_LAYOUT_STANDARD, 1) >>
	    SPA_MINBLOCKSHIFT);

We're asking the system how much deflation we have to apply to physical space when storing 128K (near best case).  If smaller ZFS recordsizes are used, the free space will drop much faster.

spa_update_space(spa_t *spa) also taxes this with a fudge factor of 1.6%.

Going through the computation in vdev_raidz with cols=8, nparity=1, the numbers posted by 'zpool list' and 'zfs list' match.

Solution

No solution available.  ZFS uses the available space within the pool intelligently and makes the initial calculations based on pool configuration and predicted future use. The initial calculations may eventually become over, under, or a very good estimate. 

Reducing the physical-block-size in the driver config file will permit ZFS pools to be created with more usable capacity.

See RFE 25304772 - "Raidz wastes more space than required for padding" that will help but won't fully resolve the issue.

References

This solution has no attachment