Sun ZFS Storage 7420
 
Sun Storage 7110 Unified Storage System
 
Sun Storage 7210 Unified Storage System
 
Sun Storage 7410 Unified Storage System
 
Sun Storage 7310 Unified Storage System
 
Sun ZFS Storage 7120
 
Sun ZFS Storage 7320
 
Oracle ZFS Storage ZS3-2
 

PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
 

That document intends to consider how much faster is L2ARC against disks. And how much faster is ARC against L2ARC.
ZFS performance is directly related to SMB, NFS and ISCSI performance.
It makes sense to get rough estimates of access times in order to correctly configure an appliance and/or to add more trays, more SSDs, more RAM on installed configurations.

In this Document

Purpose

Scope

Details

References

Applies to:

Purpose

That document intends to consider how much faster is L2ARC against disks.  And how much faster is ARC against L2ARC.

Scope

This document applies to the ZFSSA appliance where ARC is configured by default to 98% of the RAM, and where L2ARC .i.e. SSDs can be installed.

Details

Let us first have a short reminder of how the ARC and L2ARC work :

The purpose of the ARC (Adaptive Replacement Cache) is to have an efficient list of buffers handled to improve ZFS performance. At any given moment, some subset of the blocks in the cache are un-evictable because we have handed out a reference to them. Blocks are only evictable when there are no active external references. The mechanism of eviction is critical for the ARC to work efficiently. We choose to evict the evictable blocks that are “lowest” in the list.

The L2ARC is best pictured as a cache layer in-between main memory and disk, using flash memory based SSDs or other fast devices as storage. It holds non-dirty ZFS data, and is intended to improve the performance of random read workloads. L2ARC is physically sustained by cache devices added to the pool. When we add cache disks to a pool, we allow L2 cache to be used.

Let's keep in mind that READ requests are satisfied from the following sources, in order:

1) ARC    3) L2ARC devices    5) disks

Comparing DISKS and SSDS access times

Brendan Gregg (2008) shows some rough estimates of SSD enhancement against disks in his blog : https://blogs.oracle.com/brendan/entry/test

Service times are said to be 20 times faster for SSDs against disks.

Things have not changed much since Brendan Gregg wrote the above text.

Here are the new rough estimates of access times on SSDs, and on disks:

        Disks : 1 ms = 10^(-3) s

        SSD : 0.05 ms = 5.10^(-4) s

In short, reading a data on SSDs is 20 times faster than reading it on disks.

Comparing ARC and L2ARC access times

Now to compare access times on ARC against L2ARC, we have to take into account that data and metadata in the ARC and L2headers in the ARC don't take up the same space.

When one data is referenced in the ARC, we could reference many more L2 headers in the same space in memory.

L2 headers have a fixed size : 192 bytes. Data or metadata in the ARC do not have fixed sizes. Their size vary from 4KB (the size of a page on x86 boxes) up to large buffers for metadata.

But the average block size is always between 4KB and 128KB, since the ZFSSA appliance contains huge pools of data set up with block sizes of 4KB to 128KB. Data occupies much more ARC space than metadata.

Therefore, to compare ARC and L2ARC access times, we have to consider that when one block fills the ARC, around 20 L2 block-headers could fill the same space in the ARC (for a system with an average block size around 4kB: 4kB/192~20). Around 700 block-headers could fill the same space , for a system with an average block size around 128kB.

 The calculation of the average block size is available in Document 1573028.1 How to check how much memory the L2ARC table is occupying in the ARC.

Here is a picture to explain the reasoning above :

To calculate ARC and L2ARC access times, we need rough estimates of access times on SSDs and DRAM :

    DRAM : 10 ns = 10^(-8) s
    SSD : 0.05 ms = 5.10^(-4) s

Reading a block in L2ARC consists of reading the header in the ARC, then reading the block itself in L2ARC. So reading a block in L2ARC takes 10ns + 0.05 ms ~ 0.05 ms.

As already stated, we also have to consider that when one block fills the ARC,  20 to 700 L2 block-headers could fill the same space in the ARC, depending on the average block size:

    With an average block size of 4KB: (5.10^(-4) + 10^(-8))/ 20 x  10^(-8) = 2500  

    With an average block size of 128KB: (5.10^(-4) + 10^(-8))/ 700 x  10^(-8) ~ 100    

So reading blocks in the ARC instead of L2ARC :

    ==> is 2500 times faster with an average block size around 4kB

    ==> is only 100 times faster with an average block size around 128KB

In short, ARC is 100 to 2500 times faster than L2ARC, depending on the average block size on the ZFSSA.

Reading data from the ARC is also 10000 times faster than reading it from disk.

References

This solution has no attachment