Sun Storage 7000 Unified Storage System: Sufficient installed RAM is required for efficient use/performance of installed Read SSDs (readzilla)

Asset ID:	1-72-1532289.1
Update Date:	2018-05-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1532289.1 : Sun Storage 7000 Unified Storage System: Sufficient installed RAM is required for efficient use/performance of installed Read SSDs (readzilla)

Applies to:

Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Oracle Exalogic Elastic Cloud Software - Version 2.0.3.0.7 to 2.0.3.0.7
7000 Appliance OS (Fishworks)

Symptoms

Customer is losing FC connection frequently in the storage appliance 7320 (not cluster).
Only FC lun service is been used. Performance issue ?

System configuration is: 7320 + 1 JBOD (1T drives), 24G RAM, 2 x readzillas/logzillas

FC configuration looks OK (Direct Attach - No SAN switches):

Analytics before the reboot shows L2ARC headers using >12G of the available memory.
Drives >95% utilization is also very high. Cache hit rate of only 37%.
Read to write ratio: 70% reads/30% writes.

(mdb -k) '::memstat' output shows a significant amount of memory is being used
    by the kernel (but, almost no memory used for ZFS file data!)

    Page Summary                Pages                MB %Tot
    ------------     ---------------- ---------------- ----
    Kernel                    5408133             21125   86%    <<<<<<<<
    ZFS File Data               16324                63    0%    <<<<<<<<
    Anon                       194203               758    3%
    Exec and libs                2895                11    0%
    Page cache                   9546                37    0%
    Free (cachelist)             2745                10    0%
    Free (freelist)            653282              2551   10%

    Total                     6287128             24559
    Physical                  6287127             24559    <<<< Installed RAM = 24Gb

    ( NOTE: The L2ARC header (metadata) will be counted in the 'Kernel' total )

(mdb -k) '::arc' output shows a large 'l2_hdr_size' (L2ARC 'metadata') value:

    hits                      = 446299175
    misses                    = 378508831
    ........
    hdr_size                  = 12061616
    data_size                 = 58714112
    other_size                = 16415360
    ........

    l2_read_bytes             = 1602838355968
    l2_write_bytes            = 656024780288
    ........
    l2_size                   = 603288616448
    l2_hdr_size               = 13899123456        <<<<<<<< ~13.8 Gbytes
    memory_throttle_count     =         0
    arc_no_grow               =         1
    arc_tempreserve           =         0 MB
    arc_meta_used             =     12202 MB
    arc_meta_limit            =     23535 MB
    arc_meta_max              =     12224 MB

=> The L2 header size tells you how much RAM is being used to keep track of the data in the readzillas

(mdb -k) ::kmastat -g ! egrep -v ' 0G '

    > ::kmastat -g ! egrep -v '0G'
    > cache                        buf    buf      buf     memory     alloc alloc
    > name                        size   in use    total     in use   succeed fail
    > ------------------------- ------   ------   ------ ---------- --------- -----
    > kmem_va_4096                4096 4394009 4394624        16G 16362346     0    <<<<
    > kmem_va_16384              16384     1915   142576         2G 20233883     0
    > kmem_alloc_16                 16 72448579 72449644         1G 213045124    0
    > kmem_alloc_32                 32 72424573 72435250         2G 380430241    0
    > zfs_file_data_8192          8192      289 1804080        13G 117162864     0    <<<<
    > arc_buf_hdr_t                176 72454537 72463864        12G 339669130    0    <---
    > ------------------------- ------   ------   ------ ---------- --------- -----
    > Total [kmem_va]                                           19G   44389986    0
    > Total [kmem_default]                                      16G 3205801139    0
    > Total [zfs_file_data]                                     13G 118289027    0
    > ------------------------- ------   ------   ------ ---------- --------- -----
    >
    > vmem                         memory     memory    memory     alloc alloc
    > name                         in use      total    import   succeed fail
    > ------------------------- ---------- ----------- ---------- --------- -----
    >     kmem_firewall_va              1G          1G         1G    336457     0
    >         kmem_oversize             1G          1G         1G    336457     0
    >     kmem_va                      19G         19G        19G 12259587     0
    >         kmem_default             16G         16G        16G 51103372     0
    > ------------------------- ---------- ----------- ---------- --------- -----

    => The 'volblocksize' of all LUNs = 8Kb     (buffers = zfs_file_data_8192)

    => The memory heap is fragmented between the 8k buffers and the 4k metadata buffers

Summary
=======
L2 header size    = 13.8 Gb

Physical RAM size = 24 Gb

Cause

The L2ARC headers are using greater than 50% of the installed RAM.

When read SSDs (readzillas) are installed, RAM (memory space) is required to keep track of the data in the readzillas.

This array is not equipped with sufficient RAM to utilize the readzillas.

Current recommendations/best practice is maximize the installed RAM BEFORE installing readzillas.

They are using too much RAM, for little overall benefit.

Solution

Either: Add more RAM (Recommended: Maximum -> installable RAM. Minimum -> 64Gb for 'production' systems ?)

Or: Remove the readzillas from the system (for 'better' RAM usage)

***Checked for relevance on 25-MAY-2018***

References

<NOTE:1331769.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Performance Issues
<NOTE:1333120.1> - Sun Storage 7000 Unified Storage System: How to add L2ARC cache SSDs (Readzillas) to a pool

Attachments

This solution has no attachment