Sun ZFS Storage 7120
 

PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
 

It has been reported in some situations that ZFS Storage Appliances configured with 24GB of RAM display some instability after they have been updated from 2011 code to 2013.1.x (OS8) code.

Applies to:

Symptoms

It has been reported in some situations that ZFS Storage Appliances configured with 24GB of RAM display some instability after they have been updated from 2011 code to OS8 code basis.

Systems configured with 24GB are also referred to as 'low memory configurations' and after updating the appliance kit software from a 2011 code release to an 2013.1.x (OS8) release, the additional new features in that new major release require more memory to be allocated and could lead into a situation where the appliance might stop providing service to clients and appear hung due to kernel memory exhaustion.

The affected system might run stable for days or even weeks and then stop, there is no sign of broken hardware or other defects on the system and a reboot clears the situation. After about the same time, the appliance again stops working and a reboot clears the immediate situation.

Before the software update, the ZFS Storage Appliance has been working reliably and been stable for a reasonable time and since the update it seems stop working on a certain pattern.

Cause

The new software release is a major release and with it there are new features available to the system.

These new software features might consume more memory and along with the demand of ZFS for memory the system could come into a state when all memory has been allocated and now new memory allocations - required for system functions - cannot be allocated.

At his point, the kernel of the appliance seems to stop working, while in fact it is struggling with memory allocation.

You may be impacted by Bug 19591405 (akd running in RT class is blocking critical memory management threads) - please Document ID 2018298.1

Bug 20648579 - comment:

 > Memory footprint is expected to grow even between minor releases (i.e. 2013.1.2=>2013.1.3) since new functionality is added to akd or existing functionality is enhanced, modifying data structures, hence memory allocations.

 > This increase is specially significant between major releases, as this is the case: 2011.1.9 => 2013.1.2


Bug 20922619 - comment:

 > Now, from 2011.1 to 2013.1 the akd memory footprint increased around 100M, give or take (Bug 20648579) more memory usage is proportional with features enabled, therefore moving the appliance to a state of memory starvation.

 > (while in 2011.1 there was 2%-3% free memory or 200M-300M).

 >I would suggest tuning ARC (capping) to 14848MB, leaving 1GB cushion.

Solution

To determine the exact situation it will require a kernel crash dump, which will most likely be collected by issuing a NMI.  Please engage Storage-TSC NAS Support for assistance.

Resolution options include :

Change the akd threads to use the Time Sharing scheduling class instead of the Real Time Scheduling class

In order to ease the memory reorganization and let the pageout and the kmem_cache_reap threads do their job, make the non cluster related threads
of the akd daemon run the Time Sharing scheduling class.

Use the akd_RT.akwf workflow available from bug 19591405 - switch off RT scheduling.
Select "Disable RT". Must be applied on both cluster nodes. Note that akd will be restarted, so make sure akd is back up and running
on one node before applying the same script onto the second node

Limit the ZFS ARC (Adaptive Replacement Cache)

Once the situation has been confirmed as Low Memory Configuration and Kernel running out of memory it may help to limit the ARC.
Such a step will leave the kernel with more memory to run and in conjunction will limit ZFS in allocating memory.
This might have some impact on the system performance.
Please engage Storage-TSC NAS Support for assistance in limiting the ARC size.

Use the zfs_arc_max_tune_ak8.akwf workflow on both nodes. It does not restart akd and does not require a reboot. Select 80% or even less. Please contact
a senior engineer to set the correct value.

Rollback to the previous code release

Another solution might be to rollback to the previous code release, if this is still on the system.
This works best if Deferred Updates have not been applied, else it will require to recreate the data pool and restore the data to the new pool from backup

Install additional RAM

An additional solution would be to install additional RAM. This will help to overcome situations where the system is running low on memory and the kernel is having difficulties allocating memory.

As the code is getting constantly improved, the handling of memory allocation and consumption may become optimized - so that no workaround or hardware update will be required.

***Checked for relevance on 30-MAY-2018***

References

This solution has no attachment