Oracle SuperCluster M6-32 Hardware
 
Oracle SuperCluster T5-8 Hardware
 
Oracle SuperCluster M7 Hardware
 
Oracle SuperCluster Specific Software
 
Oracle SuperCluster T5-8 Half Rack
 
SPARC SuperCluster T4-4
 
Oracle SuperCluster T5-8 Full Rack
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 

In this Document

Symptoms

Cause

Solution

References

Applies to:

Symptoms

SuperCluster systems installed with the Oct QFSDP (or later) or SuperCluster Install Bundle 2.4.16 (or later 2.4.x builds), will have ZFS Storage Appliance software version of AK 2013.1.8.7 (or later) installed.
AK 2013.1.8.7 software will require v1.0.5 of the ZFS SA NFS Plugin installed for use with Solaris Cluster 3.3_u2. Version 1.0.5 of the ZFS SA NFS Plugin in turn has a hard dependency on Java 1.7.0 Update 151 (or later Java 7 updates), as well as ensuring that Java 7 Update 151 (or a later Java 7 Updates) is the default system-wide version of Java.

While it is possible for multiple versions of Java to co-exist on a single system, only one version can be the default system-wide version (i.e. the java instance symlinked to /usr/java). With this in mind, please be aware of the need to make Java 7 Update 151 (or a later Java 7 update) the default system-wide version of Java used. Put another way, if an application running under Solaris Cluster 3.3_2 requires a default system-wide version of Java 7 that is less than Update 151, or any version of Java 5, Java 6 or Java 8 to be the default system-wide version of Java on the system, then until there is a fix for this issue the SuperCluster October QFSDP or SuperCluster Installation Build 2.4.15 (or later) will not be compatible with such configurations.

This note details how to check and install both v1.0.5 of the ZFS SA Plugin and Java 7 Update 151 for systems running Solaris Cluster 3.3_u2, as well as making Java 7 Update 151 the default system-wide version of Java on the system.

It is assumed that Solaris Cluster 3.3_u2 is running at a core patch level of at least patch 145333-37.

With and incompatible version of the ZFS SA NFS Plugin (eg. 1.0.0) installed failures will be seen when doing NFS fencing/unfencing and when cluster storage resources are attempting to be brought online. Output similar to that below will be seen due to the storage resource failing to start successfully:

# /usr/cluster/bin/cluster status

=== Cluster Resource Groups ===

Group Name   Node Name                Suspended   State
----------   ---------                ---------   -----
lh-rg        ssc1-appio0101-bzc1      No          Pending_online_blocked
             ssc1-appio0201-bzc1      No          Offline

scalmnt-rg   ssc1-appio0101-bzc1      No          Online_faulted
             ssc1-appio0201-bzc1      No          Online_faulted


=== Cluster Resources ===

Resource Name       Node Name             State          Status Message
-------------       ---------             -----          --------------
apache-rs           ssc1-appio0101-bzc1   Offline        Offline
                    ssc1-appio0201-bzc1   Offline        Offline

lh-rs               ssc1-appio0101-bzc1   Online         Online - LogicalHostname online.
                    ssc1-appio0201-bzc1   Offline        Offline

apache-scalmnt-rs   ssc1-appio0101-bzc1   Start_failed   Faulted
                    ssc1-appio0201-bzc1   Start_failed   Faulted

When this occurs in a branded zone cluster, messages similar to the following can be seen in /var/adm/messages of the non-global zone:

and the exact symptom of the failure to do NFS fencing/unfencing will be reflected in messages similar to the following in global zone's /var/adm/messages file:

  
In addition, the 'clnas' Solaris Cluster command will not be able to communicate with the ZFS SA and one would see output similar to the following when attempting to run 'clnas':

Cause

The issues that are seen are a result of using a version of the ZFS SA NFS Plugin that is less than v1.0.5 with a version of the ZFS SA Software of 2013.1.7.x.

v1.0.5 of the ZFS SA NFS Plugin will require Java 7 Update 151 (or later Java 7 updates) to be installed on the target system and have that Java 7 update to be set as the system-wide default Java version.

Solution

A complete solution to remove the dependency on needing to set the default system-wide version of Java to be Java 7 update 151 (or a later Java 7 update version) is still pending.

In the interim, in order for the ZFS SA NFS Plugin to work correctly with AK 2013.1.7.8 (or later) one must have:

1. v1.0.5 of the ZFS SA NFS Plugin installed
2. Java 7 update 151 (or a later java 7 update) installed
3. Java 7 update 151 (or a later java 7 update) set as the default system-wide version of Java

The procedure to meet the minimum requirements to achieve this on the system(s) running Solaris Cluster 3.3_u2 is set out below:

1. Download and install v1.0.5 of the ZFS SA NFS Plugin.
   
   
   1. Check to see if version 1.0.5 of the SUNWsczfsnfs pkg is already installed:
      pkginfo -l SUNWsczfsnfs
   2. If not, go to http://www.oracle.com/technetwork/server-storage/sun-unified-storage/downloads/zfssa-plugins-1489830.html and download the resource listed under "Oracle ZFS Storage Appliance Network File System Plug-in for Oracle Solaris Cluster, v1.0.5 for Solaris 10", namely SunZFSNFSSC_105b1.zip and copy it to the target system.
      
      
   3. Uninstall SUNWsczfsnfs if there is a version earlier than 1.0.5 installed:
      # pkgrm SUNWsczfsnfs
   4. Unzip SunZFSNFSSC_105b1.zip to a local directory and install the v1.0.5 SUNWsczfsnfs pkg:
      
      # pwd
      /SUNWsczfsnfs
      # ls
      install pkginfo pkgmap reloc
      # pwd
      /SUNWsczfsnfs
      # cd
      # pkgadd -d .
      The following packages are available:
      1 SUNWscapc Oracle Solaris Cluster Apache Web Server Component
      (sparc) 3.3.0,REV=2010.07.26.12.56
      2 SUNWsczfsnfs Sun ZFS Network File System Plugin
      (all) 1.0.5,REV=2017.08.01.15.42.54

      Select package(s) you wish to process (or 'all' to process all packages). (default: all) [?,??,q]: 2

      Processing package instance <SUNWsczfsnfs> from </>

      Sun ZFS Network File System Plugin(all) 1.0.5,REV=2017.08.01.15.42.54
      Copyright (c) 2011, 2017, Oracle and/or its affiliates. All rights reserved.
      Using </usr> as the package base directory.
      ## Processing package information.
      ## Processing system information.
      .
      ..
      ...

   5. Verify that the pkg is installed and at the right version:
      
      # pkginfo -l SUNWsczfsnfs
      PKGINST: SUNWsczfsnfs
      NAME: Sun ZFS Network File System Plugin
      CATEGORY: application
      ARCH: all
      VERSION: 1.0.5,REV=2017.08.01.15.42.54
      BASEDIR: /usr
      VENDOR: Oracle Corporation
      DESC: Sun ZFS Network File System for Oracle Solaris Cluster Plug In
      PSTAMP: 20170801-1542.1
      INSTDATE: Nov 04 2017 23:55
      HOTLINE: Please contact your local service provider
      STATUS: completely installed
      FILES: 11 installed pathnames
      3 shared pathnames
      5 directories
      717 blocks used (approx)

      #

       
      
      
      
2. Download and install java 7 update 151.
   
   Note: You must ensure that you download the Java 7 update 80 SVR4 packages as the starting point here, so that these packages can then subsequently be updated as part of the standard QFSDP patching process for Solaris 10 instances.
   
   
   1. Check to see if any Java 7 packages are already installed:
      # pkginfo | grep SUNWj7
   2. If Java 7 packages are already installed on the system skip to section 2.5.
      If no Java 7 packages are installed, go to http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html, download the "Solaris SPARC (SVR4 Package)" via link to "jdk-7u80-solaris-sparc.tar.Z" from section "Java SE Development Kit 7u80" and copy it to a local directory on the target system.
      
      
   3. Unpack the jdk-7u80-solaris-sparc.tar.Z and install the packages it contains:
      # ls
      jdk-7u80-solaris-sparc.tar.Z
      # uncompress jdk-7u80-solaris-sparc.tar.Z
      # tar -xf jdk-7u80-solaris-sparc.tar
      # ls
      COPYRIGHT    SUNWj7dev   THIRDPARTYLICENSEREADME.txt
      LICENSE         SUNWj7jmp   jdk-7u80-solaris-sparc.tar
      README.html  SUNWj7man  SUNWj7cfg
      SUNWj7rt
      # pkgadd -d . SUNWj7cfg SUNWj7dev SUNWj7jmp SUNWj7man SUNWj7rt

      Processing package instance <SUNWj7cfg> from </java7_SVR4>

      JDK 7.0 Host Config. (1.7.0_80)(sparc) 1.7.0,REV=2011.06.27.06.20
      Copyright (c) 1995, 2015, Oracle and/or its affiliates. All rights reserved.
      Using </> as the package base directory.
      ## Processing package information.
      ## Processing system information.
      3 package pathnames are already properly installed.
      ## Verifying package dependencies.
      WARNING:
      The <SUNWj7rt> package "JDK 7.0 Runtime Env. (1.7.0)"
      is a prerequisite package and should be installed.

      Do you want to continue with the installation of <SUNWj7cfg> [y,n,?] y
      ## Verifying disk space requirements.
      ## Checking for conflicts with packages already installed.
      ## Checking for setuid/setgid programs.
      .
      ..
      ...

   4. Verify that the SUNWj7* pkgs are now installed on the system:
      # pkginfo | grep SUNWj7
      system SUNWj7cfg JDK 7.0 Host Config. (1.7.0_80)
      system SUNWj7dev JDK 7.0 Dev. Tools (1.7.0_80)
      system SUNWj7jmp JDK 7.0 Man Pages: Japan (1.7.0_80)
      system SUNWj7man JDK 7.0 Man Pages (1.7.0_80)
      system SUNWj7rt JDK 7.0 Runtime Env. (1.7.0_80)
      #
   5. Apply patch 152096-53 (or later) to the system.
      
      This patch can either be downloaded from MOS via the link above or https://updates.oracle.com/all_unsigned/152096-53.zip. It can also be found in the Solaris 10 Recommended Patchset that is shipped as part of the October 2017 QFSDP for SuperCluster.
      
      
      1. Check to see if patch 152096-53 (or later) is already installed on the system:
         To see what revision(s) of patch 152096 are installed:
         # patchadd -p | grep 152096
         #
      2. If a version less than 152096-53 is installed, then find the patch directory in the Solaris 10 Recommended Patchset (alternatively download and unzip patch 152096-53 and skip this section):
         
         # mount -f hsfs /export/SCMU_2017.10/solaris/components/s10/10_Recommended/10_Recommended.iso /tmp/rec/
         # ls -l /tmp/rec/
         10_Recommended_CPU_2017-07.README   installpatchset
         10_Recommended_CPU_2017-07.html         patch_order
         Copyright patches/                                     LEGAL_LICENSE.TXT
         patchset.conf                                             installcluster
         # ls -l /tmp/rec/patches/152096-53
         total 1872
         -r--r--r-- 1 root root 524728 Jul 14 17:17 152096-53.html
         -r--r--r-- 1 root root 725 Apr 16 2014 LEGAL_LICENSE.TXT
         -r--r--r-- 1 root root 419816 Jul 14 17:17 README.152096-53
         dr-xr-xr-x 4 root root 2048 Jul 12 03:34 SUNWj7cfg
         dr-xr-xr-x 4 root root 2048 Jul 12 03:34 SUNWj7dev
         dr-xr-xr-x 4 root root 2048 Jul 12 03:34 SUNWj7dmo
         dr-xr-xr-x 4 root root 2048 Jul 12 03:34 SUNWj7jmp
         dr-xr-xr-x 4 root root 2048 Jul 12 03:34 SUNWj7man
         dr-xr-xr-x 4 root root 2048 Jul 12 03:34 SUNWj7rt
         -r-xr-xr-x 1 root root 210 Jul 13 10:07 patchinfo
         # cd /tmp/rec/patches
      3. Install patch 152096-53 and verify it's on the system:
         
         # patchadd 152096-53
         Validating patches...

         Loading patches installed on the system...

         Done!

         Loading patches requested to install.

         Done!

         The following requested patches have packages not installed on the system
         Package SUNWj7dmo from directory SUNWj7dmo in patch 152096-53 is not installed on the system. Changes for package SUNWj7dmo will not be applied to the system.

         Checking patches that you specified for installation.

         Done!

         Approved patches will be installed in this order:

         152096-53

         Checking installed patches...
         Installing patch packages...

         Patch 152096-53 has been successfully installed.
         See /var/sadm/patch/152096-53/log for details

         Patch packages installed:
         SUNWj7cfg
         SUNWj7dev
         SUNWj7jmp
         SUNWj7man
         SUNWj7rt

         # patchadd -p | grep 152096-53
         Patch: 152096-53 Obsoletes: 147692-99 Requires: Incompatibles: Packages: SUNWj7cfg SUNWj7man SUNWj7rt SUNWj7dev SUNWj7jmp
         #

         
         
         
         
3. Ensure that Java 7 is set as the default system-wide version of java:
   
   In the example below the default Java version will be set to Java 6 Update 161 to start with and it will then be change to use Java 7 Update 151 ('java -version' can be used to verify the default system-wide version of Java):
   
   # ls -l /usr/java
   lrwxrwxrwx 1 root root 16 Nov 9 07:11 /usr/java -> /usr/jdk/jdk1.6.0_161
   bash-3.2# java -version
   java version "1.6.0_161"
   Java(TM) SE Runtime Environment (build 1.6.0_161-b13)
   Java HotSpot(TM) Server VM (build 20.161-b13, mixed mode)
   #
   
   # rm /usr/java
   # ln -s /usr/jdk/jdk1.7.0_151 /usr/java
   # ls -l /usr/java
   lrwxrwxrwx 1 root root 21 Nov 9 07:13 /usr/java -> /usr/jdk/jdk1.7.0_151
   # java -version
   java version "1.7.0_151"
   Java(TM) SE Runtime Environment (build 1.7.0_151-b15)
   Java HotSpot(TM) Server VM (build 24.151-b15, mixed mode)
   #
    

References

This solution has no attachment