SuperCluster- Best Practices for creating 8503 infiniband storage network IPMP interface in Solaris 11 local zones

Asset ID:	1-71-1630341.1
Update Date:	2016-06-21
Keywords:

Solution Type Technical Instruction Sure

Solution 1630341.1 : SuperCluster- Best Practices for creating 8503 infiniband storage network IPMP interface in Solaris 11 local zones

Related Items


Oracle SuperCluster T5-8 Full Rack
 SPARC SuperCluster T4-4 Full Rack
 Oracle SuperCluster M7 Hardware
 Oracle SuperCluster T5-8 Half Rack
 SPARC SuperCluster T4-4 Half Rack
 Oracle SuperCluster M6-32 Hardware
 Oracle SuperCluster T5-8 Hardware
 SPARC SuperCluster T4-4

Related Categories


PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST

The Oracle SuperCluster exavm (database) local zones are not built with an interface on the 8503 by default. However this is necessary for accessing the integrated  ZFSSA (7320/ZS3ES).

In this Document

Goal

This note will assist you in setting in configuring the proper components in the global zone that will then support creating the storage network IPMP group in the Solaris 11 local zones on Oracle SuperCluster.

Solution

This document assumes you have reviewed the contents of the SuperCluster Supported Versions document for your hardware type as well as the SuperCluster Critical Issues document.

Oracle SuperCluster Supported Software Versions - All Hardware Types <Document 1567979.1>

SuperCluster Critical Issues <Document 1452277.1>

1) Determine the nets in the global zone (LDom) of the links that are configured on the 8503 storage network.

2) Edit the zone configuration to add anets to the zone

3) Attach if necessary, boot and log into the local zone and create the active/passive IPMP group and give it an IP address.

4) Test an operation to the on board ZFSSA (7320/ZS3-ES).

5) Duplicate the process for all of your Solaris 11 local zones that require access on the infiniband 8503 storage network.

References

Applies to:

Oracle SuperCluster T5-8 Half Rack - Version All Versions and later
Oracle SuperCluster T5-8 Full Rack - Version All Versions and later
SPARC SuperCluster T4-4 Full Rack - Version All Versions and later
Oracle SuperCluster M7 Hardware - Version All Versions and later
Oracle SuperCluster M6-32 Hardware - Version All Versions and later
Oracle Solaris on SPARC (64-bit)
Background: The Oracle SuperCluster uses a partitioned infiniband fabric in order to segregate traffic . All database and exadata traffic utilize the default partition key (PKEY) of FFFF and the ZFSSA access is over what we refer to as "the storage network" on PKEY 8503.

Goal

This note will assist you in setting in configuring the proper components in the global zone that will then support creating the storage network IPMP group in the Solaris 11 local zones on Oracle SuperCluster.

Solution

Note: The IP address used in this document are examples replace with the proper IP address and subnet information for your private network IP scheme.

This document assumes you have reviewed the contents of the SuperCluster Supported Versions document for your hardware type as well as the SuperCluster Critical Issues document.

Oracle SuperCluster Supported Software Versions - All Hardware Types <Document 1567979.1>
SuperCluster Critical Issues <Document 1452277.1>

1) Determine the nets in the global zone (LDom) of the links that are configured on the 8503 storage network.

Note: Your net aliases may differ in each LDom depending on if it was configured as a database or application (general purpose) dedicated or IO domain.

# dladm show-part

LINK PKEY OVER STATE FLAGS

stor_ipmp0_0 8503 net14 up f--- Note: These are the two nets that you are interested in. You can identify them due to them having the 8503 PKEY.
stor_ipmp0_1 8503 net15 up f---

bondib0_0    FFFF net15        up       f---
bondib0_1    FFFF net16        up       f---
bondib1_0    FFFF net13        up       f---
bondib1_1    FFFF net14        up       f---

2) Edit the zone configuration to add anets to the zone

# zonecfg -z z1
add anet
set linkname=stor_ipmp0_0
set lower-link=net14
set pkey=0x8503
end
add anet
set linkname=stor_ipmp0_1
set lower-link=net15
set pkey=0x8503
end
z1> verify
z1> commit
z1> exit

3) Attach if necessary, boot and log into the local zone and create the active/passive IPMP group and give it an IP address.

# zoneadm -z z1 attach
# zoneadm -z z1 boot
# zlogin z1

Note: this step is done within the local zone and change for your internal IB subnet

#ipadm create-ip stor_ipmp0_0
#ipadm create-ip stor_ipmp0_1
#ipadm create-ipmp -i stor_ipmp0_0,stor_ipmp0_1 stor_ipmp0
#ipadm set-ifprop -p standby=on -m ip stor_ipmp0_1
#ipadm create-addr -a local=192.168.66.153/20 stor_ipmp0

4) Test an operation to the on board ZFSSA (7320/ZS3-ES).

# ssh 192.168.15.1 shares select ssc select s11repo ls
Properties:
                    aclinherit = restricted (inherited)
                       aclmode = discard (inherited)
                         atime = true (inherited)
               casesensitivity = mixed
                      checksum = fletcher4 (inherited)
                   compression = off (inherited)
                         dedup = false (inherited)
                 compressratio = 100
                        copies = 1 (inherited)
                      creation = Mon Aug 12 2013 12:25:18 GMT+0000 (UTC)
                       logbias = latency (inherited)
                    mountpoint = /export/ssc/s11repo (inherited)
                 normalization = none
                         quota = 0
                    quota_snap = true
                      readonly = false (inherited)
                    recordsize = 128K (inherited)
                   reservation = 0
              reservation_snap = true
                      rstchown = true (inherited)
                secondarycache = all (inherited)
                        shadow = none
                        nbmand = false (inherited)
                      sharesmb = off (inherited)
                      sharenfs = sec=sys,rw=@192.168.28.0/22,root=@192.168.28.0/22
                       snapdir = hidden (inherited)
                      utf8only = false
                         vscan = false (inherited)
                      sharedav = off (inherited)
                      shareftp = off (inherited)
                     sharesftp = off (inherited)
                     sharetftp = (inherited)
                          pool = ssc
                canonical_name = ssc/local/ssc/s11repo
                      exported = true (inherited)
                     nodestroy = false
                    space_data = 9.00G
              space_unused_res = 0
               space_snapshots = 0
               space_available = 23.7T
                   space_total = 9.00G
                    root_group = root
              root_permissions = 755
                     root_user = root
                        origin =

Children:
                        snapshots => Manage snapshots
                      replication => Manage remote replication
                            users => View per-user usage and manage user quotas
                           groups => View per-group usage and manage group
                                     quotas

5) Duplicate the process for all of your Solaris 11 local zones that require access on the infiniband 8503 storage network.

Attachments

This solution has no attachment