Oracle ZFS Storage Appliance: Unable to Change the Infiniband Partition Key to less than 0x8000

Asset ID:	1-72-1954502.1
Update Date:	2014-12-23
Keywords:

Solution Type Problem Resolution Sure

Solution 1954502.1 : Oracle ZFS Storage Appliance: Unable to Change the Infiniband Partition Key to less than 0x8000

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-BA - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

Unable to change the partition key less than 0x8000 in the Oracle ZFS Storage Appliance.

The IB switch default pkey is 0x7fff

NAS1:configuration net datalinks pffff_ibp0> show
Properties:
  class = partition
  label = IBP0
  mac = 80:0:0:49:fe:80:0:0:0:0:0:0:0:10:e0:0:2:54:ca:b1
  links = ibp0
  pkey = ffff
  linkmode = cm

IB switch appears to be unable to use the same pkey:

[root@localhost ~]# smpartition modify -n testpartition -pkey 0x8001 -m full
Wrong value for pkey. Legal value is 1 to 0x7fff

The switch side supports 1-0x7fff . but the Oracle ZFS Storage Appliance is not able to use any value below 8000

NAS1:configuration net datalinks pffff_ibp0> set pkey=1fff
error: invalid property value "1fff": pkey is out of range
NAS1:configuration net datalinks pffff_ibp0> set pkey=7fff
error: invalid property value "7fff": pkey is out of range
NAS1:configuration net datalinks pffff_ibp0>

Cause

The InfiniBand protocol reserves the most significant bit (0x8000) of the hexadecimal partition key (pkey) for its own use, which can cause admin/operator confusion if not known.

Solution

The acceptable range for a pkey for the ZFSSA product is between 0x8001 and 0xffff

However - the InfiniBand protocol reserves the most significant bit (0x8000) of the hexadecimal partition key (pkey) for its own use. Therefore, pkeys of, for example 0x8503 and 0x0503 are the same.

So - if you set the pkey to be 0xffff in the Oracle ZFS Storage Appliance, it should be able to see other hosts that are in the 0x7fff partition.

This is mentioned in page 12 of this document:

http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/zs3ba-supercluster-config-2014-2227429.pdf

Attachments

This solution has no attachment