Scripts Run As "ROOT" User Failing on NFSv4 Share On Exalogic Compute Nodes

Asset ID:	1-72-1546928.1
Update Date:	2013-05-01
Keywords:

Solution Type Problem Resolution Sure

Solution 1546928.1 : Scripts Run As "ROOT" User Failing on NFSv4 Share On Exalogic Compute Nodes

Applies to:

Exalogic Elastic Cloud X3-2 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Exalogic Elastic Cloud Software - Version 2.0.0.0.0 to 2.0.4.0.2
Oracle Exalogic Elastic Cloud X2-2 Hardware - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

On Exalogic, certain software installations such as Oracle 11g DB Client require scripts to be run as root for post installation.

When trying to execute orainstRoot.sh while logged in as root to a compute node, the following may occur:

[root@el01cn01 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/home/psoft/installs/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/oui': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/oui/srcs.lst': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/ContentsXML': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/ContentsXML/inventory.xml': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/ContentsXML/comps.xml': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/ContentsXML/libs.xml': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/logs': Operation not permitted
/bin/chmod: `/u01/home/psoft/installs/oraInventory/logs': Permission denied
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/oraInst.loc': Operation not permitted
/bin/chmod: changing permissions of `/u01/home/psoft/installs/oraInventory/orainstRoot.sh': Operation not permitted
OUI-35086:WARNING: chmod of /u01/home/psoft/installs/oraInventory
Adding read,write permissions for group.
,Removing read,write,execute permissions for world.
failed!

Cause

Misconfiguration of the NFSv4 share on the ZFS appliance has been known to cause the following issue:

OUI-35086:WARNING: chmod of /dvlp/home/psoft/installs/oraInventory
Adding read,write permissions for group.
,Removing read,write,execute permissions for world.
failed!

Solution

Before you begin troubleshooting issues on NFSv4, please make sure you meet the requirements for NFSv4 stability.

Requirements For NFSv4 stability

Following are mandatory requirements to ensure NFSv4 stability in Exalogic Linux physical environments. Not having any or all of the following may result in NFSv4 stability performance and File lock issues:

Proper NTP Configuration: NTP is an important component of the NFSv4 scheme and both the client compute nodes and the server ZFS storage appliance must sync to the same NTP server(s). Typically this would be configured as the same across all Exalogic components during the ECU setup. However, we have seen numerous incidents where either NTP was not configured properly or the NTP servers were not reachable causing NFS outage in either case.
Proper NIS Configuration: In order to use NFSv4 file system NIS setup has to be done. Not having NIS setup or improperly configured NIS also causes NFS and File lock issues. Please refer to Note 1470844.1 for configuring NIS. This note describes how to setup NIS master/slave on compute nodes.

NOTE: If NIS was configured on compute nodes prior to EECS 2.0.0.0.0, it must be again reconfigured as part of the upgrade.

Make sure you are running the following version of YP packages. It is always recommended to run YP packages which are at least at below listed versions or later. There are a few reported issues around YP NIS services crashing when using older versions of YP packages. These NIS packages can be downloaded from http://public-yum.oracle.com/public-yum-el5.repo.

ypserv: ypserv-2.19-9.el5_8.1.x86_64.rpm (or later)
yptools: yp-tools-2.9-1.el5 (or later)
ypbind: ypbind-1.19-12.el5_6.1.x86_64.rpm (or later)

IPMP Settings on ZFS: It is important to have below recommended IPMP settings change on Exalogic ZFS Storage Appliance. The steps on changing the default IPMP settings on ZFS storage appliance are also documented in Exalogic Known Issues Note 1268557.1, refer to section "Default IPMP settings in ZFS Storage Appliance causing system hang issues" under "ZFS STORAGE APPLIANCE".

Disable NFSv4 Delegation: There is a known issue with the currently available ZFSSA firmware on Exalogic releases where NFSv4 delegation enabled on ZFSSA causes hangs on NFSv4 clients. Disabling NFSv4 delegation is the official direction from Oracle at the moment. It is valid and the recommended configuration. This only applies when using NFSv4. Please follow steps listed in Note 1481713.1 for disabling NFSv4 delegation on both the ZFS storage heads (active and passive).

Using CLI

1. Login into ZFS SA from one of the Exalogic compute nodes.

$ ssh root@el01sn01
Password:
Last login: Tue Nov 6 06:22:38 2012 from sscdb01

el01sn01:>

2. Verify NFS services.

el01sn01:>configuration services nfs> show

If the service is disabled, enable using 'enable' command

el01sn01:>configuration services nfs> enable

3. Go to 'shares' menu.

el01cn01:> shares

4. Find your problem project.

el01sn01-adm:shares> show
Properties:
pool = exalogic

Projects:
ExalogicControl
u01

5. Select project u01, share home.

el01sn01-adm:shares> select u01
el01sn01-adm:shares u01> select home
el01sn01-adm:shares u01/home>

6. Check the settings for sharenfs, root_user, root_group on the NFSv4 filesystem share.

el01sn01-adm:shares u01/home> show
Properties:
aclinherit = restricted (inherited)
aclmode = discard (inherited)
atime = true (inherited)
casesensitivity = mixed
checksum = fletcher4 (inherited)
compression = off (inherited)
dedup = false (inherited)
compressratio = 100
copies = 1 (inherited)
creation = Thu Apr 18 2013 22:12:03 GMT+0000 (UTC)
logbias = latency (inherited)
mountpoint = /export/u01 (inherited)
normalization = none
quota = 0
quota_snap = true
readonly = false (inherited)
recordsize = 128K (inherited)
reservation = 0
reservation_snap = true
rstchown = true (inherited)
secondarycache = all (inherited)
shadow = none
nbmand = false (inherited)
sharesmb = off (inherited)
sharenfs = on (inherited)
snapdir = hidden (inherited)
utf8only = false
vscan = false (inherited)
sharedav = off (inherited)
shareftp = off (inherited)
sharesftp = off (inherited)
sharetftp = (inherited)
pool = exalogic
canonical_name = exalogic/local/giherna/u01
exported = true (inherited)
nodestroy = false
space_data = 31K
space_unused_res = 0
space_snapshots = 0
space_available = 13.6T
space_total = 31K
root_group = other
root_permissions = 700
root_user = nobody
origin =

Children:
snapshots => Manage snapshots
replication => Manage remote replication
users => View per-user usage and manage user quotas
groups => View per-group usage and manage group
quotas

7. To resolve this issue sharenfs default setting "on" must be changed

From

sharenfs = on

sharenfs = sec=sys,rw,root=@<ib-subnet>/<ib-cidr>

el01sn01-adm:shares u01/home> set sharenfs="sec=sys,rw,root=@192.168.10.0/24"
sharenfs = sec=sys,rw,root=@192.168.10.0/24 (uncommitted)
el01sn01-adm:shares u01/home> commit

NOTE: Use IB network interface for accessing ZFS SA. For example the IP address in the above command will be the default 192.168.10.0 with a subnet mask of /24.
If this is a multirack setup, the subnet mask may be different. See the table below for other mappings that may be applicable for a multirack, virtual scenario.

/24	256 hosts	255.255.255.0
/23	512	255.255.254.0
/22	1024	255.255.252.0
/21	2048	255.255.248.0
/20	4096	255.255.240.0

8. Verify that root_group and root_user are set to the correct NIS user or owner of software.

el01sn01-adm:shares u01/home> set root_user=psoft
root_user = (uncommitted)
el01sn01-adm:shares u01/home> commit
el01sn01-adm:shares u01/home> show
Properties:
aclinherit = restricted (inherited)
aclmode = discard (inherited)
atime = true (inherited)
casesensitivity = mixed
checksum = fletcher4 (inherited)
compression = off (inherited)
dedup = false (inherited)
compressratio = 100
copies = 1 (inherited)
creation = Thu Apr 18 2013 22:17:19 GMT+0000 (UTC)
logbias = latency (inherited)
mountpoint = /export/u01/home
normalization = none
quota = 0
quota_snap = true
readonly = false (inherited)
recordsize = 128K (inherited)
reservation = 0
reservation_snap = true
rstchown = true (inherited)
secondarycache = all (inherited)
shadow = none
nbmand = false (inherited)
sharesmb = off (inherited)
sharenfs = sec=sys,rw,root=@192.168.10.0/24
snapdir = hidden (inherited)
utf8only = false
vscan = false (inherited)
sharedav = off (inherited)
shareftp = off (inherited)
sharesftp = off (inherited)
sharetftp = (inherited)
pool = exalogic
canonical_name = exalogic/local/giherna/home
exported = true (inherited)
nodestroy = false
space_data = 31K
space_unused_res = 0
space_snapshots = 0
space_available = 13.6T
space_total = 31K
root_group = other
root_permissions = 700
root_user = psoft
origin =

Children:
snapshots => Manage snapshots
replication => Manage remote replication
users => View per-user usage and manage user quotas
groups => View per-group usage and manage group
quotas

Verify the NFS filesystem properties.

el01sn01-adm:shares u01/home> show
Properties:
aclinherit = restricted (inherited)
aclmode = discard (inherited)
atime = true (inherited)
casesensitivity = mixed
checksum = fletcher4 (inherited)
compression = off (inherited)
dedup = false (inherited)
compressratio = 100
copies = 1 (inherited)
creation = Thu Apr 18 2013 22:17:19 GMT+0000 (UTC)
logbias = latency (inherited)
mountpoint = /export/u01/home
normalization = none
quota = 0
quota_snap = true
readonly = false (inherited)
recordsize = 128K (inherited)
reservation = 0
reservation_snap = true
rstchown = true (inherited)
secondarycache = all (inherited)
shadow = none
nbmand = false (inherited)
sharesmb = off (inherited)
sharenfs = sec=sys,rw,root=@192.168.10.0/24
snapdir = hidden (inherited)
utf8only = false
vscan = false (inherited)
sharedav = off (inherited)
shareftp = off (inherited)
sharesftp = off (inherited)
sharetftp = (inherited)
pool = exalogic
canonical_name = exalogic/local/giherna/home
exported = true (inherited)
nodestroy = false
space_data = 31K
space_unused_res = 0
space_snapshots = 0
space_available = 13.6T
space_total = 31K
root_group = other
root_permissions = 700
root_user = psoft
origin =

Children:
snapshots => Manage snapshots
replication => Manage remote replication
users => View per-user usage and manage user quotas
groups => View per-group usage and manage group
quotas

eo01sn01:shares u01/home> exit
Connection to el01sn01 closed.
$

9. Verify your NFSv4 shares are created as described in documentation "9.5 Creating NFSv4 Mount Points on Oracle Linux".

10. After changes have been made to the shares on ZFS, it is necessary to unmount and remount the shares from the compute node(s).

# umount -a -t nfs

# mount -a

11. After making the changes discussed in this document, you should now be able to execute the orainstRoot.sh script.

Community Discussions

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

References

<NOTE:1470844.1> - How To Configure NIS Master, Slave And Client Configuration On Compute Nodes In Exalogic Elastic Cloud Software 2.x Physical Environment
<NOTE:1503867.1> - Configure and Mount NFS shares from SUN ZFS Storage 7320 for SPARC SuperCluster
<NOTE:1492780.1> - Troubleshooting Guide For NFSv4 File Lock & Hang Issues On Exalogic Linux Environments

Attachments

This solution has no attachment