Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
This note demonstrates an example implementation of Oracle E-Business Suite on SPARC SuperCluster.
This note only shows an example implementation on SPARC SuperCluster. Though the example only includes an example the implementation principles can be applied to many different modules that could be run on SPARC SuperCluster.
Please see the attached document for a printer friendly version of this note that will properly show the command output sections.
Overview
The objective of this document is to install Oracle E-Business Suite 12.1.3 with Oracle Database 11.2.0.3 on a SPARC SuperCluster.
1.1 Architecture
This document assumes a standard two-node SPARC SuperCluster with two SPARC T4-4 servers. SPARC SuperCluster uses Oracle VM Server for SPARC (also known as Logical Domains or LDoms) and Oracle Solaris Zones virtualization technology. This document assumes that each cluster node has a Solaris 10 domain (application domain) and a Solaris 11 domain (database domain).
Figure 1 shows the logical architecture that is created by following the procedure in this document. Three zone clusters—OPMN Zone cluster (opmn-zc), CM Zone cluster (cm-zc), and database zone cluster (db-zc) are created on the Solaris 10 application domains. Each zone cluster contains two nodes, for example: iopmn1 and iopmn2, icm1 and icm2, and idb1 and idb2. Oracle E-Business Suite is installed on the nodes in these zone clusters.
Figure 1. Logical architecture.
Oracle Database is initially installed in the db-zc zone cluster, and is later migrated to Oracle RAC with ASM in the Solaris 11 LDoms. The db-zc is a transient zone cluster and can be removed once the database has been migrated to the Solaris 11 LDoms.
1.2 Task Overview
The installation procedure described in this document includes the following high-level steps:
1. Install Oracle Solaris Cluster and Zone Clusters (see Section 2).
2. Review the system configuration (see Section 3).
3. Install Oracle E-Business Suite (see Section 4).
4. Consult E-Business Suite-related MOS Notes (see Section 5).
5. Solaris Cluster Configuration—create resources for the DB tier in zone cluster db-zc, and OPMN and CM resources for the App Tier in zone clusters opmn-zc and cm-zc (see Section 6).
6. Convert single instance database to Oracle RAC on ASM (see Section 7).
7. Perform post-Oracle RAC migration changes (see Section 8).
1.3 Related My Oracle Support Notes
The following related My Oracle Support Notes are referenced as part of this installation:
· Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Oracle Solaris on SPARC (64-bit) [ID 761568.1]
· Interoperability Notes EBS R12 with Database 11gR2 [ID 1058763.1]
· Oracle E-Business Suite Release 12.1.3 Readme [ID 1080973.1]
· Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite R12 [ID 823587.1]
Note: MOS Notes are subject to change. For that reason, specific configuration steps related to the MOS Notes are not included in this document. Instead, the reader is referred to the corresponding MOS Notes to obtain the most recent information.
2. Oracle Solaris Cluster and Zone Cluster installation
The Solaris 10 domains of the EBS rig (etc11-01-s10 and etc11-02-s10) need to be installed with Oracle Solaris Cluster 3.3 u1, and three zone clusters (opmn-zc, cm-zc, and db-zc) need to be created. The following sections describe the appropriate tasks.
2.1 Add hostnames to the global zone hosts file
Add the following hostnames to the /etc/hosts file in the global zone on both Solaris 10 domains.
|
# cat >> /etc/hosts
10.129.186.20 edb1
10.129.186.21 edb2
10.129.186.22 eopmn1
10.129.186.23 eopmn2
10.129.186.24 ecm1
10.129.186.25 ecm2
10.129.186.26 edb-lh
10.129.186.27 eweb-lh
10.129.186.30 ecm1-lh
10.129.186.31 ecm2-lh
192.168.30.1 etc11sn1-stor
192.168.30.31 iopmn1
192.168.30.32 iopmn2
192.168.30.33 iopmn1-lh
192.168.30.34 iopmn2-lh
192.168.30.41 icm1
192.168.30.42 icm2
192.168.30.43 icm1-lh
192.168.30.44 icm2-lh
192.168.30.51 idb1
192.168.30.52 idb2
192.168.30.53 idb-lh
^D
|
2.2 Configure the ZFS SA iSCSI LUN for Cluster quorum device
The following sections describe the steps to configure the ZFS SA iSCSI LUN for cluster quorum device.
2.2.1 Configure iSCSI Initiator for the cluster nodes
The steps to configure the ZFS Storage Appliance iSCSI LUN for cluster quorum device need to be performed on both nodes in the cluster. First, perform the configuration steps on the Solaris 10 domain on the first node (etc11-01-s10):
|
# hostname
etc11-01-s10
# iscsiadm modify discovery -s enable
# iscsiadm list discovery
Discovery:
Static: enabled
Send Targets: disabled
iSNS: disabled
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7811
Initiator node alias: ovmt-1
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Tunable Parameters (Default/Configured):
Session Login Response Time: 60/-
Maximum Connection Retry Time: 180/-
Login Retry Time Interval: 60/-
Configured Sessions: 1
# iscsiadm modify initiator-node -A `hostname`
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7811
Initiator node alias: etc11-01-s10
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Tunable Parameters (Default/Configured):
Session Login Response Time: 60/-
Maximum Connection Retry Time: 180/-
Login Retry Time Interval: 60/-
Configured Sessions: 1
|
Next, perform the configuration steps on second Solaris 10 domain (etc11-02-s10):
|
# hostname
etc11-02-s10
# iscsiadm modify discovery -s enable
# iscsiadm list discovery
Discovery:
Static: enabled
Send Targets: disabled
iSNS: disabled
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7811
Initiator node alias: ovmt-1
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Tunable Parameters (Default/Configured):
Session Login Response Time: 60/-
Maximum Connection Retry Time: 180/-
Login Retry Time Interval: 60/-
Configured Sessions: 1
# iscsiadm modify initiator-node -N iqn.1986-03.com.sun:01:00144ffae18a.4e5b7812 -A `hostname`
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7812
Initiator node alias: etc11-02-s10
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Tunable Parameters (Default/Configured):
Session Login Response Time: 60/-
Maximum Connection Retry Time: 180/-
Login Retry Time Interval: 60/-
Configured Sessions: 1
|
2.2.2 Configure the iSCSI Quorum Device on the 7320c
In the GUI, perform the following steps:
1. Create new quorum device:
· Go to 7320 SAN Configuration, and create iSCSI Target "QuorumTarget".
· Choose ipmp1 interface, and drag and drop it to the iSCSI Target Group to create a new group.
· Name it "QuorumGroup". Click Apply.
2. Note the target IQN. For example:
|
iqn.1986-03.com.sun:02:30c8be01-6002-61b8-9bce-fb88325899ed
|
3. Select the Initiators headlight.
· Create iSCSI Initiator with cluster nodes initiator id from above, named them with nodes IB subnet internal hostname or 10GbE hostname.
· Drag and drop it to the iSCSI Initiator Group to create a new group.
· Name it “QuorumGroup” and select both iSCSI Initiators previously created. Click Apply.
4. Create a new LUN:
· In the shares page, select Project, and create a new project “QuorumProject” in the ssc pool.
· In that project, select the LUNs tab.
· Create a new LUN of 1G size, named QuorumLUN and select the drop down menus to put it in the created QuorumGroup Initiator Group.
· Select also LU Number 0 and Fix LU number. Click Apply.
2.2.3 Configure the iSCSI LUN for the cluster nodes
Continue with the following commands:
|
# iscsiadm add static-config iqn.1986-03.com.sun:02:30c8be01-6002-61b8-9bce-fb88325899ed,192.168.30.1
# iscsiadm list static-config
Static Configuration Target: iqn.1986-03.com.sun:02:30c8be01-6002-61b8-9bce-fb88325899ed,192.168.30.1:3260
# iscsiadm list target -S
Target: iqn.1986-03.com.sun:02:30c8be01-6002-61b8-9bce-fb88325899ed
Alias: QuorumTarget
TPGT: 2
ISID: 4000002a0000
Connections: 1
LUN: 0
Vendor: SUN
Product: ZFS Storage 7320
OS Device Name:/dev/rdsk/c1t600144F0D65390FB00004F3C9C610001d0s2
|
2.3 Configure NFS file systems for EBS on the 7320c
Using the web management interface of the ZFS Storage appliance, go to the Shares panel and perform the following:
· Create a new project named "ebs-apps",
set its "Mountpoint" property to /export/etc11
· In that project, create the following file system and verify that it has the corresponding mount point:
|
NAME MOUNTPOINT
ebsfs /export/etc11/ebsfs
|
· Create a new project named "ebs-db",
set its "Mountpoint" property to /export/etc11
· In that project, create the following file systems and verify that they have the corresponding mount points:
|
NAME MOUNTPOINT
oralog /export/etc11/oralog
oradump /export/etc11/oradump
orainst /export/etc11/orainst
oradata /export/etc11/oradata
|
Note: The above operations for iSCSI Quorum device and NFS file systems could also be performed using the ZFS Storage appliance command line interface.
2.4 Install Oracle Solaris Cluster
In this step we are going to install Oracle Solaris Cluster 3.3 5/11 Release (OSC3.3u1) for SPARC, as well as its latest associated patches and the Oracle External Proxy (OEP).
2.4.1 Initial Steps
The next tasks will consist of downloading Oracle Solaris Cluster 3.3u1 and latest patches from the Oracle Technology Network (OTN) and My Oracle Support (MOS).
Note to ACS engineers having just completed the SPARC SuperCluster installation: the SW build (from its media) used for that installation should include OSC3.3u1, most of the patches and packages used in this procedure, in which case they do not need a download from MOS.
Since the steps will have to be performed on each node, it is recommended, and assumed that the files are copied to an NFS share, read-writable mounted on all Oracle Solaris 10 General Purpose domains. That NFS share could be created on the SPARC SuperCluster ZFS Storage Appliance. We will call the mounted location of that share <OSC Installation NFS share dir>.
From one Oracle Solaris 10 General Purpose domain, perform the following:
|
# mkdir <OSC Installation NFS share dir>/ebs-ssc
# cd <OSC Installation NFS share dir>/ebs-ssc
# mkdir OSC3.3u1 patches ZFSSA-SC-plugin oep
|
Download Oracle Solaris Cluster 3.3 5/11 Release (OSC3.3u1) for SPARC from OTN:
http://www.oracle.com/technetwork/server-storage/solaris-cluster/downloads/index.html
Save the downloaded file in <OSC Installation NFS share dir>/ebs-ssc/OSC3.3u1, then unzip it:
|
# cd <OSC Installation NFS share dir>/ebs-ssc/OSC3.3u1
# /bin/unzip solaris-cluster-3_3u1-ga-sparc.zip
|
Download the following patches for OSC3.3u1 from MOS (support.oracle.com) into the directory <OSC Installation NFS share dir>/ebs-ssc/patches
· 145333-10
· 145335-09
· 145337-04
· 145341-04
· 145343-04
· 145345-03
· 146089-04
· 146091-06
· 146093-03
· 146762-04
· 146764-05
Then unzip the patches:
|
# cd <OSC Installation NFS share dir>/ebs-ssc/patches
# /bin/unzip '*.zip'
# ls -1 | grep -v zip > patch_order
# cat ./patch_order
145333-10
145335-09
145337-04
145341-04
145343-04
145345-03
146089-04
146091-06
146093-03
146762-04
146764-05
|
Download patch 12736304 (Generic Platform) from MOS (support.oracle.com) into the directory <OSC Installation NFS share dir>/ebs-ssc/ZFSSA-SC-plugin and unzip it:
|
# cd <OSC Installation NFS share dir>/ebs-ssc/ZFSSA-SC-plugin
# /bin/unzip p12736304_120_Generic.zip
# unzip SunZFSNFSSC.zip#
|
Download from MOS (support.oracle.com) patch 13987101 for Oracle Solaris on SPARC in to the <OSC Installation NFS share dir>/ebs-ssc/oep directory and unzip it:
|
# cd <OSC Installation NFS share dir>/ebs-ssc/oep
# /bin/unzip p13987101_330_SOLARIS64.zip
Archive: p13987101_330_SOLARIS64.zip
inflating: readme.txt
extracting: solaris-cluster-ha-external-proxy-3_3u1-sparc.zip
# /bin/unzip solaris-cluster-ha-external-proxy-3_3u1-sparc.zip
|
2.4.2 Installing Oracle Solaris Cluster
From each node, run the JES installer:
|
# pwd
<OSC Installation NFS share dir>/ebs-ssc/SC3.3u1/Solaris_sparc
# export DISPLAY=cheetah-c5:5
# ./installer
|
Select “Oracle Solaris Core”
· “HA for Oracle”
· “Support for Oracle RAC”
· “HA for Ebusiness-Suite”
The summary should look like this:
|
Product: Oracle Solaris Cluster
Uninstall Location: /var/sadm/prod/SUNWentsyssc33u1
Space Required: 576.66 MB
---------------------------------------------------
Java DB
Java DB Server
Java DB Client
Oracle Solaris Cluster 3.3u1
Oracle Solaris Cluster Core
Oracle Solaris Cluster Manager
Oracle Solaris Cluster Agents 3.3u1
Oracle Solaris Cluster HA for Oracle
Oracle Solaris Cluster Support for Oracle RAC
Oracle Solaris Cluster HA for Oracle E-Business Suite
In order to notify you of potential updates, we need to confirm an internet connection. Do you want to proceed [Y/N] : N
|
2.4.3 Installing the latest patches
Once the software installation has completed, install the latest patches:
|
# pwd
<OSC Installation NFS share dir>/ebs-ssc/patches
# patchadd -M . ./patch_order
|
2.4.4 Installing the Oracle E-Business OSC agent patch
Download from MOS (support.oracle.com) patch 146760-04 for Oracle Solaris on SPARC into the <OSC Installation NFS share dir>/ebs-ssc/patches directory and unzip it:
|
# cd <OSC Installation NFS share dir>/ebs-ssc/patches
# /bin/unzip 146760-04.zip
# patchadd -d . 146760-04
|
2.4.5 Installing the Oracle External Proxy
The generic instructions on how this feature is installed and configured are provided in this file:
|
# ls Docs/oracle-external-proxy.pdf
Docs/oracle-external-proxy.pdf
|
Install the Oracle External Proxy (OEP) package on each node:
|
# pwd
<OSC Installation NFS share dir>/ebs-ssc/oep/Solaris_sparc/Product/sun_cluster_
agents/Solaris_10/Packages
# ls
ORCLscoep
# pkgadd -d . ORCLscoep
# pkginfo ORCLscoep
application ORCLscoep Oracle Solaris Cluster HA for Oracle External Proxy
|
2.4.6 Add /usr/cluster/bin to /etc/profile
Add /usr/cluster/bin to the PATH environment variable in /etc/profile.
2.4.7 Install the Sun ZFS Storage Appliance plugin for Oracle Solaris Cluster
|
# cd <OSC Installation NFS share dir>/ebs-ssc/ZFSSA-SC-plugin
# pkgadd -d . SUNWsczfsnfs
|
2.5 Configure OSC 3.3u1
Identify cluster interconnect interfaces:
|
# grep 8501 /etc/path_to_inst
"/pci@600/pci@2/pci@0/pci@0/pciex15b3,673c@0/ibport@1,8501,ipib" 1 "ibd"
# grep 8502 /etc/path_to_inst
"/pci@700/pci@2/pci@0/pci@3/pciex15b3,673c@0/ibport@2,8502,ipib" 7 "ibd"
|
On node etc11-02-s10:
|
# export DISPLAY=<X11 display name>
# /usr/cluster/bin/scinstall
|
Check cluster nodes after configuration:
|
# clnode status
=== Cluster Nodes ===
--- Node Status ---
Node Name Status
--------- ------
etc11-01-s10 Online
etc11-02-s10 Online
# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from (latest node reconfiguration) ---
Needed Present Possible
------ ------- --------
2 3 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
--------- ------- -------- ------
etc11-01-s10 1 1 Online
etc11-02-s10 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status
----------- ------- -------- ------
d3 1 1 Online
|
Note: Screen shots for the above tasks of ZFS SA iSCSI, OSC3.3u1 installation, patches, and cluster formation can be found within the SSC Status workspace and the PeopleSoft OOS WP.
Enable SCSI-3 protocol for faster cluster reconfiguration:
|
# clq remove -F d3
Warning: Force option used.
This cluster is at risk of experiencing a serious outage.
Configure a new quorum device to prevent loss of cluster quorum and outages.
# cluster set -p global_fencing=prefer3
Updating shared devices on node 1
Updating shared devices on node 2
# clq add d3
|
2.6 Create the zone clusters
From the global zone on each global cluster node, prepare zone file systems:
|
# zfs create -o mountpoint=/zones BIrpool/zones
|
Next, create the following three zone clusters:
· Create opmn-zc—zone cluster nodes iopmn1 and iopmn2
· Create cm-zc —zone cluster nodes icm1 and icm2
· Create db-zc —zone cluster nodes idb1 and idb2
In the following zone cluster configuration steps, the encrypted root password and name services settings are derived from what is configured in the global zone, so that the zone clusters use the same. The user can change the root password later on or use a different name service. See this documentation referencefor more details:
Oracle Solaris Cluster Software Installation Guide
http://docs.oracle.com/cd/E18728_01/html/821-2845/index.html
2.6.1 Create opmn-zc
Create the opmn-zc zone cluster. This zone cluster is created with zone cluster nodes iopmn1 and iopmn2:
|
# cat /var/tmp/opmn-zc-sroot.conf
create
set zonepath=/zones/opmn-zc
set autoboot=true
add sysid
set root_password=f/ouCDtWx.rIA
set name_service="NIS{domain_name=labtech-sca.us.oracle.com name_server=sca15-1070-jet1.us.oracle.com(10.129.184.238)}"
set nfs4_domain=dynamic
set security_policy=NONE
set system_locale=C
set terminal=xterms
set timezone=US/Pacific
end
add node
set physical-host=etc11-01-s10
set hostname=iopmn1
add net
set address=iopmn1
set physical=ibd3
end
add net
set address=eopmn1
set physical=ixgbe0
end
end
add node
set physical-host=etc11-02-s10
set hostname=iopmn2
add net
set address=iopmn2
set physical=ibd3
end
add net
set address=eopmn2
set physical=ixgbe0
end
end
set limitpriv="default,proc_priocntl,proc_clock_highres"
add net
set address=eweb-lh
end
add net
set address=iopmn1-lh
end
add net
set address=iopmn2-lh
end
commit
# time clzc configure -f /var/tmp/opmn-zc-sroot.conf opmn-zc
real 0m2.13s
user 0m0.09s
sys 0m0.05s
# time clzc install opmn-zc
Waiting for zone install commands to complete on all the nodes of the zone cluster "opmn-zc"...
# zlogin -C opmn-zc
[Connected to zone 'opmn-zc' console]
# time clzc boot opmn-zc
Waiting for zone boot commands to complete on all the nodes of the zone cluster "opmn-zc"...
real 0m2.67s
user 0m0.20s
sys 0m0.02s
|
Add entries to the zone cluster's hosts file:
|
# cat >> /etc/hosts
10.129.186.20 edb1
10.129.186.21 edb2
10.129.186.22 eopmn1
10.129.186.23 eopmn2
10.129.186.24 ecm1
10.129.186.25 ecm2
10.129.186.26 edb-lh
10.129.186.27 eweb-lh.us.oracle.com eweb-lh
10.129.186.28 eopmn1-lh
10.129.186.29 eopmn2-lh
10.129.186.30 ecm1-lh
10.129.186.31 ecm2-lh
192.168.30.1 etc11sn1-stor
192.168.30.31 iopmn1
192.168.30.32 iopmn2
192.168.30.33 iopmn1-lh.us.oracle.com iopmn1-lh
192.168.30.34 iopmn2-lh.us.oracle.com iopmn2-lh
192.168.30.41 icm1.us.oracle.com icm1
192.168.30.42 icm2.us.oracle.com icm2
192.168.30.43 icm1-lh
192.168.30.44 icm2-lh
192.168.30.51 idb1
192.168.30.52 idb2
192.168.30.53 idb-lh.us.oracle.com idb-lh
192.168.30.6 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip
192.168.30.7 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip
^D
|
Confirm configuration:
|
# ypwhich
sca15-1070-jet1.us.oracle.com
# getent hosts etc11-01
10.129.186.11 etc11-01.us.oracle.com
# ping etc11sn1-stor
etc11sn1-stor is alive
|
2.6.2 Create cm-zc
Create the cm-zc zone cluster. This zone cluster is created with zone cluster nodes icm1 and icm2:
|
# cat /var/tmp/cm-zc-sroot.conf
create
set zonepath=/zones/cm-zc
set autoboot=true
add sysid
set root_password=f/ouCDtWx.rIA
set name_service="NIS{domain_name=labtech-sca.us.oracle.com name_server=sca15-1070-jet1.us.oracle.com(10.129.184.238)}"
set nfs4_domain=dynamic
set security_policy=NONE
set system_locale=C
set terminal=xterms
set timezone=US/Pacific
end
add node
set physical-host=etc11-01-s10
set hostname=icm1
add net
set address=icm1
set physical=ibd3
end
add net
set address=ecm1
set physical=ixgbe0
end
end
add node
set physical-host=etc11-02-s10
set hostname=icm2
add net
set address=icm2
set physical=ibd3
end
add net
set address=ecm2
set physical=ixgbe0
end
end
set limitpriv="default,proc_priocntl,proc_clock_highres"
add net
set address=ecm1-lh
end
add net
set address=ecm2-lh
end
add net
set address=icm1-lh
end
add net
set address=icm2-lh
end
commit
# time clzc configure -f /var/tmp/cm-zc-sroot.conf cm-zc
real 0m1.66s
user 0m0.10s
sys 0m0.05s
# time clzc install cm-zc
Waiting for zone install commands to complete on all the nodes of the zone cluster "cm-zc"...
Waiting for zone install commands to complete on all the nodes of the zone cluster "cm-zc"...
real 2m40.51s
user 1m3.58s
sys 1m3.58s
# time clzc boot cm-zc
Waiting for zone boot commands to complete on all the nodes of the zone cluster "cm-zc"...
real 0m2.79s
user 0m0.19s
sys 0m0.04s
|
Add entries to this zone cluster's hosts file:
|
# cat >> /etc/hosts
10.129.186.20 edb1
10.129.186.21 edb2
10.129.186.22 eopmn1
10.129.186.23 eopmn2
10.129.186.24 ecm1
10.129.186.25 ecm2
10.129.186.26 edb-lh
10.129.186.27 eweb-lh.us.oracle.com eweb-lh
10.129.186.28 eopmn1-lh
10.129.186.29 eopmn2-lh
10.129.186.30 ecm1-lh
10.129.186.31 ecm2-lh
192.168.30.1 etc11sn1-stor
192.168.30.31 iopmn1
192.168.30.32 iopmn2
192.168.30.33 iopmn1-lh.us.oracle.com iopmn1-lh
192.168.30.34 iopmn2-lh.us.oracle.com iopmn2-lh
192.168.30.41 icm1.us.oracle.com icm1
192.168.30.42 icm2.us.oracle.com icm2
192.168.30.43 icm1-lh
192.168.30.44 icm2-lh
192.168.30.51 idb1
192.168.30.52 idb2
192.168.30.53 idb-lh.us.oracle.com idb-lh
192.168.30.6 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip
192.168.30.7 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip
^D
|
Confirm:
|
# ypwhich
sca15-1070-jet1.us.oracle.com
# getent hosts etc11-01
10.132.132.110 etc11-01.us.oracle.com
# ping etc11sn1-stor
etc11sn1-stor is alive
|
2.6.3 Create db-zc
Create the db-zc zone cluster. This zone cluster is created with zone cluster nodes idb1 and idb2:
|
# cat /var/tmp/db-zc-sroot.conf
create
set zonepath=/zones/db-zc
set autoboot=true
add sysid
set root_password=f/ouCDtWx.rIA
set name_service="NIS{domain_name=labtech-sca.us.oracle.com name_server=sca15-1070-jet1.us.oracle.com(10.129.184.238)}"
set nfs4_domain=dynamic
set security_policy=NONE
set system_locale=C
set terminal=xterms
set timezone=US/Pacific
end
add node
set physical-host=etc11-01-s10
set hostname=idb1
add net
set address=idb1
set physical=ibd3
end
add net
set address=edb1
set physical=ixgbe0
end
end
add node
set physical-host=etc11-02-s10
set hostname=idb2
add net
set address=idb2
set physical=ibd3
end
add net
set address=edb2
set physical=ixgbe0
end
end
set limitpriv="default,proc_priocntl,proc_clock_highres"
add net
set address=edb-lh
end
add net
set address=idb-lh
end
commit
# time clzc configure -f /var/tmp/db-zc-sroot.conf db-zc
real 0m1.66s
user 0m0.10s
sys 0m0.05s
# time clzc install db-zc
Waiting for zone install commands to complete on all the nodes of the zone cluster "db-zc"...
Waiting for zone install commands to complete on all the nodes of the zone cluster "db-zc"...
real 2m46.65s
user 1m4.99s
sys 1m3.53s
# time clzc boot db-zc
Waiting for zone boot commands to complete on all the nodes of the zone cluster "db-zc"...
real 0m3.07s
user 0m0.19s
sys 0m0.03s
|
Add entries to this zone cluster’s hosts file:
|
# cat >> /etc/hosts
10.129.186.20 edb1
10.129.186.21 edb2
10.129.186.22 eopmn1
10.129.186.23 eopmn2
10.129.186.24 ecm1
10.129.186.25 ecm2
10.129.186.26 edb-lh
10.129.186.27 eweb-lh.us.oracle.com eweb-lh
10.129.186.28 eopmn1-lh
10.129.186.29 eopmn2-lh
10.129.186.30 ecm1-lh
10.129.186.31 ecm2-lh
192.168.30.1 etc11sn1-stor
192.168.30.31 iopmn1
192.168.30.32 iopmn2
192.168.30.33 iopmn1-lh.us.oracle.com iopmn1-lh
192.168.30.34 iopmn2-lh.us.oracle.com iopmn2-lh
192.168.30.41 icm1.us.oracle.com icm1
192.168.30.42 icm2.us.oracle.com icm2
192.168.30.43 icm1-lh
192.168.30.44 icm2-lh
192.168.30.51 idb1
192.168.30.52 idb2
192.168.30.53 idb-lh.us.oracle.com idb-lh
192.168.30.6 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip
192.168.30.7 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip
^D
|
Confirm:
|
# ypwhich
sca15-1070-jet1.us.oracle.com
# getent hosts etc11-01
10.132.132.110 etc11-01.us.oracle.com
# ping etc11sn1-stor
etc11sn1-stor is alive
|
Add SPARC SuperCluster recommended tuning:
|
# cat >> /etc/system
* SSC tuning for EBS
set nfs:nfs3_bsize = 131072
set nfs:nfs4_bsize = 131072
set rpcmod:clnt_max_conns = 8
set zfs:zfs_arc_max = 0x1000000000
^D
# cat > /etc/rc2.d/S99ndd
ndd -set /dev/tcp tcp_xmit_hiwat 1048576
ndd -set /dev/tcp tcp_recv_hiwat 1048576
ndd -set /dev/tcp tcp_cwnd_max 2097152
ndd -set /dev/tcp tcp_max_buf 2097152
|
2.7 Create the initial RGs and LHs within the zone clusters
Create the initial resource groups and logical hosts within the zone clusters that were just created:
· Zone cluster opmn-zc
· Zone cluster cm-zc
· Zone cluster db-zc
2.7.1 Zone cluster opmn-zc
Create the resource groups and logical hosts for the opmn-zc zone cluster:
|
# hostname
iopmn1
# clrg create -p failback=true opmn1lh-rg
# clrslh create -g opmn1lh-rg iopmn1-lh
# clrg online -eM opmn1lh-rg
# clrg create -n 1 opmn1-rg
# clrg online -eM opmn1-rg
# clrg create -p failback=true -n 2,1 opmn2lh-rg
# clrslh create -g opmn2lh-rg iopmn2-lh
# clrg online -eM opmn2lh-rg
# clrg create -n 2 opmn2-rg
# clrg online -eM opmn2-rg
# clrg create web-rg
# clrslh create -g web-rg eweb-lh
# clrg online -eM web-rg
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
opmn1lh-rg iopmn1 No Online
iopmn2 No Offline
opmn1-rg iopmn1 No Online
opmn2lh-rg iopmn2 No Online
iopmn1 No Offline
opmn2-rg iopmn2 No Online
web-rg iopmn1 No Online
iopmn2 No Offline
# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
iopmn1-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline
iopmn2-lh iopmn2 Online Online - LogicalHostname online.
iopmn1 Offline Offline
eweb-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline
|
2.7.2 Zone cluster cm-zc
Create the resource groups and logical hosts for the cm-zc zone cluster:
|
# hostname
icm1
# clrg create -S cm-rg
# clrg online -eM cm-rg
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
cm-rg icm1 No Online
icm2 No Online
|
2.7.3 Zone cluster db-zc
Create the resource groups and logical hosts for the db-zc zone cluster:
|
# hostname
idb1
# clrg create db-rg
# clrslh create -g db-rg edb-lh
# clrslh create -g db-rg idb-lh
# clrg online -eM db-rg
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
db-rg idb1 No Online
idb2 No Offline
# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
idb-lh idb1 Online Online - LogicalHostname online.
idb2 Offline Offline
edb-lh idb1 Online Online - LogicalHostname online.
idb2 Offline Offline
|
2.8 Configure the ZFS SA file systems for the zone clusters
The steps in this section add entries to each of the zone cluster's /etc/vfstab files. The following entries will be added to all zone clusters:
|
# EBS shared file system
etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
|
In addition, the following entries will be added to Zone Cluster db-zc:
|
# EBS DB file systems
etc11sn1-stor:/export/etc11/orainst - /orainst nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid
etc11sn1-stor:/export/etc11/oradata - /oradata nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
etc11sn1-stor:/export/etc11/oralog - /oralog nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
etc11sn1-stor:/export/etc11/oradump - /oradump nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
|
Configuration for Zone Cluster opmn-zc:
|
# zlogin opmn-zc
[Connected to zone 'opmn-zc' pts/5]
Last login: Thu Feb 16 09:35:04 on pts/4
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# ksh -o emacs
# cat >> /etc/vfstab
# EBS shared file system
etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
# mkdir /ebsfs
# mount /ebsfs
# df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs
16T 31K 16T 1% /ebsfs
|
Configuration for Zone Cluster cm-zc:
|
# zlogin cm-zc
[Connected to zone 'cm-zc' pts/5]
Last login: Thu Feb 16 10:12:18 on pts/4
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# ksh -o emacs
# cat >> /etc/vfstab
# EBS shared file system
etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
# mkdir /ebsfs
# mount /ebsfs
# df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs
16T 31K 16T 1% /ebsfs
|
Configuration for Zone Cluster db-zc:
|
# zlogin db-zc
[Connected to zone 'db-zc' pts/5]
Last login: Thu Feb 16 10:13:24 on pts/4
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# ksh -o emacs
# cat >> /etc/vfstab
# EBS shared file system
etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
# EBS DB file systems
etc11sn1-stor:/export/etc11/orainst - /orainst nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid
etc11sn1-stor:/export/etc11/oradata - /oradata nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
etc11sn1-stor:/export/etc11/oralog - /oralog nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
etc11sn1-stor:/export/etc11/oradump - /oradump nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
# mkdir /ebsfs /orainst /oradata /oralog /oradump
# mountall
mount: /tmp is already mounted or swap is busy
# df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/oradata
16T 31K 16T 1% /oradata
etc11sn1-stor:/export/etc11/oradump
16T 31K 16T 1% /oradump
etc11sn1-stor:/export/etc11/orainst
16T 31K 16T 1% /orainst
etc11sn1-stor:/export/etc11/ebsfs
16T 31K 16T 1% /ebsfs
etc11sn1-stor:/export/etc11/oralog
16T 31K 16T 1% /oralog
|
Confirm zone cluster status:
|
# clzc status
=== Zone Clusters ===
--- Zone Cluster Status ---
Name Node Name Zone HostName Status Zone Status
---- --------- ------------- ------ -----------
cm-zc etc11-01-s10 icm1 Online Running
etc11-02-s10 icm2 Online Running
db-zc etc11-01-s10 idb1 Online Running
etc11-02-s10 idb2 Online Running
opmn-zc etc11-01-s10 iopmn1 Online Running
etc11-02-s10 iopmn2 Online Running
|
2.9 (Optional) Configure root remote access for telnet and ssh
Change /etc/default/login on each zone cluster as follows:
|
# grep CONSOLE= /zones/*/root/etc/default/login
/zones/cm-zc/root/etc/default/login:#CONSOLE=/dev/console
/zones/db-zc/root/etc/default/login:#CONSOLE=/dev/console
/zones/opmn-zc/root/etc/default/login:#CONSOLE=/dev/console
/zones/*-zc/root/etc/ssh/sshd_config
#PermitRootLogin no
PermitRootLogin yes
# for zc in `clzc list`; do set -x; zlogin $zc svcadm restart ssh; set +x; done
+ zlogin cm-zc svcadm restart ssh
+ zlogin db-zc svcadm restart ssh
+ zlogin opmn-zc svcadm restart ssh
|
2.10 Check file systems
Check the file systems on each zone cluster:
|
# for zc in `clzc list`; do set -x; zlogin $zc df -h -F nfs; set +x; done
+ zlogin cm-zc df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs 16T 31K 16T 1% /ebsfs
+ zlogin db-zc df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/oradata 16T 31K 16T 1% /oradata
etc11sn1-stor:/export/etc11/oradump 16T 31K 16T 1% /oradump
etc11sn1-stor:/export/etc11/orainst 16T 31K 16T 1% /orainst
etc11sn1-stor:/export/etc11/ebsfs 16T 31K 16T 1% /ebsfs
etc11sn1-stor:/export/etc11/oralog 16T 31K 16T 1% /oralog
+ zlogin opmn-zc df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs 16T 31K 16T 1% /ebsfs
|
2.11 Configure the ZFS SA for fencing
Configure the ZFS Storage Appliance for fencing:
|
oracle> ssh -l root etc11-sn1
etc11-sn1:> shares
etc11-sn1:shares> ls
Properties:
pool = ssc
Projects:
QuorumProject
default
ebs-apps
ebs-db
Children:
replication => Manage remote replication
schema => Define custom property schema
etc11-sn1:shares> select ebs-apps
etc11-sn1:shares ebs-apps> ls
Properties:
aclinherit = restricted
aclmode = discard
atime = true
checksum = fletcher4
compression = off
dedup = false
compressratio = 100
copies = 1
creation = Thu Feb 16 2012 18:19:00 GMT+0000 (UTC)
logbias = latency
mountpoint = /export/etc11
quota = 0
readonly = false
recordsize = 128K
reservation = 0
rstchown = true
secondarycache = all
nbmand = false
sharesmb = off
sharenfs = sec=sys,rw,root=@192.168.30.0/24
snapdir = hidden
vscan = false
sharedav = off
shareftp = off
sharesftp = off
sharetftp =
pool = ssc
canonical_name = ssc/local/ebs-apps
default_group = other
default_permissions = 700
default_sparse = false
default_user = nobody
default_volblocksize = 8K
default_volsize = 0
exported = true
nodestroy = false
space_data = 55.3G
space_unused_res = 0
space_unused_res_shares = 0
space_snapshots = 15.7G
space_available = 15.8T
space_total = 71.0G
origin =
Shares:
Filesystems:
NAME SIZE MOUNTPOINT
ebsfs 71.0G /export/etc11/ebsfs
Children:
groups => View per-group usage and manage group
quotas
replication => Manage remote replication
snapshots => Manage snapshots
users => View per-user usage and manage user quotas
etc11-sn1:shares ebs-apps> get sharenfs
sharenfs = sec=sys,rw,root=@192.168.30.0/24
ZC IP addresses to be enabled for fencing:
192.168.30.31 iopmn1
192.168.30.32 iopmn2
192.168.30.41 icm1
192.168.30.42 icm2
192.168.30.51 idb1
192.168.30.52 idb2
set sharenfs="sec=sys,root=@192.168.30.31/32:@192.168.30.32/32:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.51/32:@192.168.30.52/32,rw=@192.168.30.31/32:@192.168.30.32/32:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.51/32:@192.168.30.52/32"
etc11-sn1:shares ebs-apps> set readonly="false"
readonly = false (uncommitted)
etc11-sn1:shares ebs-apps> commit
etc11-sn1:shares ebs-apps> get sharenfs
sharenfs = sec=sys,root=@192.168.30.31/32:@192.168.30.32/32:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.51/32:@192.168.30.52/32,rw=@192.168.30.31/32:@192.168.30.32/32:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.51/32:@192.168.30.52/32
|
Verify on ZC:
Check GUI: Shared Mode is set to None:
|
etc11-sn1:shares ebs-apps> cd ../..
etc11-sn1:> shares
etc11-sn1:shares> select ebs-db
etc11-sn1:shares ebs-db> get readonly sharenfs
readonly = false
sharenfs = sec=sys,rw,root=@192.168.30.0/24
192.168.30.51 idb1
192.168.30.52 idb2
set sharenfs="sec=sys,root=@192.168.30.51/32:@192.168.30.52/32,rw=@192.168.30.51/32:@192.168.30.52/32"
etc11-sn1:shares ebs-db> set sharenfs="sec=sys,root=@192.168.30.51/32:@192.168.30.52/32,rw=@192.168.30.51/32:@192.168.30.52/32"
sharenfs = sec=sys,root=@192.168.30.51/32:@192.168.30.52/32,rw=@192.168.30.51/32:@192.168.30.52/32 (uncommitted)
etc11-sn1:shares ebs-db> commit
etc11-sn1:shares ebs-db> get readonly sharenfs
readonly = false
sharenfs = sec=sys,root=@192.168.30.51/32:@192.168.30.52/32,rw=@192.168.30.51/32:@192.168.30.52/32
etc11-sn1:shares ebs-db> cd ../..
etc11-sn1:> maintenance
etc11-sn1:maintenance> workflows
etc11-sn1:maintenance workflows> ls
Properties:
showhidden = false
Workflows:
WORKFLOW NAME OWNER SETID ORIGIN
workflow-000 Configure for Oracle Solaris Cluster NFS root false Oracle Corporation
workflow-001 Unconfigure Oracle Solaris Cluster NFS root false Oracle Corporation
workflow-002 Configure for Oracle Enterprise Manager Monitoring root false Sun Microsystems, Inc.
workflow-003 Unconfigure Oracle Enterprise Manager Monitoring root false Sun Microsystems, Inc.
etc11-sn1:maintenance workflows> select workflow-000
etc11-sn1:maintenance workflow-000> execute
etc11-sn1:maintenance workflow-000 execute (uncommitted)> set password=labuser
password = *******
etc11-sn1:maintenance workflow-000 execute (uncommitted)> set changePassword=false
changePassword = false
etc11-sn1:maintenance workflow-000 execute (uncommitted)> commit
error: invalid property value "false" for "changePassword": Oracle Solaris
Cluster NFS Configuration already complete, must explicitly choose to
change the password.
error: cannot execute workflow
etc11-sn1:maintenance workflow-000 execute (uncommitted)> set changePassword=true
changePassword = true
etc11-sn1:maintenance workflow-000 execute (uncommitted)> commit
Password successfully changed.
etc11-sn1:maintenance workflow-000> ls
Properties:
name = Configure for Oracle Solaris Cluster NFS
description = Sets up environment for Oracle Solaris Cluster NFS
owner = root
origin = Oracle Corporation
setid = false
alert = false
scheduled = false
etc11-sn1:> configuration users
etc11-sn1:configuration users> show
Users:
NAME USERNAME UID TYPE
Oracle Solaris Cluster Agent osc_agent 2000000000 Loc
Super-User root 0 Loc
|
2.12 Configure each zone cluster for fencing and add storage resource
Configure the three zone cluster for fencing and add storage resources:
· Zone cluster opmn-zc
· Zone cluster cm-zc
· Zone cluster db-zc
2.12.1 Zone cluster opmn-zc
Configure fencing on zone cluster opmn-zc:
|
# zlogin opmn-zc
[Connected to zone 'opmn-zc' pts/8]
Last login: Thu Mar 1 05:35:48 from 10.175.56.195
Oracle Corporation SunOS 5.10 Generic Patch January 2005
You have new mail.
# ksh -o emacs
# getent hosts `clnode list`
192.168.30.31 iopmn1 loghost
192.168.30.31 iopmn1 loghost
192.168.30.32 iopmn2
# clnas add -t sun_uss -p userid=osc_agent etc11sn1-stor
Enter password:
# clnas set -p "nodeIPs{iopmn1}"=192.168.30.31 -p "nodeIPs{iopmn2}"=192.168.32 etc11sn1-stor
# clnas find-dir etc11sn1-stor
=== NAS Devices ===
Nas Device: etc11sn1-stor
Type: sun_uss
Unconfigured Project: ssc/local/QuorumProject
Unconfigured Project: ssc/local/ebs-apps
# clnas add-dir -d ssc/local/ebs-apps etc11sn1-stor
# clnas show -v -d all
=== NAS Devices ===
Nas Device: etc11sn1-stor
Type: sun_uss
userid: osc_agent
nodeIPs{iopmn2}: 192.168.30.32
nodeIPs{iopmn1}: 192.168.30.31
Project: ssc/local/ebs-apps
File System: /export/etc11/ebsfs
|
Add storage resources on zone cluster opmn-zc:
|
# clrg create -S scalmnt-rg
# df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs
16T 55G 16T 1% /ebsfs
# clrt register ScalMountPoint
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/ebsfs -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/ebsfs ebs-fs-rs
# clrg online -eM scalmnt-rg
# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
iopmn1-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
iopmn2-lh iopmn2 Online Online - LogicalHostname online.
iopmn1 Offline Offline - LogicalHostname offline.
eweb-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
ebs-fs-rs iopmn1 Online Online
iopmn2 Online Online
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
opmn1lh-rg iopmn1 No Online
iopmn2 No Offline
opmn2lh-rg iopmn2 No Online
iopmn1 No Offline
web-rg iopmn1 No Online
iopmn2 No Offline
opmn1-rg iopmn1 No Online
opmn2-rg iopmn2 No Online
scalmnt-rg iopmn1 No Online
iopmn2 No Online
# clrg set -p RG_affinities=++scalmnt-rg web-rg opmn1-rg opmn2-rg
|
2.12.2 Zone cluster cm-zc
Configure fencing on zone cluster cm-zc:
|
# zlogin cm-zc
[Connected to zone 'cm-zc' pts/8]
Last login: Thu Mar 1 05:35:48 from 10.175.56.195
Oracle Corporation SunOS 5.10 Generic Patch January 2005
You have new mail.
# ksh -o emacs
# getent hosts `clnode list`
192.168.30.41 icm1 loghost icm1
192.168.30.41 icm1 loghost icm1
192.168.30.42 icm2.us.oracle.com icm2
# clnas add -t sun_uss -p userid=osc_agent etc11sn1-stor
Enter password:
# clnas set -p "nodeIPs{icm1}"=192.168.30.41 -p "nodeIPs{icm2}"=192.168.30.42 etc11sn1-stor
# clnas find-dir etc11sn1-stor
=== NAS Devices ===
Nas Device: etc11sn1-stor
Type: sun_uss
Unconfigured Project: ssc/local/QuorumProject
Unconfigured Project: ssc/local/ebs-apps
# clnas add-dir -d ssc/local/ebs-apps etc11sn1-stor
# clnas show -v -d all
=== NAS Devices ===
Nas Device: etc11sn1-stor
Type: sun_uss
userid: osc_agent
nodeIPs{icm2}: 192.168.30.42
nodeIPs{icm1}: 192.168.30.41
Project: ssc/local/ebs-apps
File System: /export/etc11/ebsfs
|
Add storage resources on zone cluster cm-zc:
|
# clrg create -S scalmnt-rg
# df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs
16T 55G 16T 1% /ebsfs
# clrt register ScalMountPoint
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/ebsfs -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/ebsfs ebs-fs-rs
# clrg online -eM scalmnt-rg
# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
ebs-fs-rs icm1 Online Online
icm2 Online Online
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
cm-rg icm1 No Online
icm2 No Online
scalmnt-rg icm1 No Online
icm2 No Online
# clrg set -p RG_affinities=++scalmnt-rg cm-rg
|
2.12.3 Zone cluster db-zc
Configure fencing on zone cluster db-zc:
|
# zlogin db-zc
[Connected to zone 'db-zc' pts/8]
Last login: Thu Mar 1 05:35:48 from 10.175.56.195
Oracle Corporation SunOS 5.10 Generic Patch January 2005
You have new mail.
# ksh -o emacs
# getent hosts `clnode list`
192.168.30.51 idb1 loghost
192.168.30.51 idb1 loghost
192.168.30.52 idb2
# clnas add -t sun_uss -p userid=osc_agent etc11sn1-stor
Enter password:
(labuser)
# clnas set -p "nodeIPs{idb1}"=192.168.30.51 -p "nodeIPs{idb2}"=192.168.30.52 etc11sn1-stor
# clnas find-dir etc11sn1-stor
=== NAS Devices ===
Nas Device: etc11sn1-stor
Type: sun_uss
Unconfigured Project: ssc/local/QuorumProject
Unconfigured Project: ssc/local/ebs-apps
Unconfigured Project: ssc/local/ebs-db
# clnas add-dir -d ssc/local/ebs-apps,ssc/local/ebs-db etc11sn1-stor
# clnas show -v -d all
=== NAS Devices ===
Nas Device: etc11sn1-stor
Type: sun_uss
userid: osc_agent
nodeIPs{idb2}: 192.168.30.52
nodeIPs{idb1}: 192.168.30.51
Project: ssc/local/ebs-apps
File System: /export/etc11/ebsfs
Project: ssc/local/ebs-db
File System: /export/etc11/oralog
File System: /export/etc11/orainst
File System: /export/etc11/oradump
File System: /export/etc11/oradata
|
Configure storage resources on zone cluster db-zc:
|
# clrg create -S scalmnt-rg
# df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/oradata
16T 31G 16T 1% /oradata
etc11sn1-stor:/export/etc11/oradump
16T 11G 16T 1% /oradump
etc11sn1-stor:/export/etc11/orainst
16T 63G 16T 1% /orainst
etc11sn1-stor:/export/etc11/ebsfs
16T 55G 16T 1% /ebsfs
etc11sn1-stor:/export/etc11/oralog
16T 4.0G 16T 1% /oralog
|
Edit /etc/vfstab to set mount at boot to no:
|
# grep etc11sn1-stor /zones/*-zc/root/etc/vfstab
/zones/cm-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
/zones/db-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
/zones/db-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/orainst - /orainst nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid
/zones/db-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/oradata - /oradata nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
/zones/db-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/oralog - /oralog nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
/zones/db-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/oradump - /oradump nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirectio
/zones/opmn-zc/root/etc/vfstab:etc11sn1-stor:/export/etc11/ebsfs - /ebsfs nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
|
Create cluster resources:
|
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/ebsfs -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/ebsfs ebs-fs-rs
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/orainst -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/orainst orainst-fs-rs
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/oradata -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/oradata oradata-fs-rs
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/oralog -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/oralog oralog-fs-rs
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/oradump -x FileSystemType=nas -x TargetFileSystem=etc11sn1-stor:/export/etc11/oradump oradump-fs-rs
# clrg online -eM scalmnt-rg
# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
idb-lh idb1 Offline Offline - LogicalHostname offline.
idb2 Online Online - LogicalHostname online.
oradump-fs-rs idb1 Online Online
idb2 Online Online
oralog-fs-rs idb1 Online Online
idb2 Online Online
oradata-fs-rs idb1 Online Online
idb2 Online Online
orainst-fs-rs idb1 Online Online
idb2 Online Online
ebs-fs-rs idb1 Online Online
idb2 Online Online
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
db-rg idb1 No Offline
idb2 No Online
scalmnt-rg idb1 No Online
idb2 No Online
# clrg set -p RG_affinities=++scalmnt-rg db-rg
etc11-01-s10:db-zc - Ignoring ORACLE_HOME set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-01-s10:db-zc - Ignoring ORACLE_SID set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-01-s10:db-zc - Ignoring command execution `<command>`
etc11-02-s10:db-zc - Ignoring ORACLE_HOME set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-02-s10:db-zc - Ignoring ORACLE_SID set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-02-s10:db-zc - Ignoring command execution `<command>`
|
3. System Configuration
This section reviews the system configuration that has been performed up to this point. See Appendix A: Zone Cluster Listings on page 125 for zone cluster listings. Also note that db-zc is a transient zone cluster and can be removed once the database has been migrated to Oracle RAC with ASM in the Solaris 11 LDoms.
The Solaris 10 domains of the EBS rig (etc11) have been installed with Solaris Cluster 3.3u1 and its patches in patch train #3, all from SSC build 09. They also have the SPARC SuperCluster recommended NFS tuning in place.
|
# clnode status
=== Cluster Nodes ===
--- Node Status ---
Node Name Status
--------- ------
etc11-01-s10 Online
etc11-02-s10 Online
|
Zone clusters have been created as follows:
|
# clzc status
=== Zone Clusters ===
--- Zone Cluster Status ---
Name Node Name Zone HostName Status Zone Status
---- --------- ------------- ------ -----------
cm-zc etc11-01-s10 icm1 Online Running
etc11-02-s10 icm2 Online Running
db-zc etc11-01-s10 idb1 Online Running
etc11-02-s10 idb2 Online Running
opmn-zc etc11-01-s10 iopmn1 Online Running
etc11-02-s10 iopmn2 Online Running
|
The db-zc zone cluster is used to support the initial database from the EBS install process or the database copy from the workload setup.
The following example shows the relationship between the zone names and hostnames. For non-global zones, the zone has the name of its zone cluster. But its hostname is as per the listing above. For example:
|
# hostname
etc11-01-s10
# zonename
global
# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
1 opmn-zc running /zones/opmn-zc cluster shared
2 db-zc running /zones/db-zc cluster shared
3 cm-zc running /zones/cm-zc cluster shared
#
# zlogin opmn-zc
[Connected to zone 'opmn-zc' pts/6]
Last login: Thu Feb 16 11:03:19 on pts/5
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# hostname
iopmn1
# zonename
opmn-zc
|
Each zone cluster node has its hostname i<name> which is the network hostname on the internal IpoIB subnet 192.168.30.0. But it also has a network hostname e<name> on the 10GbE network which is on OWAN.
Resource groups have been created for LHs and future resources for Web HTTP server, OPMN servers and CM servers:
|
# clrg status -Z all
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
cm-zc:cm-rg icm1 No Online
icm2 No Online
db-zc:db-rg idb1 No Online
idb2 No Offline
opmn-zc:opmn1lh-rg iopmn1 No Online
iopmn2 No Offline
opmn-zc:opmn1-rg iopmn1 No Online
opmn-zc:opmn2lh-rg iopmn2 No Online
iopmn1 No Offline
opmn-zc:opmn2-rg iopmn2 No Online
opmn-zc:web-rg iopmn1 No Online
iopmn2 No Offline
|
Logical hosts are also provided for both IPoIB network and the 10GbE network:
|
bash-3.2# clrs status -t SUNW.LogicalHostname -Z all
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
db-lh idb1 Offline Offline
idb2 Online Online - LogicalHostname online.
opmn1-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
opmn2-lh iopmn2 Online Online - LogicalHostname online.
iopmn1 Offline Offline
web-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
|
Logical hostname use the same convention as the regular hostnames: i<lhname> is for IPoIB and e<lhname> is for 10GbE (and in OWAN name services).
The only accessible logical host via 10GbE OWAN is via the web-lh resource, i.e. eweb-lh.
|
bash-3.2# clrs show -p Hostnamelist -Z opmn-zc web-lh
=== Resources ===
Resource: opmn-zc:web-lh
--- Standard and extension properties ---
HostnameList: eweb-lh
Class: extension
Description: List of hostnames this resource manages
Per-node: False
Type: stringarray
bash-3.2#
bash-3.2# grep eweb-lh /etc/hosts
10.129.186.27 eweb-lh.us.oracle.com eweb-lh
|
The other logical hosts are all on the IpoIB network or i<lhname> hostnames:
|
bash-3.2# clrs show -p Hostnamelist -Z opmn-zc opmn1-lh | grep HostnameList
HostnameList: iopmn1-lh
bash-3.2# clrs show -p Hostnamelist -Z opmn-zc opmn2-lh | grep HostnameList
HostnameList: iopmn2-lh
bash-3.2# clrs show -p Hostnamelist -Z db-zc db-lh | grep HostnameList
HostnameList: idb-lh
|
EBS components installation in a given zone cluster node should use the logical host online on that node, instead of the "fixed" hostname. The one exception is the CM components in PCP mode.
While there is a duality of networks for communication between components, the expectation is that at the end only the traffic from external clients (workload drivers) to the Web HTTP Server would be on the 10GbE network. From there any other traffics would be on the IPoIB network.
As to NFS file systems from the ZFS Storage Appliance:
|
# for zc in `clzc list`; do set -x; zlogin $zc df -h -F nfs; echo; set +x; done
+ zlogin cm-zc df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs 16T 31K 16T 1% /ebsfs
+ zlogin db-zc df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/oradata 16T 31K 16T 1% /oradata
etc11sn1-stor:/export/etc11/oradump 16T 31K 16T 1% /oradump
etc11sn1-stor:/export/etc11/orainst 16T 31K 16T 1% /orainst
etc11sn1-stor:/export/etc11/ebsfs 16T 31K 16T 1% /ebsfs
etc11sn1-stor:/export/etc11/oralog 16T 31K 16T 1% /oralog
+ zlogin opmn-zc df -h -F nfs
Filesystem size used avail capacity Mounted on
etc11sn1-stor:/export/etc11/ebsfs 16T 31K 16T 1% /ebsfs
|
Note the following:
· The mount points in the ZFS SA has the etc11 subdirectory as per the best practice for backup and restore.
· The mounts are done via the IPoIB network (etc11sn1-stor is the ZFS SA active head host name on the IPoIB network).
· Only the db-zc nodes have the access to the additional file systems which are used to store a database deployment; these file systems are not visible to other zone clusters.
4. Oracle E-Business Suite 12.1.1 Installation
The following steps are followed to install Oracle E-Business Suite 12.1.1:
· Consult MOS Note 761568.1
· Host Names Settings
· Install Oracle E-Business Suite
· Oracle E-Business Suite Post Installation
· ZFS SA Snapshot
4.1 Consult MOS Note 761568.1
Before starting the E-Business Suite installation, consult the following MOS Note:
· E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Oracle Solaris on SPARC (64-bit) [ID 761568.1]
Because MOS Notes are subject to change, it is important to reference the most recent information on installing E-Business Suite on the SPARC platform. For that reason, specific configuration steps related to the MOS Note are not included in this document. The reader is referred to the MOS Note for the latest information.
4.2 Host Names Settings
Each zone cluster node (idb1, idb2, iopmn2, iopmn2, icm1 and icm2) has fully qualified domain name entries for the logical hosts and physical hosts for icm1 and icm2. Previously (see section 2.1), hostnames were added to the /etc/hosts files on Solaris 10 domains in the global zone. Update the /etc/hosts file to add the fully qualified names. For example:
10.129.186.27 eweb-lh.us.oracle.com eweb-lh
192.168.30.33 iopmn1-lh.us.oracle.com iopmn1-lh
192.168.30.34 iopmn2-lh.us.oracle.com iopmn2-lh
192.168.30.41 icm1.us.oracle.com icm1
192.168.30.42 icm2.us.oracle.com icm2
192.168.30.53 idb-lh.us.oracle.com idb-lh
4.3 Install Oracle E-Business Suite
In order to deploy Oracle E-Business Suite across the various zone clusters, this procedure configures a multi-node deployment within Oracle E-Business Suite. That involves creating a multi-node configuration file using the installation tool rapidwiz. Once defined, this multi-node configuration file is used to install Oracle E-Business Suite 12.1.1 within each appropriate zone cluster.
This procedure builds the multit-node configuration file on the db-zc zone cluster. In order to speed up the installation screen shots, this procedure also starts a VNC Server on zone cluster node idb1.
4.3.1 Create DB user and directories (idb1 & idb2)
Create the database users and groups on zone cluster nodes idb1 and idb2:
|
# bash-3.2# uname -n
idb1
bash-3.2# groupadd -g 1000 dba
bash-3.2# useradd -u 1000 -g 1000 -d /orainst oracle
bash-3.2#
bash-3.2# uname -n
idb2
bash-3.2# groupadd -g 1000 dba
bash-3.2# useradd -u 1000 -g 1000 -d /orainst oracle
bash-3.2#
|
Create database directories, and set permissions and ownership:
|
# bash-3.2# uname -n
idb1
bash-3.2# mkdir -p /orainst/d01/oracle/PROD
bash-3.2# mkdir -p /oradata/d01/oracle/PROD
bash-3.2# mkdir -p /oradump/d01/oracle/PROD
bash-3.2# mkdir -p /oralog/d01/oracle/PROD
bash-3.2# chmod -R 755 /orainst
bash-3.2# chmod -R 755 /oradata
bash-3.2# chmod -R 755 /oradump
bash-3.2# chmod -R 755 /oralog
bash-3.2# chown -R oracle:dba /orainst
bash-3.2# chown -R oracle:dba /oradata
bash-3.2# chown -R oracle:dba /oradump
bash-3.2# chown -R oracle:dba /oralog
|
4.3.2 Start VNC Server
Start the VNC Server:
|
# bash-3.2# uname -n
idb1
bash-3.2# cat .profile
PATH=${PATH}:/usr/cluster/bin:/usr/X11/bin:/usr/openwin/bin:/usr/ccs/bin; export PATH
MANPATH=/usr/man:/usr/cluster/man; export MANPATH
bash-3.2#
bash-3.2# . .profile
bash-3.2# vncserver :10
You will require a password to access your desktops.
Password:
Verify:
xauth: creating new authority file //.Xauthority
New 'idb1:10 ()' desktop is idb1:10
Creating default startup script //.vnc/xstartup
Starting Applications specified in //.vnc/xstartup
Log file is //.vnc/idb1:10.log
bash-3.2#
|
4.3.3 Ensure the Oracle E-Business Suite 12.1.1 software is available
Confirm the Oracle E-Business Suite 12.1.1 software is available:
|
# bash-3.2# uname -n
idb1
bash-3.2# dfshares ptata1
RESOURCE SERVER ACCESS TRANSPORT
ptata1:/swlib/ebs12.1.1 ptata1 - -
ptata1:/swlib/OC ptata1 - -
ptata1:/swlib/ora11.2.0.3 ptata1 - -
bash-3.2#
bash-3.2# mount ptata1:/swlib/ebs12.1.1 /mnt
|
4.3.4 Run rapidwiz to Build the Oracle E-Business Suite Configuration File
From a VNC client issue step through the following screens. Note, however, that this procedure just calls rapidwiz without any parameters:
Note, before clicking OK, be sure to save the configuration file.
|
bash-3.2# uname -n
idb1
bash-3.2#
bash-3.2# mkdir -p /var/tmp/EBS
bash-3.2# cd /tmp/02200413
bash-3.2# ls -l
total 1152
-rw-r--r-- 1 root root 19663 Feb 22 06:07 02220601.log
-rw-r--r-- 1 root root 50778 Feb 22 06:07 conf_PROD.txt
-rwxr-xr-x 1 root root 93616 Feb 22 06:06 eweb-lh_PROD_apps.xml
-rwxr-xr-x 1 root root 93097 Feb 22 06:07 icm1_PROD_apps.xml
-rwxr-xr-x 1 root root 93097 Feb 22 06:07 icm2_PROD_apps.xml
-rwxr-xr-x 1 oracle dba 13534 Feb 22 06:04 idb-lh_PROD_db.xml
-rwxr-xr-x 1 root root 93710 Feb 22 06:06 iopmn1-lh_PROD_apps.xml
-rwxr-xr-x 1 root root 93710 Feb 22 06:06 iopmn2-lh_PROD_apps.xml
bash-3.2# cp conf_PROD.txt /var/tmp/EBS
bash-3.2# chmod 777 /var/tmp/EBS
bash-3.2# ls -l /var/tmp/EBS
total 1
-rw-r--r-- 1 root root 50778 Feb 22 06:08 conf_PROD.txt
bash-3.2#
|
4.3.5 Install the DB Tier (idb-lh)
Using the configuration file from the previous step (conf_PROD.txt), install the DB Tier on the node where idb-lh is currently online.
Note: the following steps use the command rapidwiz -servername idb-lh.
Note: the error message (shown in the following screen shot) occurs because a Central Inventory does not exist. You can ignore the error message
At this point the DB Tier installation has been completed. If required, you can verify this. The local inventory should be located within ORACLE_HOME.
|
bash-3.2# uname -n
idb1
bash-3.2# su - oracle
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ bash
bash-3.2$ pwd
/orainst
bash-3.2$ find . -name oraInst.loc
./d01/oracle/PROD/db/tech_st/11.1.0/oraInst.loc
./d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInst.loc
bash-3.2$
bash-3.2$ cat ./d01/oracle/PROD/db/tech_st/11.1.0/oraInst.loc
inventory_loc=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInventory
bash-3.2$ cat ./d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInst.loc
inventory_loc=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInventory
bash-3.2$
|
As several tasks require the DB Tier environment, create a symlink to the DB Tier environment variables for convenience. Alternatively, you could include this within the oracle user's profile.
|
bash-3.2$ cd /orainst/d01/oracle/PROD/db/tech_st/11.1.0
bash-3.2$ ls -l *env
-rw-r--r-- 1 oracle dba 4318 Feb 20 05:19 PROD_idb-lh.env
bash-3.2$ cd
bash-3.2$ ln -s /orainst/d01/oracle/PROD/db/tech_st/11.1.0/PROD_idb-lh.env env
bash-3.2$ ln -s /orainst/d01/oracle/PROD/db/tech_st/11*/appsutil/scripts/PROD_idb-lh scripts
bash-3.2$ pwd
/orainst
ash-3.2$ ls -l
total 8
drwxr-xr-x 3 oracle dba 3 Feb 20 02:35 d01
lrwxrwxrwx 1 oracle dba 58 Feb 20 05:27 env -> /orainst/d01/oracle/PROD/db/tech_st/11.1.0/PROD_idb-lh.env
drwxr-xr-x 3 oracle dba 3 Feb 20 05:34 oradiag_oracle
lrwxrwxrwx 1 oracle dba 71 Feb 20 05:47 scripts -> /orainst/d01/oracle/PROD/db/tech_st/11.1.0/appsutil/scripts/PROD_idb-lh
bash-3.2$
bash-3.2$ . ./env
bash-3.2$ env
NLS_SORT=binary
ADJREOPTS=-Xms128M -Xmx512M
HZ=100
TERM=vt100
SHELL=/bin/sh
NLS_LANG=American_America.US7ASCII
PERL5LIB=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/perl/lib/5.8.3:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/perl/lib/site_perl/5.8.3:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/appsutil/perl
OLDPWD=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/network/admin
LD_LIBRARY_PATH=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/lib:/usr/dt/lib:/usr/openwin/lib:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/ctx/lib:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/lib32
LINK_CNTRL=
ORA_NLS10=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/nls/data/9idata
ORACLE_SID=PROD
LIBPATH=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/lib:/usr/dt/lib:/usr/openwin/lib:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/ctx/lib:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/lib32
TNS_ADMIN=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/network/admin/PROD_idb-lh
PATH=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/perl/bin:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/bin:/usr/ccs/bin:/usr/sbin:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/appsutil/jre/bin:/usr/bin::/usr/cluster/bin:.
MAIL=/var/mail/oracle
ADJVAPRG=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/appsutil/jre/bin/java
PWD=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/network/admin/PROD_idb-lh
CONTEXT_NAME=PROD_idb-lh
CONTEXT_FILE=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/appsutil/PROD_idb-lh.xml
TZ=US/Pacific
ADPERLPRG=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/perl/bin/perl
ORA_TZFILE=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/oracore/zoneinfo/timezlrg.dat
SHLVL=1
HOME=/orainst
LOGNAME=oracle
NLS_NUMERIC_CHARACTERS=.,
SHLIB_PATH=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/lib:/usr/lib:/orainst/d01/oracle/PROD/db/tech_st/11.1.0/ctx/lib
ORACLE_HOME=/orainst/d01/oracle/PROD/db/tech_st/11.1.0
NLS_DATE_FORMAT=DD-MON-RR
_=/usr/bin/env
bash-3.2$
bash-3.2$ echo $ORACLE_HOME
/orainst/d01/oracle/PROD/db/tech_st/11.1.0
bash-3.2$
|
Using the ORACLE_HOME local inventory, list what is within the inventory.
|
bash-3.2$ $ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc /orainst/d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInst.loc
Invoking OPatch 11.1.0.6.2
Oracle Interim Patch Installer version 11.1.0.6.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : /orainst/d01/oracle/PROD/db/tech_st/11.1.0
Central Inventory : /orainst/d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInventory
from : /orainst/d01/oracle/PROD/db/tech_st/11.1.0/admin/oui/PROD_idb-lh/oraInst.loc
OPatch version : 11.1.0.6.2
OUI version : 11.1.0.7.0
OUI location : /orainst/d01/oracle/PROD/db/tech_st/11.1.0/oui
Log file location : /orainst/d01/oracle/PROD/db/tech_st/11.1.0/cfgtoollogs/opatch/opatch2012-02-20_05-32-40AM.log
Lsinventory Output file location : /orainst/d01/oracle/PROD/db/tech_st/11.1.0/cfgtoollogs/opatch/lsinv/lsinventory2012-02-20_05-32-40AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (3):
Oracle Database 11g 11.1.0.6.0
Oracle Database 11g Examples 11.1.0.6.0
Oracle Database 11g Patch Set 1 11.1.0.7.0
There are 3 products installed in this Oracle Home.
Interim patches (26) :
Patch 8199107 : Applied on Thu Mar 26 03:20:34 PDT 2009
Created on 27 Feb 2009, 14:59:51 hrs PST8PDT
Bugs fixed:
8199107
Patch 7627743 : Applied on Thu Mar 26 03:19:35 PDT 2009
Created on 24 Feb 2009, 18:27:13 hrs PST8PDT
Bugs fixed:
7627743
Patch 7639602 : Applied on Thu Mar 26 03:18:37 PDT 2009
Created on 18 Feb 2009, 09:20:31 hrs PST8PDT
Bugs fixed:
7639602
Patch 7211965 : Applied on Thu Mar 26 03:17:39 PDT 2009
Created on 13 Feb 2009, 04:02:27 hrs PST8PDT
Bugs fixed:
7211965
Patch 7708501 : Applied on Thu Mar 26 03:16:33 PDT 2009
Created on 19 Feb 2009, 00:39:35 hrs PST8PDT
Bugs fixed:
7708501
Patch 7330434 : Applied on Thu Mar 26 03:16:02 PDT 2009
Created on 11 Feb 2009, 13:44:29 hrs PST8PDT
Bugs fixed:
7330434
Patch 7580744 : Applied on Thu Mar 26 03:15:00 PDT 2009
Created on 12 Dec 2008, 02:07:43 hrs PST8PDT
Bugs fixed:
7580744
Patch 7708340 : Applied on Thu Mar 26 03:14:01 PDT 2009
Created on 28 Jan 2009, 06:54:50 hrs PST8PDT
Bugs fixed:
7708340
Patch 7327166 : Applied on Thu Mar 26 03:13:22 PDT 2009
Created on 6 Jan 2009, 23:13:50 hrs US/Pacific
Bugs fixed:
7327166
Patch 7486407 : Applied on Thu Mar 26 03:12:30 PDT 2009
Created on 11 Dec 2008, 20:51:14 hrs US/Eastern
Bugs fixed:
7486407
Patch 7243270 : Applied on Thu Mar 26 03:11:39 PDT 2009
Created on 13 Oct 2008, 02:27:02 hrs PST8PDT
Bugs fixed:
7243270
Patch 7277741 : Applied on Thu Mar 26 03:11:24 PDT 2009
Created on 13 Nov 2008, 23:00:39 hrs PST8PDT
Bugs fixed:
7277741
Patch 7319922 : Applied on Thu Mar 26 03:10:33 PDT 2009
Created on 16 Nov 2008, 18:30:39 hrs PST8PDT
Bugs fixed:
7319922
Patch 4247037 : Applied on Thu Mar 26 03:09:56 PDT 2009
Created on 15 OCT 2008, 01:58:00 hrs US/Pacific
Bugs fixed:
4247037
Patch 6530141 : Applied on Thu Mar 26 03:09:40 PDT 2009
Created on 14 Nov 2008, 04:24:50 hrs US/Pacific
Bugs fixed:
6530141
Patch 6598432 : Applied on Thu Mar 26 03:08:55 PDT 2009
Created on 14 Nov 2008, 05:38:17 hrs US/Pacific
Bugs fixed:
6598432
Patch 6815733 : Applied on Thu Mar 26 03:08:06 PDT 2009
Created on 14 Nov 2008, 05:35:51 hrs US/Pacific
Bugs fixed:
6815733
Patch 6991626 : Applied on Thu Mar 26 03:07:22 PDT 2009
Created on 13 Nov 2008, 08:23:52 hrs US/Pacific
Bugs fixed:
6991626
Patch 7111245 : Applied on Thu Mar 26 03:06:33 PDT 2009
Created on 25 Nov 2008, 21:22:15 hrs US/Pacific
Bugs fixed:
7111245
Patch 7253531 : Applied on Thu Mar 26 03:04:31 PDT 2009
Created on 25 Nov 2008, 07:50:26 hrs US/Pacific
Bugs fixed:
7253531
Patch 7556778 : Applied on Thu Mar 26 03:03:45 PDT 2009
Created on 2 Dec 2008, 06:57:38 hrs US/Pacific
Bugs fixed:
7556778
Patch 7000281 : Applied on Thu Mar 26 03:03:06 PDT 2009
Created on 3 Dec 2008, 20:53:58 hrs PST8PDT
Bugs fixed:
7000281
Patch 6263237 : Applied on Thu Mar 26 03:02:13 PDT 2009
Created on 14 Nov 2008, 05:27:42 hrs US/Pacific
Bugs fixed:
6263237
Patch 6870937 : Applied on Thu Mar 26 03:01:21 PDT 2009
Created on 14 Nov 2008, 05:10:20 hrs US/Pacific
Bugs fixed:
6870937
Patch 7295298 : Applied on Thu Mar 26 03:00:35 PDT 2009
Created on 18 Dec 2008, 23:41:05 hrs US/Pacific
Bugs fixed:
7295298
Patch 6972189 : Applied on Thu Mar 26 02:59:33 PDT 2009
Created on 13 Nov 2008, 21:02:58 hrs PST8PDT
Bugs fixed:
6972189
--------------------------------------------------------------------------------
OPatch succeeded.
bash-3.2$
|
Try to connect to the database.
|
bash-3.2$ echo $ORACLE_SID
PROD
bash-3.2$ echo $ORACLE_HOME
/orainst/d01/oracle/PROD/db/tech_st/11.1.0
bash-3.2$
bash-3.2$ $ORACLE_HOME/bin/sqlplus "/as sysdba"
SQL*Plus: Release 11.1.0.7.0 - Production on Mon Feb 20 05:34:45 2012
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> show sga
Total System Global Area 1069252608 bytes
Fixed Size 2115496 bytes
Variable Size 431265880 bytes
Database Buffers 624951296 bytes
Redo Buffers 10919936 bytes
SQL>
SQL> show parameters listener
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
local_listener string
remote_listener string
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
bash-3.2$
|
List the Oracle processes.
|
bash-3.2$ ptree oracle
1398 zsched
3389 /usr/lib/inet/inetd start
8118 /usr/sbin/in.rlogind
8120 -sh
8149 bash
13654 -sh
13658 bash
13710 ptree oracle
12220 /orainst/d01/oracle/PROD/db/tech_st/11.1.0/bin/tnslsnr PROD -inherit
13152 ora_pmon_PROD
13154 ora_vktm_PROD
13158 ora_diag_PROD
13160 ora_dbrm_PROD
13162 ora_psp0_PROD
13164 ora_dia0_PROD
13166 ora_mman_PROD
13168 ora_dbw0_PROD
13170 ora_dbw1_PROD
13172 ora_dbw2_PROD
13174 ora_dbw3_PROD
13176 ora_dbw4_PROD
13178 ora_dbw5_PROD
13180 ora_dbw6_PROD
13182 ora_dbw7_PROD
13184 ora_dbw8_PROD
13186 ora_dbw9_PROD
13188 ora_dbwa_PROD
13190 ora_dbwb_PROD
13192 ora_dbwc_PROD
13194 ora_dbwd_PROD
13196 ora_dbwe_PROD
13198 ora_dbwf_PROD
13200 ora_lgwr_PROD
13202 ora_ckpt_PROD
13204 ora_smon_PROD
13206 ora_reco_PROD
13208 ora_mmon_PROD
13210 ora_mmnl_PROD
13218 ora_fbda_PROD
13222 ora_qmnc_PROD
13281 ora_q000_PROD
13478 ora_q001_PROD
13639 ora_smco_PROD
13709 ora_w000_PROD
bash-3.2$
|
List the Oracle TNS entries.
|
bash-3.2$ echo $TNS_ADMIN
/orainst/d01/oracle/PROD/db/tech_st/11.1.0/network/admin/PROD_idb-lh
bash-3.2$
bash-3.2$ cat $TNS_ADMIN/tnsnames.ora
###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
#$Header: NetServiceHandler.java 120.19 2008/04/28 06:40:02 schagant ship $
#
###############################################################
PROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb-lh.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=PROD)
(INSTANCE_NAME=PROD)
)
)
PROD_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb-lh.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=PROD)
(INSTANCE_NAME=PROD)
)
)
PROD_LOCAL=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb-lh.us.oracle.com)(PORT=1521))
)
PROD_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=idb-lh.us.oracle.com)(PORT=1521))
)
(CONNECT_DATA=
(SERVICE_NAME=PROD)
)
)
PROD_REMOTE=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb-lh.us.oracle.com)(PORT=1521))
)
)
extproc_connection_data =
(DESCRIPTION=
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROCPROD))
)
(CONNECT_DATA=
(SID=PLSExtProc)
(PRESENTATION = RO)
) )
IFILE=/orainst/d01/oracle/PROD/db/tech_st/11.1.0/network/admin/PROD_idb-lh/PROD_idb-lh_ifile.ora
bash-3.2$
bash-3.2$ $ORACLE_HOME/bin/lsnrctl status PROD
LSNRCTL for Solaris: Version 11.1.0.7.0 - Production on 20-FEB-2012 05:44:08
Copyright (c) 1991, 2008, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=idb-lh.us.oracle.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias PROD
Version TNSLSNR for Solaris: Version 11.1.0.7.0 - Production
Start Date 20-FEB-2012 05:12:11
Uptime 0 days 0 hr. 31 min. 57 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /orainst/d01/oracle/PROD/db/tech_st/11.1.0/network/admin/PROD_idb-lh/listener.ora
Listener Log File /orainst/d01/oracle/PROD/db/tech_st/11.1.0/log/diag/tnslsnr/idb1/prod/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.30.53)(PORT=1521)))
Services Summary...
Service "PROD" has 1 instance(s).
Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
bash-3.2$
|
4.3.6 Create App Tier user and directories (iopmn1 and iopmn2)
Create the App Tier user and directories for zone cluster nodes iopmn1 and iopmn2. Start by performing the following commands on zone cluster node iopmn1.
|
bash-3.2# uname -n
iopmn1
bash-3.2# groupadd -g 1000 dba
bash-3.2# useradd -u 1001 -g 1000 -d /ebsfs applmgr
UX: useradd: ERROR: applmgr is already in use. Choose another.
Bash-3.2#
bash-3.2# getent passwd applmgr
applmgr:*NP*:64100:70100:Oracle Application Manager:/apps/applmgr:/bin/true
bash-3.2#
|
Temporarily remove nis from passwd within /etc/nsswitch as user applmgr already exists within nis.
|
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
#passwd: files nis
passwd: files
bash-3.2#
bash-3.2# useradd -u 1001 -g 1000 -d /ebsfs applmgr
bash-3.2#
|
Reinstate nis from passwd within /etc/nsswitch.
|
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
passwd: files nis
#passwd: files
bash-3.2#
|
Check that applmgr can be resolved.
|
bash-3.2# getent passwd applmgr
applmgr:x:1001:1000::/ebsfs:/bin/sh
bash-3.2#
|
Repeat on zone cluster node iopmn2.
|
bash-3.2# uname -n
iopmn2
bash-3.2# groupadd -g 1000 dba
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
#passwd: files nis
passwd: files
bash-3.2#
bash-3.2# useradd -u 1001 -g 1000 -d /ebsfs applmgr
bash-3.2#
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
passwd: files nis
#passwd: files
bash-3.2#
bash-3.2# getent passwd applmgr
applmgr:x:1001:1000::/ebsfs:/bin/sh
bash-3.2#
|
Create the directory and set the ownership and permission:
|
bash-3.2# mkdir -p /ebsfs/d01/oracle
bash-3.2# chmod -R 755 /ebsfs
bash-3.2# chown -R applmgr:dba /ebsfs
|
4.3.7 Start VNC Server
|
bash-3.2# uname -n
iopmn1
bash-3.2# cat .profile
PATH=${PATH}:/usr/cluster/bin:/usr/X11/bin:/usr/openwin/bin:/usr/ccs/bin; export PATH
MANPATH=/usr/man:/usr/cluster/man; export MANPATH
bash-3.2#
bash-3.2# . .profile
bash-3.2# vncserver :10
You will require a password to access your desktops.
Password:
Verify:
xauth: creating new authority file //.Xauthority
New 'iopmn1:10 ()' desktop is iopmn1:10
Creating default startup script //.vnc/xstartup
Starting Applications specified in //.vnc/xstartup
Log file is //.vnc/iopmn1:10.log
bash-3.2#
|
4.3.8 Ensure the Oracle E-Business Suite 12.1.1 software is available
|
bash-3.2# uname -n
iopmn1
bash-3.2# dfshares ptata1
RESOURCE SERVER ACCESS TRANSPORT
ptata1:/swlib/ebs12.1.1 ptata1 - -
ptata1:/swlib/OC ptata1 - -
ptata1:/swlib/ora11.2.0.3 ptata1 - -
bash-3.2#
bash-3.2# mount ptata1:/swlib/ebs12.1.1 /mnt
bash-3.2#
|
4.3.9 Install the App Tier (eweb-lh)
The installation uses the configuration file conf_PROD.txt that was built earlier51. However, that file first needs to be copied to this zone cluster node.
Copy /var/tmp/EBS/* from zone cluster idb1 to iopmn1. Temporarily allow root as an ftpuser on idb1.
|
bash-3.2# uname -n
idb1
bash-3.2# grep root /etc/ftpd/ftpusers
#root
bash-3.2#
|
FTP the configuration file.
|
bash-3.2# uname -n
iopmn1
bash-3.2# mkdir /var/tmp/EBS
bash-3.2# chmod 777 /var/tmp/EBS
bash-3.2# cd /var/tmp/EBS
bash-3.2# ftp idb1
Connected to idb1.
220 idb1 FTP server ready.
Name (idb1:root):
331 Password required for root.
Password:
230 User root logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /var/tmp/EBS
250 CWD command successful.
ftp> prompt
Interactive mode off.
ftp> mget *
200 PORT command successful.
150 Opening BINARY mode data connection for 02200413.log (16071 bytes).
226 Transfer complete.
local: 02200413.log remote: 02200413.log
16071 bytes received in 0.011 seconds (1482.71 Kbytes/s)
200 PORT command successful.
150 Opening BINARY mode data connection for conf_PROD.txt (41469 bytes).
226 Transfer complete.
local: conf_PROD.txt remote: conf_PROD.txt
41469 bytes received in 0.012 seconds (3445.97 Kbytes/s)
ftp> bye
221-You have transferred 57540 bytes in 2 files.
221-Total traffic for this session was 58364 bytes in 3 transfers.
221-Thank you for using the FTP service on idb1.
221 Goodbye.
bash-3.2#
|
Using the configuration file conf_PROD.txt, install the App Tier on the node where eweb-lh is currently online. Note, here the command rapidwiz -servername eweb-lh is used.
From a VNC client, step through the following screens:
As before, you can ignore the message that a central inventory does not exist (see screen shot above).
When the above has passed verification, stop everything on this node (iopmn1) so that installation can proceed on the other App Tiers (iopmn1-lh and iopmn2-lh).
As several tasks require the App Tier environment, create a symlink to the App Tier environment variables for convenience.
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$
$ bash
bash-3.2$ cd /ebsfs/d01/oracle/PROD/apps/apps_st/appl
bash-3.2$ ls -l *env
-rw-r--r-- 1 applmgr dba 1086 Feb 20 07:38 AppSPROD_eweb-lh.env
-rw-r--r-- 1 applmgr dba 17295 Feb 20 07:38 PROD_eweb-lh.env
bash-3.2$
bash-3.2$ cd
bash-3.2$ ln -s /ebsfs/d01/oracle/PROD/apps/apps_st/appl/AppSPROD_eweb-lh.env eweb-lh
bash-3.2$ ls -l
total 4
drwxr-xr-x 3 applmgr dba 3 Feb 20 06:00 d01
lrwxrwxrwx 1 applmgr dba 61 Feb 20 08:00 env_eweb-lh -> /ebsfs/d01/oracle/PROD/apps/apps_st/appl/AppSPROD_eweb-lh.env
bash-3.2$
|
Stop the application tier services on this node (iopmn1):
|
bash-3.2$ . env_eweb-lh
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ ./adstpall.sh
You are running adstpall.sh version 120.10
Enter the APPS username: apps
Enter the APPS password:
The logfile for this session is located at /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adstpall.log
Executing service control script:
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts/adoafmctl.sh stop
script returned:
****************************************************
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
adoafmctl.sh: exiting with status 0
adoafmctl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adoafmctl.txt for more information ...
.end std out.
.end err out.
****************************************************
Executing service control script:
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts/adformsctl.sh stop
script returned:
****************************************************
You are running adformsctl.sh version 120.16
Stopping OPMN managed FORMS OC4J instance ...
adformsctl.sh: exiting with status 0
adformsctl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adformsctl.txt for more information ...
.end std out.
.end err out.
****************************************************
Executing service control script:
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts/adoacorectl.sh stop
script returned:
****************************************************
You are running adoacorectl.sh version 120.13
Stopping OPMN managed OACORE OC4J instance ...
adoacorectl.sh: exiting with status 0
adoacorectl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adoacorectl.txt for more information ...
.end std out.
.end err out.
****************************************************
Executing service control script:
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts/adapcctl.sh stop
script returned:
****************************************************
You are running adapcctl.sh version 120.7.12010000.2
Stopping OPMN managed Oracle HTTP Server (OHS) instance ...
adapcctl.sh: exiting with status 0
adapcctl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adapcctl.txt for more information ...
.end std out.
.end err out.
****************************************************
Executing service control script:
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts/adalnctl.sh stop
script returned:
****************************************************
adalnctl.sh version 120.3
Shutting down listener process APPS_PROD.
adalnctl.sh: exiting with status 0
adalnctl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adalnctl.txt for more information ...
.end std out.
.end err out.
****************************************************
Executing service control script:
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts/adopmnctl.sh stop
script returned:
****************************************************
You are running adopmnctl.sh version 120.6
Stopping Oracle Process Manager (OPMN) and the managed processes ...
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adopmnctl.txt for more information ...
.end std out.
.end err out.
****************************************************
All enabled services on this node are stopped.
adstpall.sh:Exiting with status 0
adstpall.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/logs/appl/admin/log/adstpall.log for more information ...
bash-3.2$
|
4.3.10 Install the App Tier (iopmn1-lh)
Before installing the iopmn1-lh Application Tier, ensure that the eweb-lh App Tier has been stopped, otherwise there will be a port conflict.
|
bash-3.2# clrs disable eweb-lh
bash-3.2# clrs status eweb-lh
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
eweb-lh iopmn1 Offline Offline - LogicalHostname offline.
iopmn2 Offline Offline
bash-3.2#
|
Using the configuration file conf_PROD.txt, install the App Tier on the node where iopmn1-lh is currently online. Note, here the command rapidwiz -servername iopmn1-lh is used.
From a VNC client, start the installation:
The screens to install the App Tier on iopmn1-lh are the same as the screens that were used to install the eweb-lh App Tier. If needed, follow the screens shown previously in the section Install the App Tier (eweb-lh) in Section 4.3.9.
As before, after the installation step everything on this node (iopmn1) so that installation can proceed on the other App Tier (iopmn2-lh).
As several tasks require the App Tier environment, create a symlink to the App Tier environment variables for convenience.
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$
$ bash
bash-3.2$
bash-3.2$ ln -s /ebsfs/d01/oracle/PROD/apps/apps_st/appl/AppSPROD_iopmn1-lh.env env_iopmn1-lh
bash-3.2$
|
Stop the application tier services on this node:
|
bash-3.2$ . env_iopmn1-lh
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ ./adstpall.sh
<output truncated for brevity>
|
4.3.11 Install the App Tier (iopmn2-lh)
Before installing the iopmn2-lh Application Tier, ensure that the iopmn1-lh AppTier has been stopped, otherwise there will be a port conflict:
|
bash-3.2# clrs disable iopmn1-lh
bash-3.2# clrs status iopmn1-lh
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
iopmn1-lh iopmn1 Offline Offline - LogicalHostname offline.
iopmn2 Offline Offline
bash-3.2#
|
Using the configuration file conf_PROD.txt, install the App Tier on the node where iopmn2-lh is online. If required, switch the Solaris Cluster opmnlh2-rg resource group to the zone cluster node iopmn1.
|
bash-3.2# clrs switch -n iopmn1 opmn2lh-rg
|
Note, the following steps use the command: rapidwiz -servername iopmn2-lh.
From a VNC client, start the installation:
The screens to install the App Tier on iopmn2-lh are the same as the screens that were used for the Primary App Tier installation. If needed, follow the screens shown previously in the section Install the App Tier (eweb-lh) in Section 4.3.9.
As before, after the installation stop everything on this node (iopmn1).
As several tasks require the App Tier environment, create a symlink to the App Tier environment variables for convenience.
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$
$ bash
bash-3.2$
bash-3.2$ ln -s /ebsfs/d01/oracle/PROD/apps/apps_st/appl/AppSPROD_iopmn2-lh.env env_iopmn2-lh
bash-3.2$
|
Stop the application tier services on this node:
|
bash-3.2$ . env_iopmn2-lh
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ ./adstpall.sh
<output truncated for brevity>
|
4.3.12 Create App Tier user and directories (icm1 & icm2)
Create the App Tier user and directories for zone cluster nodes icm1 and icm2. Start by performing the following commands on zone cluster node icm1.
|
bash-3.2# uname -n
icm1
bash-3.2# groupadd -g 1000 dba
bash-3.2# useradd -u 1001 -g 1000 -d /ebsfs applmgr
UX: useradd: ERROR: applmgr is already in use. Choose another.
bash-3.2#
bash-3.2# getent passwd applmgr
applmgr:*NP*:64100:70100:Oracle Application Manager:/apps/applmgr:/bin/true
bash-3.2#
|
Temporarily remove nis from passwd within /etc/nsswitch:
|
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
#passwd: files nis
passwd: files
bash-3.2#
bash-3.2# useradd -u 1001 -g 1000 -d /ebsfs applmgr
bash-3.2#
|
Reinstate nis from passwd within /etc/nsswitch:
|
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
passwd: files nis
#passwd: files
bash-3.2#
|
Check that applmgr can be resolved.
|
bash-3.2# getent passwd applmgr
applmgr:x:1001:1000::/ebsfs:/bin/sh
bash-3.2#
|
Repeat on zone cluster node icm2.
|
bash-3.2# uname -n
icm2
bash-3.2# groupadd -g 1000 dba
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
#passwd: files nis
passwd: files
bash-3.2#
bash-3.2# useradd -u 1001 -g 1000 -d /ebsfs applmgr
bash-3.2#
bash-3.2# grep passwd /etc/nsswitch.conf
# the following two lines obviate the "+" entry in /etc/passwd and /etc/group.
passwd: files nis
#passwd: files
bash-3.2#
bash-3.2# getent passwd applmgr
applmgr:x:1001:1000::/ebsfs:/bin/sh
bash-3.2#
|
Create the directory and set permissions and ownership:
|
bash-3.2# mkdir -p /ebsfs/d01/oracle
bash-3.2# chmod -R 755 /ebsfs
bash-3.2# chown -R applmgr:dba /ebsfs
|
4.3.13 Start VNC Server
Start the VNC server:
|
bash-3.2# uname -n
icm1
bash-3.2# cat .profile
PATH=${PATH}:/usr/cluster/bin:/usr/X11/bin:/usr/openwin/bin:/usr/ccs/bin; export PATH
MANPATH=/usr/man:/usr/cluster/man; export MANPATH
bash-3.2#
bash-3.2# . .profile
bash-3.2# vncserver :10
You will require a password to access your desktops.
Password:
Verify:
xauth: creating new authority file //.Xauthority
New 'icm1:10 ()' desktop is icm1:10
Creating default startup script //.vnc/xstartup
Starting Applications specified in //.vnc/xstartup
Log file is //.vnc/icm1:10.log
bash-3.2#
|
4.3.14 Ensure the Oracle E-Business Suite 12.1.1 software is available
Ensure the Oracle E-Business Suite 12.1.1 software is available:
|
bash-3.2# uname -n
icm1
bash-3.2# dfshares ptata1
RESOURCE SERVER ACCESS TRANSPORT
ptata1:/swlib/ebs12.1.1 ptata1 - -
ptata1:/swlib/OC ptata1 - -
ptata1:/swlib/ora11.2.0.3 ptata1 - -
bash-3.2#
bash-3.2# mount ptata1:/swlib/ebs12.1.1 /mnt
bash-3.2#
|
4.3.15 Install the App Tier (icm1)
The installation will use the configuration file conf_PROD.txt that was built earlier51. However, that file first needs to be copied to this zone cluster node.
Copy /var/tmp/EBS/* from idb1 to icm1.
Using the configuration file conf_PROD.txt, install the App Tier on the icm1 node. Note, here the command rapidwiz -servername icm1 is used, even though this App Tier is being installed on the physical zone cluster node. Failure to specify the argument -servername icm1 will cause problems later on that will require the re-installation of all App Tiers.
From a VNC client, step through the rapidwiz installation screens:
The screens to install the App Tier on icm1 are the same as the screens that were used for the Primary App Tier installation.
As before, after the installation stop everything on this node (icm1).
As several tasks require the App Tier environment, create a symlink to the App Tier environment variables for convenience:
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$
$ bash
bash-3.2$
bash-3.2$ ln -s /ebsfs/d01/oracle/PROD/apps/apps_st/appl/AppSPROD_icm1.env env_icm1
bash-3.2$
|
Stop the application tier services on this node:
|
bash-3.2$ . env_icm1
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ ./adstpall.sh
<output truncated for brevity>
|
4.3.16 Start VNC Server
Start the VNC server:
|
bash-3.2# uname -n
icm2
bash-3.2# cat .profile
PATH=${PATH}:/usr/cluster/bin:/usr/X11/bin:/usr/openwin/bin:/usr/ccs/bin; export PATH
MANPATH=/usr/man:/usr/cluster/man; export MANPATH
bash-3.2#
bash-3.2# . .profile
bash-3.2# vncserver :10
You will require a password to access your desktops.
Password:
Verify:
xauth: creating new authority file //.Xauthority
New 'icm2:10 ()' desktop is icm2:10
Creating default startup script //.vnc/xstartup
Starting Applications specified in //.vnc/xstartup
Log file is //.vnc/icm2:10.log
bash-3.2#
|
4.3.17 Ensure the Oracle E-Business Suite 12.1.1 software is available
Ensure the Oracle E-Business Suite 12.1.1 software is available on icm2:
|
bash-3.2# uname -n
icm2
bash-3.2# dfshares ptata1
RESOURCE SERVER ACCESS TRANSPORT
ptata1:/swlib/ebs12.1.1 ptata1 - -
ptata1:/swlib/OC ptata1 - -
ptata1:/swlib/ora11.2.0.3 ptata1 - -
bash-3.2#
bash-3.2# mount ptata1:/swlib/ebs12.1.1 /mnt
bash-3.2#
|
Here, the installation uses the configuration file conf_PROD.txt that was built earlier. However, that file first needs to be copied to this zone cluster node.
Copy /var/tmp/EBS/* from idb1 to icm2.
Using the configuration file conf_PROD.txt, install the App Tier on the icm2 node. Note, here the command rapidwiz -servername icm2 is used, even though this App Tier is installed on the physical zone cluster node. Failure, to specify the -servername icm2 option will cause problems later on that will require the re-installation of all App Tiers.
From a VNC client, issue the following command and then step through the wizard screens (make sure that ‑ servername icm2 is used with this command):
|
# rapidwiz –servername –icm2
|
The screens to install the App Tier on icm2 are the same as the screens that were used for the eweb-lh App Tier installation. If needed, follow the screens shown previously in the section Install the App Tier (eweb-lh) in Section 4.3.9.
As before, after the installation stop everything on this node (icm2).
As several tasks require the App Tier environment, create a symlink to the App Tier environment variables for convenience.
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$
$ bash
bash-3.2$
bash-3.2$ ln -s /ebsfs/d01/oracle/PROD/apps/apps_st/appl/AppSPROD_icm2.env env_icm2
bash-3.2$
|
Stop the application tier services on this node:
|
bash-3.2$ . env_icm2
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ ./adstpall.sh
<output truncated for brevity>
|
4.4 Oracle E-Business Suite Post Installation
Now that Oracle E-Business Suite 12.1.1 is complete, some specific options must be configured so that the multi-node configuration meets the design goals.
Figure 2: E-Business Suite on SPARC SuperCluster.
The high level steps are as follows:
· Amend each App Tier Context File to configure clustered OPMN OC4J instances.
· Setup the Solaris Cluster shared object library, to interpose requests to determine the physical host name and instead return the logical host name.
· Create Custom<Context_Name>.env files to facilitate logical host interposing via LD_PRELOAD.
· Create and modify custom template files to facilitate further logical host interposing.
· Run AutoConfig for the App Tiers, eweb-lh, iopmn1-lh and iopmn2-lh.
· Configure Parallel Concurrent Processing and run AutoConfig for the App Tiers icm1 and icm2.
4.4.1 Configuration for Clustered OPMN OC4J instances
Amend each App Tier's context file as required. Note, that once each context file has been updated, AutoConfig is not run until some additional steps have been completed.
4.4.1.1 Context File changes for eweb-lh
Make the context file changes for eweb-lh:
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ bash
bash-3.2$
bash-3.2$ . env_eweb-lh
bash-3.2$ vi $CONTEXT_FILE
|
Edit the Context File and change the following:
|
s_oacorestatus disabled
s_formsstatus disabled
s_oafmstatus disabled
s_oc4j_cluster_nodes
eweb-lh.us.oracle.com:6201,iopmn1-lh.us.oracle.com:6200,iopmn2-lh.us.oracle.com:6200
|
Bump the following ports by 1:
|
ons_localport 6101
ons_remoteport 6201
ons_requestport 6501
|
Note, there are two entries on each line that need to be bumped. For example:
|
bash-3.2$ grep ons_ $CONTEXT_FILE | grep PORT
<ons_localport oa_var="s_ons_localport" oa_type="PORT" base="6101" step="1" range="-1" label="ONS Local Port">6101</ons_localport>
<ons_remoteport oa_var="s_ons_remoteport" oa_type="PORT" base="6201" step="1" range="-1" label="ONS Remote Port">6201</ons_remoteport>
<ons_requestport oa_var="s_ons_requestport" oa_type="PORT" base="6501" step="1" range="-1" label="ONS Request Port">6501</ons_requestport>
bash-3.2$
|
4.4.1.2 Context File changes for iopmn1-lh
Make the context file changes for iopmn1-lh:
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ bash
bash-3.2$
bash-3.2$ . env_iopmn1-lh
bash-3.2$ vi $CONTEXT_FILE
|
Edit the Context File and change the following:
|
s_webentryhost eweb-lh.us.oracle.com
s_login_page eweb-lh.us.oracle.com
s_external_url eweb-lh.us.oracle.com
s_oacorestatus enabled (should already be enabled)
s_formsstatus enabled (should already be enabled)
s_oafmstatus enabled (should already be enabled)
s_oc4j_cluster_nodes
eweb-lh.us.oracle.com:6201,iopmn1-lh.us.oracle.com:6200,iopmn2-
lh.us.oracle.com:6200
|
4.4.1.3 Context File changes for iopmn2-lh
Make the context file changes for iopmn2-lh:
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ bash
bash-3.2$
bash-3.2$ . env_iopmn2-lh
bash-3.2$ vi $CONTEXT_FILE
|
Edit the Context File and change the following:
|
s_webentryhost eweb-lh.us.oracle.com
s_login_page eweb-lh.us.oracle.com
s_external_url eweb-lh.us.oracle.com
s_oacorestatus enabled (should already be enabled)
s_formsstatus enabled (should already be enabled)
s_oafmstatus enabled (should already be enabled)
s_oc4j_cluster_nodes
eweb-lh.us.oracle.com:6201,iopmn1-lh.us.oracle.com:6200,iopmn2-
lh.us.oracle.com:6200
|
4.4.2 Create /usr/lib/secure/libschost.so.1
Within each Solaris 10 global zone, create the following symlinks on each node.
First, on node etc11-01-210:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2#
bash-3.2# cd /usr/lib/secure
bash-3.2# ln -s /usr/cluster/lib/libschost.so.1 libschost.so.1
bash-3.2# cd sparcv9
bash-3.2# ln -s /usr/cluster/lib/sparcv9/libschost.so.1 libschost.so.1
|
Repeat the above commands for node etc11-02-s10.
4.4.3 Create Custom<Context_Name>.env files
Create a Custom<Context_Name>.env file for the following App Tiers: eweb-lh, iopmn1-lh, and iopmn2-lh.
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ bash
bash-3.2$ cd /ebsfs/d01/oracle/PROD/apps/apps_st/appl
bash-3.2$
bash-3.2$ vi customPROD_eweb-lh.env
|
Add entries, as follows:
|
bash-3.2$ cat customPROD_eweb-lh.env
SC_LHOSTNAME=eweb-lh.us.oracle.com
LD_PRELOAD_32=/usr/lib/secure/libschost.so.1
LD_PRELOAD_64=/usr/lib/secure/sparcv9/libschost.so.1
export SC_LHOSTNAME LD_PRELOAD_32 LD_PRELOAD_64
bash-3.2$
|
Repeat for iopmn1-lh and iopmn2-lh:
|
bash-3.2$ pwd
/ebsfs/d01/oracle/PROD/apps/apps_st/appl
bash-3.2$ ls -l *env
-rw-r--r-- 1 applmgr dba 1086 Mar 2 03:19 AppSPROD_eweb-lh.env
-rw-r--r-- 1 applmgr dba 1071 Mar 2 03:03 AppSPROD_icm1.env
-rw-r--r-- 1 applmgr dba 1071 Mar 2 03:04 AppSPROD_icm2.env
-rw-r--r-- 1 applmgr dba 1096 Mar 2 03:14 AppSPROD_iopmn1-lh.env
-rw-r--r-- 1 applmgr dba 1096 Mar 2 03:14 AppSPROD_iopmn2-lh.env
-rw-r--r-- 1 applmgr dba 16734 Mar 2 03:19 PROD_eweb-lh.env
-rw-r--r-- 1 applmgr dba 16662 Mar 2 03:03 PROD_icm1.env
-rw-r--r-- 1 applmgr dba 16662 Mar 2 03:04 PROD_icm2.env
-rw-r--r-- 1 applmgr dba 16782 Mar 2 03:14 PROD_iopmn1-lh.env
-rw-r--r-- 1 applmgr dba 16782 Mar 2 03:14 PROD_iopmn2-lh.env
-rw-r--r-- 1 applmgr dba 216 Feb 26 05:11 customPROD_eweb-lh.env
-rw-r--r-- 1 applmgr dba 184 Feb 22 09:34 customPROD_iopmn1-lh.env
-rw-r--r-- 1 applmgr dba 184 Feb 22 09:34 customPROD_iopmn2-lh.env
bash-3.2$
bash-3.2$ cat customPROD_iopmn1-lh.env
SC_LHOSTNAME=iopmn1-lh.us.oracle.com
LD_PRELOAD_32=/usr/lib/secure/libschost.so.1
LD_PRELOAD_64=/usr/lib/secure/sparcv9/libschost.so.1
export SC_LHOSTNAME LD_PRELOAD_32 LD_PRELOAD_64
bash-3.2$
bash-3.2$ cat customPROD_iopmn2-lh.env
SC_LHOSTNAME=iopmn2-lh.us.oracle.com
LD_PRELOAD_32=/usr/lib/secure/libschost.so.1
LD_PRELOAD_64=/usr/lib/secure/sparcv9/libschost.so.1
export SC_LHOSTNAME LD_PRELOAD_32 LD_PRELOAD_64
bash-3.2$
|
Note, each Custom<Context_Name>.env file specifies its appropriate SC_LHOSTNAME entry.
4.4.4 Create Custom Templates
There are two Oracle E-Business Suite files that need to be modified to allow us to interpose logical hosts.
Note, you cannot just amend generated files as those files would simply be overwritten whenever AutoConfig was performed. As such, you need to create custom template files for the files you want to modify.
If required, refer to the following My Oracle Support note:
· Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12 [ID 387859.1]
4.4.4.1 Create Custom Template afjava_sh_1013.tmp
Creating a custom template file for afjava_sh_1013.tmp allows us to enable logical host interposing for OPMN/OC4J instances. Subsequently, after this change, Distributed Java Object Cache (DJOC) can function as designed:
|
bash-3.2$ grep s_long_running_jvm $CONTEXT_FILE
<jms_cache oa_var="s_long_running_jvm">true</jms_cache>
bash-3.2$
|
This enables DJOC by default but only works if you modify the afjava_sh_1013.tmp so that the logical host can be interposed.
If required, refer to the following OSC CR and Oracle BUGs:
· *Change Request ID*: 7038092
*Synopsis*: HA-Oracle E-Business Suite R12.1.x distributed java cache object (long_running_jvm=true) issues
· Bug 12320700 - SOLARIS CLUSTER AND DISTRIBUTED JAVA OBJECT CACHE (LONG_RUNNING_JVM=TRUE) ISSUES
Create the custome template file:
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ . env_eweb-lh
$ bash
bash-3.2$ pwd
/ebsfs
bash-3.2$ find . -name afjava_sh_1013.tmp
./d01/oracle/PROD/apps/apps_st/appl/fnd/12.0.0/admin/template/afjava_sh_1013.tmp
bash-3.2$
bash-3.2$ mkdir -p /ebsfs/d01/oracle/PROD/apps/apps_st/appl/fnd/12.0.0/admin/template/custom
bash-3.2$ cd /ebsfs/d01/oracle/PROD/apps/apps_st/appl/fnd/12.0.0/admin/template/custom
bash-3.2$ cp ../afjava_sh_1013.tmp .
bash-3.2$ vi afjava_sh_1013.tmp
|
Edit ~/templat/custom/afjava_sh_1013.tmp and add the highlighted line.
|
bash-3.2$ cat afjava_sh_1013.tmp
#!/bin/sh
# $Header: afjava_sh_1013.tmp 120.3.12010000.2 2009/04/17 08:41:22 sbandla ship $
#/*===========================================================================+
# | Copyright (c) 2003 Oracle Corporation, Redwood Shores, California, USA |
# | All rights reserved |
# | Applications Division |
# +===========================================================================+
# |
# | FILENAME
# | java.sh
# |
# | DESCRIPTION
# | This script is used by opmn and java concurrent programs to call java
# | executable with additional arguments.
# |
# | PLATFORM
# | Unix Generic
# |
# | NOTES
# |
# | HISTORY
# |
# +===========================================================================+
#
# $AutoConfig$
#
# dbdrv: none
JSERVJAVA=%s_jdktop%/bin/java
ARGV="$*"
PATH="%s_jdktop%/bin:$PATH:/bin:/usr/bin"
export PATH
LD_LIBRARY_PATH="%s_oacore_prepend_ld_lib_path%%s_oacore_ld_lib_path%"
export LD_LIBRARY_PATH
LIBPATH="%s_oacore_prepend_ld_lib_path%%s_oacore_ld_lib_path%"
export LIBPATH
SHLIB_PATH="%s_oacore_prepend_ld_lib_path%%s_oacore_ld_lib_path%"
export SHLIB_PATH
OPTIT_HOME="%s_optit_home%"
export OPTIT_HOME
. /ebsfs/d01/oracle/PROD/apps/apps_st/appl/custom%s_contextname%.env
JAVA_ADDITIONAL_ARGS=" -DCLIENT_PROCESSID=$$ "
exec $JSERVJAVA $JAVA_ADDITIONAL_ARGS $ARGV
bash-3.2$
|
Note, amending a custom afjava_sh_1013.tmp will ensure that the appropriate Custom<Context_Name>.env file is sourced for each App Tier. Consequently, the generated file will be within $ADMIN_SCRIPTS/java.sh, which is used later on by OPMN.
The following example shows the generated $ADMIN_SCRIPTS/java.sh file for the eweb-lh App Tier.
Note that %s_contextname% has been replaced with PROD_eweb-lh. However this will only happen after AutoConfig has run, which is performed at the end of this section.
|
bash-3.2$ cat $ADMIN_SCRIPTS_HOME/java.sh
#!/bin/sh
# $Header: afjava_sh_1013.tmp 120.3.12010000.2 2009/04/17 08:41:22 sbandla ship $
#/*===========================================================================+
# | Copyright (c) 2003 Oracle Corporation, Redwood Shores, California, USA |
# | All rights reserved |
# | Applications Division |
# +===========================================================================+
# |
# | FILENAME
# | java.sh
# |
# | DESCRIPTION
# | This script is used by opmn and java concurrent programs to call java
# | executable with additional arguments.
# |
# | PLATFORM
# | Unix Generic
# |
# | NOTES
# |
# | HISTORY
# |
# +===========================================================================+
#
# ###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
# ###############################################################
#
# dbdrv: none
JSERVJAVA=/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/appsutil/jdk/bin/java
ARGV="$*"
PATH="/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/appsutil/jdk/bin:$PATH:/bin:/usr/bin"
export PATH
LD_LIBRARY_PATH="/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/lib32:/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/lib:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/cz/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/iby/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/pon/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/sht/12.0.0/lib"
export LD_LIBRARY_PATH
LIBPATH="/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/lib32:/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/lib:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/cz/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/iby/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/pon/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/sht/12.0.0/lib"
export LIBPATH
SHLIB_PATH="/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/lib32:/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.3/lib:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/cz/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/iby/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/pon/12.0.0/bin:/ebsfs/d01/oracle/PROD/apps/apps_st/appl/sht/12.0.0/lib"
export SHLIB_PATH
OPTIT_HOME=""
export OPTIT_HOME
. /ebsfs/d01/oracle/PROD/apps/apps_st/appl/customPROD_eweb-lh.env
JAVA_ADDITIONAL_ARGS=" -DCLIENT_PROCESSID=$$ "
exec $JSERVJAVA $JAVA_ADDITIONAL_ARGS $ARGV
bash-3.2$
|
4.4.4.2 Create Custom Template txkGenDefaultEnv.tmp
Creating a custom template file for txkGenDefaultEnv.tmp allows us to enable logical host interposing for Forms. Currently, the Forms deployment hard codes a setting for LD_PRELOAD_32, i.e. it does not consider that a previously setting of LD_PRELOAD_32 exists.
Note, earlier you set LD_PRELOAD_32 for logical host interposing.
The following generated files hard code LD_PRELOAD_32:
|
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ pwd
/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/admin/scripts
bash-3.2$ grep /default.env *
adformsctl.sh:DEFAULTENV_FILE="/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/ora/10.1.2/forms/server/default.env"
bash-3.2$
bash-3.2$ grep /socket.env *
adformsrvctl.sh:SOCKETENV_FILE="/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/ora/10.1.2/forms/server/socket.env"
bash-3.2$
bash-3.2$ pwd
bash-3.2$ /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/ora/10.1.2/forms/server
bash-3.2$ ls -l
total 139
-rw-r--r-- 1 applmgr dba 31374 Mar 2 03:19 Appsweb.cfg
-rw-r--r-- 1 applmgr dba 18698 Mar 2 03:19 default.env
-rw-r--r-- 1 applmgr dba 1324 Mar 2 03:19 ftrace.cfg
-rw-r--r-- 1 applmgr dba 7614 Mar 2 03:19 socket.env
-rwx------ 1 applmgr dba 3032 Mar 2 03:19 txkLoadXMLInDB.sh
-rwx------ 1 applmgr dba 1474 Mar 2 03:19 txkcsa.cfg
bash-3.2$ grep LD_PRELOAD_32 *
default.env:LD_PRELOAD_32=/usr/lib/extendedFILE.so.1
socket.env:LD_PRELOAD_32=/usr/lib/extendedFILE.so.1
bash-3.2$
|
Consequently, you need to create a custom txkGenDefaultEnv.tmp file, which in turn will amend the generated files, default.env and socket.env.
|
bash-3.2# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ . env_eweb-lh
$ bash
bash-3.2$ pwd
/ebsfs
bash-3.2$ find . -name txkGenDefaultEnv.tmp
./d01/oracle/PROD/apps/apps_st/appl/fnd/12.0.0/admin/template/txkGenDefaultEnv.tmp
bash-3.2$
bash-3.2$ cd /ebsfs/d01/oracle/PROD/apps/apps_st/appl/fnd/12.0.0/admin/template/custom
bash-3.2$ cp ../txkGenDefaultEnv.tmp .
bash-3.2$ vi txkGenDefaultEnv.tmp
|
Edit ~/template/custom/txkGenDefaultEnv.tmp and add the highlighted line. Note, the file is too long to list, so only the relevant section that needs changing is listed below.
|
if (checkFileExists($temp_file))
{
$temp_file = join '', $ENV{'LD_PRELOAD_32'}, ':', $temp_file;
$io->print("LD_PRELOAD_32=$temp_file \n");
$io_sock->print("LD_PRELOAD_32=$temp_file \n");
}
|
The subsequent generated files will then contain and appended LD_PRELOAD_32 value:
|
bash-3.2$ pwd
bash-3.2$ /ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh/ora/10.1.2/forms/server
bash-3.2$ ls -l
total 139
-rw-r--r-- 1 applmgr dba 31374 Mar 2 03:19 Appsweb.cfg
-rw-r--r-- 1 applmgr dba 18698 Mar 2 03:19 default.env
-rw-r--r-- 1 applmgr dba 1324 Mar 2 03:19 ftrace.cfg
-rw-r--r-- 1 applmgr dba 7614 Mar 2 03:19 socket.env
-rwx------ 1 applmgr dba 3032 Mar 2 03:19 txkLoadXMLInDB.sh
-rwx------ 1 applmgr dba 1474 Mar 2 03:19 txkcsa.cfg
bash-3.2$
-bash-4.1$ grep LD_PRELOAD_32 *
default.env:LD_PRELOAD_32=/usr/lib/secure/libschost.so.1:/usr/lib/extendedFILE.so.1
socket.env:LD_PRELOAD_32=/usr/lib/secure/libschost.so.1:/usr/lib/extendedFILE.so.1
-bash-4.1$
|
4.4.4.3 AutoConfig for eweb-lh, iopmn1-lh and iopmn2-lh
The following shows autoConfig for eweb-lh. This needs to be repeated for iopmn1-lh and iopmn2-lh, by sourcing the correct env file prior to running AutoConfig.
|
# su - applmgr
Oracle Corporation SunOS 5.10 Generic Patch January 2005
$ bash
bash-3.2$
bash-3.2$ . env_eweb-lh
bash-3.2$ cd $ADMIN_SCRIPTS_HOME
bash-3.2$ ls -l
total 302
-rwx------ 1 applmgr dba 7422 Feb 21 02:45 adalnctl.sh
-rwx------ 1 applmgr dba 8501 Feb 21 02:46 adapcctl.sh
-rwx------ 1 applmgr dba 1572 Feb 21 02:45 adautocfg.sh
-rwx------ 1 applmgr dba 16589 Feb 21 02:45 adcmctl.sh
-rwx------ 1 applmgr dba 7288 Feb 21 02:45 adexecsql.pl
-rwx------ 1 applmgr dba 9007 Feb 21 02:46 adformsctl.sh
-rwx------ 1 applmgr dba 10377 Feb 21 02:46 adformsrvctl.sh
-rwx------ 1 applmgr dba 8259 Feb 21 02:46 adoacorectl.sh
-rwx------ 1 applmgr dba 8267 Feb 21 02:46 adoafmctl.sh
-rwx------ 1 applmgr dba 7513 Feb 21 02:46 adopmnctl.sh
-rwx------ 1 applmgr dba 15500 Feb 21 02:45 adpreclone.pl
-rwx------ 1 applmgr dba 6060 Feb 21 02:45 adstpall.sh
-rwx------ 1 applmgr dba 6204 Feb 21 02:45 adstrtal.sh
-rwx------ 1 applmgr dba 2250 Feb 21 02:46 gsmstart.sh
drwxr-xr-x 2 applmgr dba 10 Feb 21 02:44 ieo
-rwx------ 1 applmgr dba 2552 Feb 21 02:46 java.sh
-rwx------ 1 applmgr dba 6769 Feb 21 02:48 jtffmctl.sh
drwxr-xr-x 2 applmgr dba 3 Feb 21 02:44 msc
-rwxrwxr-- 1 applmgr dba 6648 Feb 21 02:48 mwactl.sh
-rwxrwxr-- 1 applmgr dba 6645 Feb 21 02:48 mwactlwrpr.sh
bash-3.2$ ./adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /ebsfs/d01/oracle/PROD/inst/apps/PROD_iopmn2-lh/admin/log/02210447/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
Using CONFIG_HOME location : /ebsfs/d01/oracle/PROD/inst/apps/PROD_iopmn2-lh
Classpath : /ebsfs/d01/oracle/PROD/apps/apps_st/comn/java/lib/Appsborg2.zip:/ebsfs/d01/oracle/PROD/apps/apps_st/comn/java/classes
Using Context file : /ebsfs/d01/oracle/PROD/inst/apps/PROD_iopmn2-lh/appl/admin/PROD_iopmn2-lh.xml
Context Value Management will now update the Context file
Context Value Management has found new variables that must be defined to update the context file and continue with system configuration
Variable : s_javamailer_imap_user
Details : The Java-based Oracle Workflow Notification Mailer, requires an
IMAP username for inbound mail processing. If the Java-based
Workflow Notification Mailer is not being used, accept the default
parameter value. For more information, please refer to the Oracle
Workflow Administrator's Guide.
Enter the IMAP username [NoImapUser]:
Variable : s_javamailer_reply_to
Details : The Java-based Oracle Workflow Notification Mailer, requires a
reply-to email address for inbound mail processing. If the Java-based
Workflow Notification Mailer is not being used, accept the default
parameter value. For more information, please refer to the Oracle
Workflow Administrator's Guide.
Enter the IMAP reply-to email address [NoReplyTo]:
Variable : s_javamailer_imaphost
Details : The Java-based Oracle Workflow Notification Mailer, requires an
IMAP server hostname for inbound mail processing. If the Java-based
Workflow Notification Mailer is not being used, accept the default
parameter value. For more information, please refer to the Oracle
Workflow Administrator's Guide.
Enter the IMAP server hostname [NoImapHost]:
Variable : s_javamailer_imapdomainname
Details : The Java-based Oracle Workflow Notification Mailer, requires an
IMAP domain name for inbound mail processing . If the Java-based
Workflow Notification Mailer is not being used, accept the default
parameter value. For more information, please refer to the Oracle
Workflow Administrator's Guide.
Enter the IMAP domain name [NoImapDomain]:
Variable : s_ohs_serveradmin
Details : This parameter sets the e-mail address that the OHS
includes in any error messages it returns to the client.
It may be worth setting up a dedicated address for this purpose.
Enter the Oracle HTTP Server Administrator email address [applmgr@us.oracle.com]:
Updating Context file...COMPLETED
Attempting upload of Context file and templates to database...COMPLETED
Configuring templates from all of the product tops...
Configuring AD_TOP........COMPLETED
Configuring FND_TOP.......COMPLETED
Configuring ICX_TOP.......COMPLETED
Configuring MSC_TOP.......COMPLETED
Configuring IEO_TOP.......COMPLETED
Configuring BIS_TOP.......COMPLETED
Configuring AMS_TOP.......COMPLETED
Configuring CCT_TOP.......COMPLETED
Configuring WSH_TOP.......COMPLETED
Configuring CLN_TOP.......COMPLETED
Configuring OKE_TOP.......COMPLETED
Configuring OKL_TOP.......COMPLETED
Configuring OKS_TOP.......COMPLETED
Configuring CSF_TOP.......COMPLETED
Configuring IGS_TOP.......COMPLETED
Configuring IBY_TOP.......COMPLETED
Configuring JTF_TOP.......COMPLETED
Configuring MWA_TOP.......COMPLETED
Configuring CN_TOP........COMPLETED
Configuring CSI_TOP.......COMPLETED
Configuring WIP_TOP.......COMPLETED
Configuring CSE_TOP.......COMPLETED
Configuring EAM_TOP.......COMPLETED
Configuring FTE_TOP.......COMPLETED
Configuring ONT_TOP.......COMPLETED
Configuring AR_TOP........COMPLETED
Configuring AHL_TOP.......COMPLETED
Configuring OZF_TOP.......COMPLETED
Configuring IES_TOP.......COMPLETED
Configuring CSD_TOP.......COMPLETED
Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.
bash-3.2$
|
Repeat AutoConfig (shown above) for iopmn1-lh and iopmn2-lh.
After AutoConfig has been completed successfully run for eweb-lh, iopmn1-lh and iopmn2-lh, start the App Tiers:
For eweb-lh, start the following from $ADMIN_SCRIPTS_HOME:
For iopmn1-lh and iopmn2-lh, start the following from $ADMIN_SCRIPTS_HOME:
|
adopmnctl.sh
adformsctl.sh
adoacorectl.sh
adoafmctl.sh
|
Note: before starting each component, you will need to ensure that the Solaris Cluster logical host is online on the correct node. Furthermore, you must ensure that you issue the start on the zone cluster node where that Solaris Cluster logical host is online:
|
bash-3.2# clrs status -t SUNW.LogicalHostname
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
opmn1-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
opmn2-lh iopmn2 Online Online - LogicalHostname online.
iopmn1 Offline Offline
web-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
bash-3.2#
|
Once all App Tiers eweb-lh, iopmn1-lh and iopmn2-lh have been started, you can confirm that everything has been successful:
|
bash-3.2$ ./adopmnctl.sh @cluster status
You are running adopmnctl.sh version 120.6
Checking status of OPMN managed processes in a cluster...
Processes in Instance: PROD_iopmn2-lh.iopmn2-lh.us.oracle.com
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group | OC4J:oafm | 5353 | Alive
OC4JGroup:default_group | OC4J:forms | 5355 | Alive
OC4JGroup:default_group | OC4J:oacore | 5350 | Alive
HTTP_Server | HTTP_Server | N/A | Down
Processes in Instance: PROD_eweb-lh.eweb-lh.us.oracle.com
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
HTTP_Server | HTTP_Server | 25573 | Alive
Processes in Instance: PROD_iopmn1-lh.iopmn1-lh.us.oracle.com
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group | OC4J:oafm | 4968 | Alive
OC4JGroup:default_group | OC4J:forms | 4895 | Alive
OC4JGroup:default_group | OC4J:oacore | 4936 | Alive
HTTP_Server | HTTP_Server | N/A | Down
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /ebsfs/d01/oracle/PROD/inst/apps/PROD_iopmn1-lh/logs/appl/admin/log/adopmnctl.txt for more information ...
bash-3.2$
|
Finally, to test you can login using the following URL:
|
http://eweb-lh.us.oracle.com:8000
|
Log into the site using username/password sysadmin/sysadmin:
4.4.5 Configure Parallel Concurrent Processing
Once you can login, you can setup PCP.
From the login screen, select the following:
· Select “System Administration”
· Select “System Configuration Overview”
At the top of the screen:
· Select “Site Map”
· Select “Generic Services”
· Select/View Details “Internal Monitor”
· Select/Edit Internal Monitor for ICM1 [Repeat for ICM2]
· Within Work Shifts Select “Add from Available Shifts”
· Include “Standard: Active 24 hours every day” and “Create New”
· Within Work Shifts include 1 “Processes”
At the top of the screen:
· Select “Site Map”
· Select “Parallel Concurrent Programming Setup”
· Expand All Managers
· Set Primary & Secondary Nodes for the following:
· Standard manager
· Scheduler/Prereleaser Manager
· Conflict Resolution Manager
· Internal Manager
· Click update/OK
Back to the home page:
· Select the “System Administrator”
· Select “Concurrent Requests”
· From the Forms page, double click on
· Concurrent Manager
· Define
· Click the find icon
· Select Standard Manager / Application Object Library
· Change “Standard Manager” to “Standard Manager 2”, i.e. you are creating another “Standard Manager”
· Set Primary & Secondary Nodes to icm2 and icm1 – hit enter takes you to work shifts where you can specify the number of processes (128)
· Click save or save and proceed.
· Amend “Standard Manager” to also have 128 processes.
Back to the home page.
· Select the “System Administration”
· Select “System Configuration Overview”
· Select “Site Map”
· Select “Request Processing Managers”
· Select “Start All” and Go
· Select/Edit “Select Standard Manager” [Repeat for Standard Manager 2]
· Within Work Shifts, set 64 “Failover Processes”
Run AutoConfig for icm1 and icm2. Once AutoConfig has completed successfully, you can start the Concurrent Managers.
For icm1 and icm2, start the following from $ADMIN_SCRIPTS_HOME:
To confirm that the Concurrent Managers are running, list the processes.
|
bash-3.2# uname -n
icm1
bash-3.2# ptree applmgr
1433 zsched
28235 /ebsfs/d01/oracle/PROD/apps/tech_st/10.1.2/bin/tnslsnr APPS_PROD -inher
28266 sh
28275 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" sleep=30 pmon=4 quesiz=1 diag=N
28736 FNDSM
28772 RCVOLTM APPS/ZG483D2F7114904055008103189D960EBFBE03E0AADFAEC8BA8C7380
28773 POXCON APPS/ZG0D84C168C0E04EF084D4CF0E69BC01EFA85E0C326B65C89C4A6B742
28774 INCTM APPS/ZG172D19C6D57977BE1F5F2B7D2940F5A9FCB601A56BF1B3AA14B45759
28775 INVLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG7CF9C741D7342C
28776 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG6E7EF00BF28238
28777 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGE2EB3DA8FA3C79
28778 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGEC9B00243F4FC8
28779 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28780 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28781 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28782 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28783 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28784 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28785 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28786 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28787 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28788 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG3189A032764BA7
28839 PALIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZG921D15256C9AE55
bash-3.2#
bash-3.2# uname -n
icm2
bash-3.2# ptree applmgr
1260 zsched
12001 /ebsfs/d01/oracle/PROD/apps/tech_st/10.1.2/bin/tnslsnr APPS_PROD -inher
12708 FNDSM
12747 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGE3EF3BD44174FF
12748 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12749 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12750 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12751 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12752 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12753 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12754 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12755 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
12756 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/ZGAD30DFF383E299
bash-3.2#
|
To test that you can submit a job scheduled by a Concurrent Manager issue the following:
· Select “System Administration”
· Select “Concurrent Requests”
· Submit a New Request (Single Request)
· Enter name, “active users”, click submit/OK/No
· Click find (All my requests)
· Verify that you request completed normally
4.5 ZFS SA Snapshot
At this point, it is worthwhile to perform a ZFS SA snapshot.
Shutdown the DB and App Tiers. Then, login to the following URL:
|
https://etc11-sn1.us.oracle.com:215/
|
5. Consult Related MOS Notes
Before continuing, consult the following MOS Notes and take any indicated actions. It is recommended to perform a ZFS SA snapshot after completing configuration actions from the MOS Notes.
Because MOS Notes are subject to change, it is important to reference the most recent information on installing E-Business Suite on the SPARC platform. For that reason, specific configuration steps related to the MOS Note are not included in this document. The reader is referred to the MOS Note for the latest information.
5.1 Interoperability Notes EBS R12 with Database 11gR2 (MOS 1058763.1)
Before continuing, consult the following MOS Note:
· Interoperability Notes EBS R12 with Database 11gR2 [ID 1058763.1]
Implement the steps outlined in this MOS Note.
After performing any indicated configuration steps, it is worthwhile performing a ZFS SA snapshot.
Ensure that the DB/App Tiers are shutdown first. Then, login to to the following URL:
|
https://etc11-sn1.us.oracle.com:215/
|
5.2 Oracle E-Business Suite Release 12.1.3 Readme (MOS 1080973.1)
Before continuing, consult the following MOS Note:
· Oracle E-Business Suite Release 12.1.3 Readme [ID 1080973.1]
Implement the steps outlined in this MOS Note.
After performing any indicated configuration steps, it is worthwhile performing a ZFS SA snapshot.
Ensure that the DB/App Tiers are shutdown first. Then, login to to the following URL:
|
https://etc11-sn1.us.oracle.com:215/
|
5.3 Best Practices E-Business Suite Performance Tuning
It is recommended that users consult the following MOS resources for guidelines on performance tuning:
· EBS - Webcast Recording Best Practices E-Business Suite Performance Tuning [Audio] [ID 1530509.1]
· Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 [ID 823587.1]
· Optimizing Item Import Performance in Oracle Product Hub/Inventory [ID 1208945.1]
· Useful Articles for the Oracle E-Business Suite Technology Stack for Technical Consultant / DBA [ID 828157.1]
· Configuring various JVM tuning parameters for Oracle E-Business suite 11i and R12 [ID 567551.1]
· Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1.3 [ID 1362660.1]
· Tuning performance on eBusiness suite [ID 744143.1]
· How Can I Diagnose Poor E-Business Suite Performance? [ID 864226.1]
· E-Business Suite Diagnostics Tips and Tricks [ID 398665.1]
· E-Business Suite Diagnostic Tools FAQ and Troubleshooting Guide for Release 11i and R12 [ID 235307.1]
· Concurrent Processing - Best Practices for Performance for Concurrent Managers in E-Business Suite [ID 1057802.1]
Also, refer to Error: Reference source not found for additional E-Business Suite tuning best practices.
6. Solaris Cluster Configuration
This section includes steps to configure the Solaris Cluster resources. Specifically:
· Create DB Tier resources
· Create App Tier OPMN resources
· Create App Tier CM resources
6.1 Create DB Tier resources
Create the DB Tier resources on the zone cluster db-zc:
· Resource for DB (db-rs)
· Resource for Listener (lsr-rs)
6.1.1 Create database fault monitor user
On zone cluster node idb1:
|
bash-3.2# uname -n
idb1
bash-3.2#
|
Connect to the database and setup the fault monitor user:
|
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> create user hauser identified by hauser;
User created.
SQL> alter user hauser default tablespace system quota 1m on system;
User altered.
SQL> grant select on v_$sysstat to hauser;
Grant succeeded.
SQL> grant select on v_$archive_dest to hauser;
Grant succeeded.
SQL> grant create session to hauser;
Grant succeeded.
SQL> grant create table to hauser;
Grant succeeded.
SQL> grant select on v_$database to hauser;
Grant succeeded.
SQL> grant execute on dbms_lock to hauser;
Grant succeeded.
SQL> grant select on v_$instance to hauser;
Grant succeeded.
SQL> create profile hauser limit PASSWORD_LIFE_TIME UNLIMITED;
Profile created.
SQL> alter user hauser identified by hauser profile hauser;
User altered.
SQL>
|
6.1.2 Create the DB & Listener resources
Create the DB and Listener resources (db-rs and lsr-rs) on zone cluster idb-zc:
|
bash-3.2# uname -n
idb1
bash-3.2#
bash-3.2# clrt register SUNW.oracle_server
bash-3.2# clrt register SUNW.oracle_listener
bash-3.2# clrs create -g db-rg -t SUNW.oracle_server -p connect_string=hauser/hauser -p oracle_sid=PROD -p oracle_home=/orainst/d01/oracle/PROD/db/tech_st/11.2.0 -p alert_log_file=/orainst/d01/oracle/PROD/db/tech_st/11.2.0/admin/PROD_idb-lh/diag/rdbms/prod/PROD/trace/alert_PROD.log -d db-rs
bash-3.2#
bash-3.2# cd /opt/SUNWscebs/cmg
bash-3.2# ls -l
total 9
drwxr-xr-x 2 root bin 6 Feb 15 22:41 bin
drwxr-xr-x 2 root bin 3 Feb 15 22:41 etc
drwxr-xr-x 2 root bin 5 Feb 15 22:41 util
bash-3.2# cd util
bash-3.2# ls -l
total 28
-rw-r--r-- 1 root bin 3751 Jul 26 2010 cmg_config
-r-xr-xr-x 1 root bin 2351 Mar 22 2011 cmg_register
-r-xr-xr-x 1 root bin 5679 Jul 26 2010 copy_env
bash-3.2#
bash-3.2# ./copy_env /orainst/d01/oracle/PROD/db/tech_st/11.2.0 PROD_idb-lh
Copied environment variables from /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh.env to /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
bash-3.2#
bash-3.2# clrs create -g db-rg -t SUNW.oracle_listener -p listener_name=LISTENER -p oracle_home=/orainst/d01/oracle/PROD/db/tech_st/11.2.0 -p user_env=/orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env -d lsr-rs
bash-3.2#
|
6.1.3 Online the DB & Listener resources
Configure the resource dependency of the newly created resources on the file system resources:
|
# clrs set -p Resource_dependencies_offline_restart=orainst-fs-rs lsr-rs db-rs
etc11-01-s10:db-zc - Ignoring ORACLE_HOME set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-01-s10:db-zc - Ignoring ORACLE_SID set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-01-s10:db-zc - Ignoring command execution `<command>`
etc11-02-s10:db-zc - Ignoring ORACLE_HOME set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-02-s10:db-zc - Ignoring ORACLE_SID set in environment file /orainst/d01/oracle/PROD/db/tech_st/11.2.0/PROD_idb-lh_ha.env
etc11-02-s10:db-zc - Ignoring command execution `<command>`
# clrs set -p Resource_dependencies_offline_restart+=oradata-fs-rs,oralog-fs-rs,oradump-fs-rs db-rs
|
Bring the DB and Listener resources online:
|
bash-3.2# clrg online -eM db-rg
bash-3.2# clrs status -g db-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
db-lh idb1 Offline Offline
idb2 Online Online - LogicalHostname online.
lsr-rs idb1 Offline Offline
idb2 Online Online
db-rs idb1 Offline Offline
idb2 Online Online
bash-3.2#
|
6.2 Create App Tier OPMN resources
Create the App Tier OPMN resources on the zone cluster opmn-zc:
· Resource for OPMN HTTP_SERVER (web-rs)
· Resources for OPMN OC4J (opmn1-rs and opmn2-rs)
6.2.1 Create the resource for OPMN HTTP_SERVER (web-rs)
As the App Tier OPMN resources are being deployed within a zone cluster, you need to temporarily lofs mount /var/cluster/logs from the global zone:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
|
Register the resource type:
|
bash-3.2# uname -n
iopmn1
bash-3.2# clrt register SUNW.gds
|
Edit the configuration file:
|
bash-3.2# cd /opt/SUNWscebs/opmn/util
bash-3.2# ls -l
total 13
-rw-r--r-- 1 root bin 2593 Mar 2 05:29 opmn_config
-r-xr-xr-x 1 root bin 2378 Jul 17 2011 opmn_register
bash-3.2#
bash-3.2# vi opmn_config
|
Edit the opmn_config file as follows:
|
RS=web-rs
RG=web-rg
LH=web-lh
HAS_RS=ebs-fs-rs
VERSION=12.1
COMNTOP=/ebsfs/d01/oracle/PROD/inst/apps/PROD_eweb-lh
APPSUSER=applmgr
APP_SID=PROD
OPMN_COMPONENTS="opmn/http_server"
HOSTNAME=
|
Register the web-rs resource:
|
bash-3.2# pwd
/opt/SUNWscebs/opmn/util
bash-3.2# ls -l
total 13
-rw-r--r-- 1 root bin 2593 Mar 2 05:29 opmn_config
-r-xr-xr-x 1 root bin 2378 Jul 17 2011 opmn_register
bash-3.2# ./opmn_register
bash-3.2# clrs enable web-rs
bash-3.2# clrs status -g web-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
web-rs iopmn1 Online Online - Service is online.
iopmn2 Offline Offline
web-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
Bash-3.2#
|
Unmount the file systems:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2# umount /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2# umount /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
|
6.2.2 Create the resource for OPMN OC4J (opmn1-rs)
As the APP Tier OPMN resources are being deployed within a zone cluster, you need to temporarily lofs mount /var/cluster/logs from the global zone:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
|
Edit the configuration file:
|
bash-3.2# uname -n
iopmn1
bash-3.2# cd /opt/SUNWscebs/opmn/util
bash-3.2# ls -l
total 13
-rw-r--r-- 1 root bin 2593 Mar 2 05:29 opmn_config
-r-xr-xr-x 1 root bin 2378 Jul 17 2011 opmn_register
bash-3.2#
bash-3.2# vi opmn_config
|
Edit the opmn_config file as follows:
|
RS=opmn1-rs
RG=opmn1-rg
LH=opmn1-lh
HAS_RS=ebs-fs-rs
VERSION=12.1
COMNTOP=/ebsfs/d01/oracle/PROD/inst/apps/PROD_iopmn1-lh
APPSUSER=applmgr
APP_SID=PROD
OPMN_COMPONENTS="opmn/oacore/forms/oafm"
HOSTNAME=
|
Register the opmn1-rs resource:
|
bash-3.2# pwd
/opt/SUNWscebs/opmn/util
bash-3.2# ls -l
total 13
-rw-r--r-- 1 root bin 2593 Mar 2 05:29 opmn_config
-r-xr-xr-x 1 root bin 2378 Jul 17 2011 opmn_register
bash-3.2# ./opmn_register
bash-3.2# clrs enable opmn1-rs
bash-3.2# clrs status -g opmn1-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
opmn1-rs iopmn1 Online Online - Service is online.
bash-3.2#
|
Unmount the file systems:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2# umount /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2# umount /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
|
6.2.3 Create the resource for OPMN OC4J (opmn2-rs)
As the APP Tier OPMN resources are being deployed within a zone cluster, you need to temporarily lofs mount /var/cluster/logs from the global zone:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
|
Edit the configuration file:
|
bash-3.2# uname -n
iopmn1
bash-3.2# cd /opt/SUNWscebs/opmn/util
bash-3.2# ls -l
total 13
-rw-r--r-- 1 root bin 2593 Mar 2 05:29 opmn_config
-r-xr-xr-x 1 root bin 2378 Jul 17 2011 opmn_register
bash-3.2#
bash-3.2# vi opmn_config
|
Edit the opmn_config file as follows:
|
RS=opmn2-rs
RG=opmn2-rg
LH=opmn2-lh
HAS_RS=ebs-fs-rs
VERSION=12.1
COMNTOP=/ebsfs/d01/oracle/PROD/inst/apps/PROD_iopmn2-lh
APPSUSER=applmgr
APP_SID=PROD
OPMN_COMPONENTS="opmn/oacore/forms/oafm"
HOSTNAME=
|
Register the opmn2-rs resource:
|
bash-3.2# pwd
/opt/SUNWscebs/opmn/util
bash-3.2# ls -l
total 13
-rw-r--r-- 1 root bin 2593 Mar 2 05:29 opmn_config
-r-xr-xr-x 1 root bin 2378 Jul 17 2011 opmn_register
bash-3.2# ./opmn_register
bash-3.2# clrs enable opmn2-rs
bash-3.2# clrs status -g opmn2-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
opmn2-rs iopmn2 Online Online - Service is online.
Bash-3.2#
|
Unmount the file systems:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2# umount /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2# umount /zones/opmn-zc/root/var/cluster/logs
bash-3.2#
|
6.2.4 Confirm App Tier OPMN Resource Status
Configure the resource dependency of the newly created resources on the file system resource:
|
# clrs set -p Resource_dependencies_offline_restart=ebs-fs-rs eweb-rs iopmn1-rs iopmn2-rs
|
Confirm the status of the OPMN resources created in the App Tier (zone cluster opmn-zc):
|
bash-3.2# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
opmn1-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
opmn2-lh iopmn2 Online Online - LogicalHostname online.
iopmn1 Offline Offline
web-rs iopmn1 Online Online - Service is online.
iopmn2 Offline Offline
web-lh iopmn1 Online Online - LogicalHostname online.
iopmn2 Offline Offline - LogicalHostname offline.
opmn1-rs iopmn1 Online Online - Service is online.
opmn2-rs iopmn2 Online Online - Service is online.
ebs-fs-rs iopmn1 Online Online
iopmn2 Online Online
bash-3.2#
|
6.3 Create App Tier CM Resources
Create the App Tier CM resources on the zone cluster cm-zc:
· Resource for Oracle External Proxy (oep-rs)
· Resource for CM Listener (cmlsr-rs)
· Resource for CM (cm-rs)
6.3.1 Create the Oracle External Proxy Resource (oep-rs)
Create a /var/opt/oracle/tnsnames.ora file on icm1 and icm2:
|
bash-3.2# cat /var/opt/oracle/tnsnames.ora
PROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb-lh.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=PROD)
(INSTANCE_NAME=PROD)
)
)
bash-3.2#
|
Create an encrypted password file on icm1 and icm2:
|
bash-3.2# dd if=/dev/urandom of=/var/cluster/scoep_key bs=8 count=1
bash-3.2# echo hauser | /usr/sfw/bin/openssl enc -aes128 -e -pass file:/var/cluster/scoep_key -out /opt/ORCLscoep/.oep-rs_passwd
bash-3.2#
|
Verify that the encrypted password can be decrypted:
|
bash-3.00# /usr/sfw/bin/openssl enc -aes128 -d -pass file:/var/cluster/scoep_key -in /opt/ORCLscoep/.oep-rs_passwd
hauser
bash-3.00#
bash-3.00# chmod 400 /var/cluster/scoep_key
bash-3.00# chmod 400 /opt/ORCLscoep/.oep-rs_passwd
|
Register the oep-rs resource:
|
bash-3.2# uname -n
icm1
bash-3.2#
bash-3.2# clrt register -f /opt/ORCLscoep/etc/ORCL.oracle_external_proxy ORCL.oracle_external_proxy
bash-3.2#
bash-3.00# clrg create -S oep-rg
bash-3.00# clrs create -g oep-rg -t ORCL.oracle_external_proxy -p service_name=prod -p ons_nodes=idb-lh.us.oracle.com:6200 -p dbuser=hauser -d oep-rs
bash-3.00#
bash-3.00# clrg online -eM oep-rg
bash-3.2# clrs status -g oep-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
oep-rs icm1 Online Online - Service PROD is UP
icm2 Online Online - Service PROD is UP
bash-3.2#
|
6.3.2 Create the resource for the CM Listener Resources (cmlsr-rs)
As the APP Tier CM resources are being deployed within a zone cluster, you need to temporarily lofs mount /var/cluster/logs from the global zone:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/cm-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/cm-zc/root/var/cluster/logs
bash-3.2#
|
Edit the configuration file:
|
bash-3.2# uname -n
icm1
bash-3.2#
bash-3.2# cd /opt/SUNWscebs/cmglsr/util
bash-3.2# ls -l
total 12
-rw-r--r-- 1 root bin 2679 Feb 29 07:55 cmglsr_config
-r-xr-xr-x 1 root bin 1948 Mar 22 2011 cmglsr_register
bash-3.2#
bash-3.2# vi cmglsr _config
|
Edit the cmglsr_config file as follows:
|
RS=cmlsr-rs
RG=cm-rg
LH=
HAS_RS=
COMNTOP=/ebsfs/d01/oracle/PROD/inst/apps
APPSUSER=applmgr
APP_SID=PROD
VERSION=12.1
|
Register the cmlsr-rs resource:
|
bash-3.2# pwd
/opt/SUNWscebs/cmglsr/util
bash-3.2# ls -l
total 12
-rw-r--r-- 1 root bin 2679 Feb 29 07:55 cmglsr_config
-r-xr-xr-x 1 root bin 1948 Mar 22 2011 cmglsr_register
bash-3.2# ./cmglsr_register
bash-3.2# clrs enable cmlsr-rs
bash-3.2# clrs status cmlsr-rs
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
cmlsr-rs icm1 Online Online - Service is online.
icm2 Online Online - Service is online.
bash-3.2#
|
Unmount the file systems:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2# umount /zones/cm-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2# umount /zones/cm-zc/root/var/cluster/logs
bash-3.2#
|
6.3.3 Create the resource for the CM Resources (cm-rs)
As the APP Tier CM resources are being deployed within a zone cluster, you need to temporarily lofs mount /var/cluster/logs from the global zone:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/cm-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2#
bash-3.2# mount -F lofs /var/cluster/logs /zones/cm-zc/root/var/cluster/logs
bash-3.2#
|
Edit the configuration file for the cm-rs resource:
|
bash-3.2# uname -n
icm1
bash-3.2#
bash-3.2# cd /opt/SUNWscebs/cmg/util
bash-3.2# ls -l
total 28
-rw-r--r-- 1 root bin 3857 Feb 29 07:57 cmg_config
-r-xr-xr-x 1 root bin 2351 Mar 22 2011 cmg_register
-r-xr-xr-x 1 root bin 5679 Jul 26 2010 copy_env
bash-3.2#
bash-3.2# vi cmg_config
|
Edit the cmg_config file as follows:
|
RS=cm-rs
RG=cm-rg
LH=
HAS_RS=
LSR_RS=cmlsr-rs
VERSION=12.1
COMNTOP=/ebsfs/d01/oracle/PROD/inst/apps
APPSUSER=applmgr
APP_SID=PROD
APPS_PASSWD=apps
ORACLE_HOME=/ebsfs/d01/oracle/PROD/apps/tech_st/10.1.2
CON_LIMIT=0
#
# Required for Oracle E-Business Suite version 11.5.10 CU2 or later
#
ORASVR_RS=
ORALSR_RS=
|
Register the cm-rs resource:
|
bash-3.2# pwd
/opt/SUNWscebs/cmg/util
bash-3.2# ls -l
total 28
-rw-r--r-- 1 root bin 3857 Feb 29 07:57 cmg_config
-r-xr-xr-x 1 root bin 2351 Mar 22 2011 cmg_register
-r-xr-xr-x 1 root bin 5679 Jul 26 2010 copy_env
bash-3.2# ./cmg_register
bash-3.2# clrs enable cm-rs
bash-3.2# clrs status cm-rs
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
cm-rs icm1 Online Online - Service is online.
icm2 Online Online - Service is online.
bash-3.2#
|
Set resource dependency:
|
bash-3.2# clrs set -p Resource_dependencies_offline_restart+=oep-rs cm-rs
|
Unmount file systems:
|
bash-3.2# uname -n
etc11-01-s10
bash-3.2# umount /zones/cm-zc/root/var/cluster/logs
bash-3.2#
bash-3.2# uname -n
etc11-02-s10
bash-3.2# umount /zones/cm-zc/root/var/cluster/logs
bash-3.2#
|
6.3.4 Confirm App Tier CM Resource Status
Configure the resource dependency of the newly created CM resources on the file system resource:
|
# clrs set -p Resource_dependencies_offline_restart+=ebs-fs-rs cmlsr-rs cm-rs
|
Confirm the status of the CM resources created in the App Tier (zone cluster cm-zc):
|
bash-3.2# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
cm-rs icm1 Online Online - Service is online.
icm2 Online Online - Service is online.
cmlsr-rs icm1 Online Online - Service is online.
icm2 Online Online - Service is online.
oep-rs icm1 Online Online - Service PROD is UP
icm2 Online Online - Service PROD is UP
ebs-fs-rs icm1 Online Online
icm2 Online Online
bash-3.2#
|
7. Convert Oracle 11g Database to Oracle RAC
Consult the following MOS Note for information on converting the single instance Oracle 11g Database to Oracle RAC:
· Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite R12 [ID 823587.1]
Implement the steps outlined in this MOS Note before continuing.
8. Post Oracle RAC migration changes
After the migration to Oracle RAC, the following steps need to be performed:
· Modify the Oracle RAC Listeners to use IB VIP interfaces
· Modify the Oracle External Proxy resource
8.1 Modify Oracle RAC Listeners to use IB VIP Interfaces
Here, you will makes changes so that the Oracle RAC Listeners will use the IB VIP interfaces.
As user oracle in the DB domain add the public IB network interface:
|
-bash-4.1$ oifcfg getif
bondeth0 10.129.205.0 global public
bondib0 192.168.10.0 global cluster_interconnect
bondib1 192.168.10.0 global cluster_interconnect
-bash-4.1$
-bash-4.1$ oifcfg iflist
bondeth0 10.129.205.0
bondib0 192.168.10.0
bondib1 192.168.10.0
ipmpapp0 192.168.30.0
-bash-4.1$
-bash-4.1$ srvctl config network
Network exists: 1/10.129.205.0/255.255.255.0/bondeth0, type static
-bash-4.1$
-bash-4.1$ oifcfg setif -global ipmpapp0/192.168.30.0:public
-bash-4.1$
-bash-4.1$ oifcfg getif
bondeth0 10.129.205.0 global public
bondib0 192.168.10.0 global cluster_interconnect
bondib1 192.168.10.0 global cluster_interconnect
ipmpapp0 192.168.30.0 global public
|
Then as root, add a VIP to each node:
|
root@etc11-01:~# srvctl add vip -n etc11-01 -A etc11-01-ib-vip/255.255.255.0 -k 2
root@etc11-01:~# srvctl add vip -n etc11-02 -A etc11-02-ib-vip/255.255.255.0 -k 2
|
Back to oracle user, add and start the IB Listeners:
|
-bash-4.1$ srvctl add listener -l LISTENER_IB -k 2 -p 1521 -s
-bash-4.1$
-bash-4.1$ srvctl status vip -n etc11-01
VIP etc11-01-ib-vip is enabled
VIP etc11-01-ib-vip is not running
VIP etc11-01-vip is enabled
VIP etc11-01-vip is running on node: etc11-01
-bash-4.1$
-bash-4.1$ srvctl status vip -n etc11-02
VIP etc11-02-ib-vip is enabled
VIP etc11-02-ib-vip is not running
VIP etc11-02-vip is enabled
VIP etc11-02-vip is running on node: etc11-02
-bash-4.1$
-bash-4.1$ srvctl status listener -l listener_ib
Listener LISTENER_IB is enabled
Listener LISTENER_IB is not running
-bash-4.1$
-bash-4.1$ srvctl start listener -l listener_ib
-bash-4.1$
-bash-4.1$ srvctl status vip -n etc11-01
VIP etc11-01-ib-vip is enabled
VIP etc11-01-ib-vip is running on node: etc11-01
VIP etc11-01-vip is enabled
VIP etc11-01-vip is running on node: etc11-01
-bash-4.1$
-bash-4.1$ srvctl status vip -n etc11-02
VIP etc11-02-ib-vip is enabled
VIP etc11-02-ib-vip is running on node: etc11-02
VIP etc11-02-vip is enabled
VIP etc11-02-vip is running on node: etc11-02
-bash-4.1$
-bash-4.1$ srvctl status listener -l listener_ib
Listener LISTENER_IB is enabled
Listener LISTENER_IB is running on node(s): etc11-01,etc11-02
|
Add entries in the $ORACLE_HOME/network/admin/tnsnames.ora file.
Note: There are different definitions for each DB domain.
On DB domain etc11-01:
|
LISTENER_IBREMOTE =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-02-ib-vip)(PORT = 1521))
)
)
LISTENER_IBLOCAL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-01-ib-vip)(PORT = 1521))
)
)
LISTENER_10GBEREMOTE =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-scan)(PORT = 1521))
)
)
LISTENER_10GBELOCAL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-01-vip)(PORT = 1521))
)
)
|
On DB domain etc11-02:
|
LISTENER_IBREMOTE =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-01-ib-vip)(PORT = 1521))
)
)
LISTENER_IBLOCAL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-02-ib-vip)(PORT = 1521))
)
)
LISTENER_10GBEREMOTE =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-scan)(PORT = 1521))
)
)
LISTENER_10GBELOCAL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = etc11-02-vip)(PORT = 1521))
)
)
|
Amend the listener_networks parameter:
|
-bash-4.1$ sqlplus "/as sysdba"
SQL*Plus: Release 11.2.0.3.0 Production on Thu May 31 16:33:49 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> show parameter listener
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
listener_networks string
local_listener string (DESCRIPTION=(ADDRESS_LIST=(AD
DRESS=(PROTOCOL=TCP)(HOST=10.1
29.186.13)(PORT=1521))))
remote_listener string etc11-scan:1521
SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_10GBELOCAL)(REMOTE_LISTENER=LISTENER_10GBEREMOTE))' scope=both;
System altered.
SQL> show parameter listener
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
listener_networks string ((NAME=network2)(LOCAL_LISTENE
R=LISTENER_IBLOCAL)(REMOTE_LIS
TENER=LISTENER_IBREMOTE)), ((N
AME=network1)(LOCAL_LISTENER=L
ISTENER_10GBELOCAL)(REMOTE_LIS
TENER=LISTENER_10GBEREMOTE))
local_listener string (DESCRIPTION=(ADDRESS_LIST=(AD
DRESS=(PROTOCOL=TCP)(HOST=10.1
29.186.13)(PORT=1521))))
remote_listener string etc11-scan:1521
SQL> alter system register;
System altered.
|
At this point the EBS app tier should be configured with RAC listeners on IB. Verify the connectivity:
|
$ sqlplus apps/apps@VIS_BALANCE
SQL*Plus: Release 10.1.0.5.0 - Production on Thu May 31 16:36:20 2012
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select node_name, node_mode, support_cp, support_web, support_admin, support_forms from FND_NODES;
NODE_NAME N S S S S
------------------------------ - - - - -
ETC11-01 O N N N N
ETC11-02 O N N N N
EWEB-LH O Y Y Y Y
AUTHENTICATION O N N N N
IOPMN1-LH O Y Y Y Y
IOPMN2-LH O Y Y Y Y
ICM1 O Y Y Y Y
ICM2 O Y Y Y Y
8 rows selected.
=> OK
|
8.2 Modify the Oracle External Proxy resource
After you have migrated the Oracle database to RAC DB, you need to reflect that change within the Oracle Solaris Cluster Oracle External Proxy resource.
In this regard, you simply need to disable the OEP resource and amend the /var/opt/oracle/tnsnames.ora file on each zone cluster node. Then you can enable the OEP resource again.
Disable the OEP resource:
|
bash-3.2# uname -n
icm1
bash-3.2#
bash-3.2# clrs disable oep-rs
|
Amend the /var/opt/oracle/tnsnames.ora file as follows:
|
bash-3.2# cat /var/opt/oracle/tnsnames.ora
PROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=etc11-01-ib-vip.us.oracle.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=etc11-02-ib-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=PROD)
)
)
bash-3.2#
|
Re-enable the OEP resource:
|
bash-3.2# clrs enable oep-rs
|
<NOTE:823587.1> - Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12
<NOTE:761568.1> - R12.1.1 : Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Oracle Solaris on SPARC (64-bit)
