Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1466664.1
Update Date:2015-02-05
Keywords:

Solution Type  Problem Resolution Sure

Solution  1466664.1 :   ODA (Oracle Database Appliance): GI update is failing with oraInventory corruption  


Related Items
  • Oracle Database - Enterprise Edition
  •  
  • Oracle Database Appliance
  •  
  • Oracle Database Appliance Software
  •  
Related Categories
  • PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Database Appliance>DB: ODA_EST
  •  




Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 to 11.2.0.4 [Release 11.2]
Oracle Database Appliance Software - Version 2.2.0.0 to 2.10.0.0
Oracle Database Appliance - Version All Versions and later
Information in this document applies to any platform.

Symptoms

 On ODA (Oracle Database Appliance) you are upgrading the Grid Infrastructure (GI) the following command:

# cd /opt/oracle/oak/bin
# ./oakcli update -patch <new version> --gi

and you are receiving the following errors:

(...)
INFO   : Running on the local node: /bin/su grid -c /tmp/opfile
INFO   : Background process 28421 (node: zaoda101) gets done with the exit code 0
INFO   : This is root, will become grid and run: /bin/su grid -c /usr/bin/ssh -l grid zaoda102 /tmp/opfile
INFO   : Background process 28490 (node: zaoda102) gets done with the exit code 0
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
ERROR: 2012-06-12 08:10:24: Failed to upgrade the GI to 11.2.0.3.2(13696216,13696251)
...
INFO   : Running as root: /usr/bin/ssh -l root zaoda102 /tmp/update_inventory

INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /opt/oracle/oak/pkgrepos/System/2.2.0.0.0/bin/postpatch -v 2.2.0.0.0 --gi

Moreover issuing:

/opt/oracle/oak/bin/oakcli show version -detail

you are getting the following errors (the GI home is also missing):

# /opt/oracle/oak/bin/oakcli show version -detail
Reading the metadata. It takes a while...
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
System Version  Component Name            Installed Version         Supported Version       
--------------  ---------------           ------------------        -----------------       
2.2.0.0.0                                                                                   
                Controller                05.00.29.00               Up-to-date              
                Expander                  0342                      Up-to-date              
                SSD_SHARED                E125                      Up-to-date              
                HDD_LOCAL                 SA03                      Up-to-date              
                HDD_SHARED                0A25                      Up-to-date              
                ILOM                      3.0.14.13.a r70764        Up-to-date              
                BIOS                      12010304                  Up-to-date              
                IPMI                      1.8.10.4                  Up-to-date              
                HMP                       2.2.0                     Up-to-date              
                OAK                       2.2.0.0.0                 Up-to-date              
                OEL                       5.8                       Up-to-date              
                DB_HOME                   11.2.0.2.5(13343424,      No-update               
                                          13343447)                                         
                ASR                       Unknown                   3.7         

Checking your SCAN port setting you are getting more then one port: 1521 & 1522 (in this case):

/u01/app/11.2.0/grid/bin/srvctl config scan_listener -i 1
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521,1522

 

Note as this article is applicable to
  • GI update from 11.2.0.3.x to 11.2.0.4.x
  • GI update from 11.2.0.2.x to 11.2.0.3.x

----------------------------------------------------------------------------------------------------------------------------

>>>> This article is not applicable when you are upgrading to GI 12c <<<<

----------------------------------------------------------------------------------------------------------------------------


Changes

You have add another port to the SCAN (Single Client Access Network), executing in example a command like:

/u01/app/11.2.0/grid/bin/srvctl modify scan_listener -p TCP:1521/TCP:1522

Cause

 Due to a bug on 11.2.0.3 the SCAN listener can have just one port. The GI upgrade process is generating the grid response file (grid.rsp) with the following entry:

(...)
oracle.install.crs.config.gpnp.scanPort=1521,1522
(...)

as consequence the GI clone process will fails and under "/u01/app/oraInventory/logs" on the oraInventory configAction file you will find the following entry:

INFO: Createing properties map - in ExtendedPropertyFileFormat.loadPropertiesMap()
Jun 12, 2012 8:08:34 AM oracle.install.commons.util.exception.DefaultErrorAdvisor$AbstractErrorAdvisor getDetailedMessage
SEVERE: [FATAL] [INS-10105] The given response file /opt/oracle/oak/pkgrepos/System/2.2.0.0.0/bin/tmp/grid.rsp is not valid.
   CAUSE: Syntactically incorrect response file. Either unexpected variables are specified or expected variables are not specified in the response file.
   ACTION: Refer the latest product specific response file template
   SUMMARY:
       - cvc-datatype-valid.1.2.1: '1521,1522' is not a valid value for 'integer'.
cvc-type.3.1.3: The value '1521,1522' of element 'oracle.install.crs.config.gpnp.scanPort' is not valid.

oracle.install.commons.base.driver.common.InstallerException: [INS-10105] The given response file /opt/oracle/oak/pkgrepos/System/2.2.0.0.0/bin/tmp/grid.rsp is not valid.
        at oracle.install.commons.base.driver.common.Installer.validateResponseFile(Installer.java:375)
        at oracle.install.commons.base.driver.common.Installer.run(Installer.java:327)
        at oracle.install.ivw.common.util.OracleInstaller.run(OracleInstaller.java:106)
        at oracle.install.commons.util.Application.startup(Application.java:891)
        at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:165)
        at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:182)
        at oracle.install.commons.base.driver.common.Installer.startup(Installer.java:348)
        at oracle.install.ivw.crs.driver.CRSConfigWizard.startup(CRSConfigWizard.java:84)
        at oracle.install.ivw.crs.driver.CRSConfigWizard.main(CRSConfigWizard.java:91)
Caused by: java.lang.Exception: cvc-datatype-valid.1.2.1: '1521,1522' is not a valid value for 'integer'.
cvc-type.3.1.3: The value '1521,1522' of element 'oracle.install.crs.config.gpnp.scanPort' is not valid.

        at oracle.install.commons.util.XmlSupport.validate(XmlSupport.java:110)
        at oracle.install.commons.bean.xml.XmlBeanStoreFormat.validate(XmlBeanStoreFormat.java:201)
        at oracle.install.commons.bean.xml.PropertyFileFormat.validate(PropertyFileFormat.java:144)
        at oracle.install.commons.base.driver.common.Installer.validateResponseFile(Installer.java:373)
        ... 8 more
Jun 12, 2012 8:08:34 AM oracle.install.commons.util.exception.DefaultErrorAdvisor$AbstractErrorAdvisor advise
INFO: Advice is ABORT
Jun 12, 2012 8:08:34 AM oracle.install.commons.util.exception.DefaultExceptionHandler handleException
SEVERE: Unconditional Exit
Jun 12, 2012 8:08:34 AM oracle.install.commons.util.ExitStatusSet add
INFO: Adding ExitStatus FAILURE to the exit status set



Solution

GI update from 11.2.0.3.x to 11.2.0.4.x

1. Execute on both nodes the following commands (change nodename1,nodename2 with your ODA host nodes name):

export OLD_HOME=/u01/app/11.2.0.4/grid
export ORACLE_HOME=$OLD_HOME
$OLD_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_HOME
.
export OLD_HOME=/u01/app/11.2.0.3/grid
export ORACLE_HOME=$OLD_HOME
$OLD_HOME/oui/bin/runInstaller -attachHome -silent -local ORACLE_HOME=$OLD_HOME ORACLE_HOME_NAME=OraGrid11gR3 "CLUSTER_NODES=nodename1,nodename2" CRS=true

instead the above commands you can also

1. Edt the central inventory: "/u01/app/oraInventory/ContentsXML/inventory.xml" on  both nodes

It had the following entries:

1) <HOME NAME="OraGrid11gR204" LOC="/u01/app/11.2.0.4/grid" TYPE="O" IDX="3" CRS="true">
2)   <NODE_LIST>
3)      <NODE NAME="nodename1"/>
4)      <NODE NAME="nodename2"/>
5)   </NODE_LIST>
6) </HOME>
7) <HOME NAME="OraGrid11gR3" LOC="/u01/app/11.2.0.3/grid" TYPE="O" IDX="1" REMOVED="T"/>
(...)

i)   I have removed the entry for GI old home -> that is above line 7.
ii)  Replaced the line1 with:
<HOME NAME="OraGrid11gR3" LOC="/u01/app/11.2.0.3/grid" TYPE="O" IDX="1" CRS="true">

So the contents of the above file will look like:

1) <HOME NAME="OraGrid11gR3" LOC="/u01/app/11.2.0.3/grid" TYPE="O" IDX="1" CRS="true">
2)   <NODE_LIST>
3)      <NODE NAME="nodename1"/>
4)      <NODE NAME="nodename2"/>
5)   </NODE_LIST>
6) </HOME>
(...)

2. Remove the GI 11.2.0.4 home on both nodes:

rm -fr /u01/app/11.2.0.4
 

3. Check on both nodes the oak version details:

# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version        
--------------  ---------------           ------------------        -----------------        
2.2.0.0.0                                                                                    
                Controller                05.00.29.00               Up-to-date               
                Expander                  0342                      Up-to-date               
                SSD_SHARED                E125                      Up-to-date               
                HDD_LOCAL                 SA03                      Up-to-date               
                HDD_SHARED                0A25                      Up-to-date               
                ILOM                      3.0.14.13.a r70764        Up-to-date               
                BIOS                      12010304                  Up-to-date               
                IPMI                      1.8.10.4                  Up-to-date               
                HMP                       2.2.0                     Up-to-date               
                OAK                       2.2.0.0.0                 Up-to-date               
                OEL                       5.8                       Up-to-date               
                GI_HOME                   11.2.0.3.2                11.2.0.4.0                   
                DB_HOME                   11.2.0.2.5(13343424,      No-update                
                                          13343447)                                          
                ASR                       Unknown                   3.7         

 

4. Setup your SCAN listener with one single port executing the following command:

$ /u01/app/11.2.0.3/grid/bin/srvctl modify scan_listener -p "TCP:1521"
$ /u01/app/11.2.0.3/grid/bin/srvctl config scan_listener -i 1
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

5. Verify that Oracle Grid Infrastructure (GI) stack is up and running on both the nodes.

# /u01/app/11.2.0.3/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

If the GI stack is not up, execute the following command as user root to check and wait for the stack to start up by running the above command again.

# /u01/app/11.2.0.3/grid/bin/crsctl start crs
or
# /u01/app/11.2.0.3/grid/bin/crsctl start cluster (Most likely on 2nd node, you will execute this command)

6. Run again the ODA GI upgrade as usual

# cd /opt/oracle/oak/bin
# ./oakcli update -patch <new version> --gi

7. Add the second port to SCAN listener again:

$/u01/app/11.2.0.4/grid/bin/srvctl modify scan_listener -p TCP:1521/TCP:1522
$ /u01/app/11.2.0.4/grid/bin/srvctl config scan_listener -i 1
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521,1522

 

GI update from 11.2.0.2.x to 11.2.0.3.x

1. Execute on both nodes the following commands (change nodename1,nodename2 with your ODA host nodes name):

export OLD_HOME=/u01/app/11.2.0.3/grid
export ORACLE_HOME=$OLD_HOME
$OLD_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_HOME
.
export OLD_HOME=/u01/app/11.2.0/grid
export ORACLE_HOME=$OLD_HOME
$OLD_HOME/oui/bin/runInstaller -attachHome -silent -local ORACLE_HOME=$OLD_HOME ORACLE_HOME_NAME=OraGrid11gR2 "CLUSTER_NODES=nodename1,nodename2" CRS=true

instead the above commands you can also

1. Edt the central inventory: "/u01/app/oraInventory/ContentsXML/inventory.xml" on  both nodes

It had the following entries:

1) <HOME NAME="OraGrid11gR203" LOC="/u01/app/11.2.0.3/grid" TYPE="O" IDX="3" CRS="true">
2)   <NODE_LIST>
3)      <NODE NAME="nodename1"/>
4)      <NODE NAME="nodename2"/>
5)   </NODE_LIST>
6) </HOME>
7) <HOME NAME="OraGrid11gR2" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" REMOVED="T"/>
(...)

i)   I have removed the entry for GI old home -> that is above line 7.
ii)  Replaced the line1 with:
<HOME NAME="OraGrid11gR2" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">

So the contents of the above file will look like:

1) <HOME NAME="OraGrid11gR2" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
2)   <NODE_LIST>
3)      <NODE NAME="nodename1"/>
4)      <NODE NAME="nodename2"/>
5)   </NODE_LIST>
6) </HOME>
(...)

2. Remove the GI 11.2.0.3.2 home on both nodes:

rm -fr /u01/app/11.2.0.3

3. Check on both nodes the oak version details:

# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version        
--------------  ---------------           ------------------        -----------------        
2.2.0.0.0                                                                                    
                Controller                05.00.29.00               Up-to-date               
                Expander                  0342                      Up-to-date               
                SSD_SHARED                E125                      Up-to-date               
                HDD_LOCAL                 SA03                      Up-to-date               
                HDD_SHARED                0A25                      Up-to-date               
                ILOM                      3.0.14.13.a r70764        Up-to-date               
                BIOS                      12010304                  Up-to-date               
                IPMI                      1.8.10.4                  Up-to-date               
                HMP                       2.2.0                     Up-to-date               
                OAK                       2.2.0.0.0                 Up-to-date               
                OEL                       5.8                       Up-to-date               
                GI_HOME                   11.2.0.2.5(13343424,      11.2.0.3.2(13696216,     
                                          13343447)                 13696251)                
                DB_HOME                   11.2.0.2.5(13343424,      No-update                
                                          13343447)                                          
                ASR                       Unknown                   3.7         

 

4. Setup your SCAN listener with one single port executing the following command:

$ /u01/app/11.2.0/grid/bin/srvctl modify scan_listener -p "TCP:1521"
$ /u01/app/11.2.0/grid/bin/srvctl config scan_listener -i 1
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

5. Verify that Oracle Grid Infrastructure (GI) stack is up and running on both the nodes.

# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

If the GI stack is not up, execute the following command as user root to check and wait for the stack to start up by running the above command again.

# /u01/app/11.2.0/grid/bin/crsctl start crs
or
# /u01/app/11.2.0/grid/bin/crsctl start cluster (Most likely on 2nd node, you will execute this command)

6. Run again the ODA GI upgrade as usual

# cd /opt/oracle/oak/bin
# ./oakcli update -patch 2.2.0.0.0 --gi

7. You could add the second port to SCAN listener again using the old GI home as following:

$/u01/app/11.2.0/grid/bin/srvctl modify scan_listener -p TCP:1521/TCP:1522
$ /u01/app/11.2.0/grid/bin/srvctl config scan_listener -i 1
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521,1522

References

<BUG:14151562> - LNX64-112-CMT: NEED TO CLEANUP THE INVENTORY ENTRIES IF GI/DB UPGRADE IS FAILED.
<BUG:13798847> - ADD MULTILPLE PORTS TO SCAN_LISTENER FAILS

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback