Exadata Database Machine V2
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

In this Document

Goal

Solution

Applies to:

Goal

 There is an increase number of cases related to failures of dbnodeupdate.sh and all related to similar characteristics.
This document contains some details and hints to diagnose the problem and hopefully lead to a faster resolution.

Solution

First, let's describe a simple workflow of dbnodeupdate.

1. When the current version of the Exadata Software image is 11.2.2.4.2 (this is the minimum version that dbnodeupdate.sh works with), there are two simple steps:

    * dbnodeupdate.sh will execute the bootstrap.sh phase 1 when executing dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip
    * After machine reboots, phase 2 is executed via ./dbnodeupdate.sh -u -p 2

2. When the current version of the Exadata image is 11.2.3.2.X, also two steps are executed:
    *  ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip
    * After machine reboots, dbnodeupdate.sh -c is required to complete the installation.

There are specific differences on the procedure based on the current Exadata Software image, but at some point, the following workflow will be the same when applying the new Exadata  Software image.

• Installation of the exadata-sun-computenode via yum -y install command:

Some important data reported in the log file:

As part of the installation of the exadata-sun-computenode* rpms, the log file will display the progress of differenct activities like:

• • Resolve Dependencies

• • Transaction summary and indication that rpms have been installed

   

  


    We can see that all the 387 files were installed, also the exadata-sun-computenode* was installed.

• • The exadata-sun-computenode have a preinstall/postinstall scriptlet.  They are triggered as soon as the exadata-sun-computenode* rpm are installed.  dbnodeupdate.sh will wait up to 900 seconds for completion.

• • Here is where update could be interrupted.   The expected outcome is that before 900 seconds, the post-install is complete and machine should reboot.  If machine is not rebooted  after 900 seconds, dbnodeupdate.log will report something like:

   

• • The post-install scriplet is a shell script running different things, but there is one important command associated to this situation:

NOTE:  Each version of the exadata-sun-computenode* will have similar calls but with different arguments.  Below is the case for 11.2.3.3.0:


    /sbin/service exachkcfg postrpm 11.2.3.3.0.131014.1 exadata-sun-computenode $EXADATA_DBNODEUPDATE_ACTIVE

In file /etc/init.d/exachfg, will execute

    nohup opt/oracle.cellos/exadata.computenode.post.sh $version $name "$yum_opts" $dbupdate_active

    Example for 11.2.3.3.0

    /opt/oracle.cellos/exadata.computenode.post.sh 11.2.3.3.0.131014.1 exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64

• • As mentioned before, in this stage is where normal process can be interrupted and a common cause is associated with the modification of the "profile" scripts (environment definition).

Known cases

1. Modification of /etc/profile.   Bug 18245341

    Following code was added to file /etc/profile

     if [[ ! -d $HOME ]]
     then
             echo "You are not authorized to access this server."
             logger -p user.err "Login denied by /etc/profile due to no home directory..."
             exit
     fi

    This interrupted the execution of the post-install scriptlet because during the execution, the /etc/profile is loaded but HOME variable is null.  Because of the referenced code, execution of the scriplet will exit
    and will not complete.

    Workaround

    1. Remove the entry from /etc/profile
    2. Adjust code to

     if [[ ! -d $HOME ]]
     then
         if [ "$EXADATA_DBNODEUPDATE_ACTIVE" != 'yes' ]
         then
             echo "You are not authorized to access this server."
             logger -p user.err "Login denied by /etc/profile due to no home directory..."
           exit
        fi
     fi


2. Modification of file /root/.bash_profile

    This case, file /root/.bash_profile was modified with a lot of code, but most important:

    # rootlog script
       DATE=`date +%m%d%y`
       TIME=`date +%T`
       script /var/log/rootlogs/root.$DATE.$TIME.$PPID.out
       exit 0



    The script /opt/oracle.cellos/exadata-computenode-post.sh, the first line is "#!/bin/bash -l" where the -l flag means: Make bash act as if it had been invoked as a login shell.
    The /root/.bash_profile will be executed and when the line "exit 0" will cause to terminate the execution of the session, not completing the post-install scriplet execution.

Workaround

• Remove line with exit 0 or better use a clean /root/.bash_profile (no modifications).

  

3. Another modification of file /root/.bash_profile

    This time, file /root/.bash_profile was modified with following line

    cd /tmp

    This cause the confusion of the post-install scriplet

    [1400677260][2014-05-21 15:41:34 +0200][INFO][/opt/oracle.SupportTools/dbnodeupdate_helpers/11.2.3.3.0/bootstrap.uln/dbnodeupdate.sh][PrintMsg][]    (ACTION:) Waiting for post rpm script to finish. Sleeping another 60 seconds (900 / 900)
    [1400677260][2014-05-21 15:42:34 +0200][WARNING][/opt/oracle.SupportTools/dbnodeupdate_helpers/11.2.3.3.0/bootstrap.uln/dbnodeupdate.sh][HandlePostScriptletOneTime][]  [FILE: /var/log/cellos/exadata.computenode.post.log]
        File is not an     absolute path or doesn't exist in current working directory /opt/oracle.SupportTools/dbnodeupdate_helpers/11.2.3.3.0/bootstrap.uln
    [1400677260][2014-05-21 15:42:34 +0200][ERROR][/opt/oracle.SupportTools/dbnodeupdate_helpers/11.2.3.3.0/bootstrap.uln/dbnodeupdate.sh][HandlePostScriptletOneTime][]  bootstrap.sh did not complete, exiting

Workaround

Remove customization to the file /root/.bash_profile.

Additional Notes

• • How to complete the execution of dbnodeupdate?

The clean method will be running dbnodeupdate.sh rollback after identifying and fixing the cause of the problem.  Of course this will take longer time.

There is another option but needs to be executed with caution and it may be different through releases.   Below is an example of 11.2.3.3.0 and basically is the execution of certain scripts.

Everything is based on the ID assigned for the last execution of dbnodeupdate.sh

i) create the following directories ( You can refer the dbnodeupdate.log of last try runid -

In this example :- dbnodeupdate.sh runid - 220514160310

# mkdir /var/www/html/yum/unknown/EXADATA/dbserver/220514160310
# mkdir /u01/app/oracle/stage.220514160310

ii) unzip the file for 11.2.3.3.0 (p17809253_112330_Linux-x86-64.zip )

# unzip /u01/software/patch/p17809253_112330_Linux-x86-64.zip /u01/app/oracle/stage.220514160310/.

iii) mount the ISO

# mount -o loop /u01/app/oracle/stage.220514160310/repoimage.iso /var/www/html/yum/unknown/EXADATA/dbserver/220514160310

iv) Run exadata.computenode.post.sh

For 112330 this results in execution of:
 .
# /opt/oracle.cellos/exadata.computenode.post.sh 11.2.3.3.0.131014.1 exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64

• • How to get the content of the scriplet

# rpm -q --scripts exadata-sun-computenode > postscrp.txt

This solution has no attachment