Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1955767.1
Update Date:2017-08-10
Keywords:

Solution Type  Problem Resolution Sure

Solution  1955767.1 :   Rebuild IXP Oracle Database  


Related Items
  • Oracle Communications Performance Intelligence Center (PIC) Software
  •  
Related Categories
  • PLA-Support>Sun Systems>CommsGBU>Global Signaling Solutions>SN-SND: Tekelec PIC
  •  




In this Document
Symptoms
Cause
Solution
References


Created from <SR 3-9824934291>

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 6.6.3 and later
Information in this document applies to any platform.

Symptoms

Oracle couldn't start on IXP and need to be reinstalled.

Cause

The IXP Database reinstallation may be needed after different issues:

  • data partition corrupted
  • inode table on data partition corrupted

Solution

Warnings:

  1. The followin procedure needs an Oracle administrator knowledge to be used.
  2. If there is a single XDR storage in the IXP subsystem, all IXP should be stopped before proceeding to the installation.
  3. If the XDr storage is within a pool with other XDR storage, the XDR storage that will be rebuild should be set in "MAINTENANCE" mode in the CCM

 

StepActionCommandsComments
1 Prevent IxpStore to write in Database If only one xDR server in the pool: As root, service TKLCixp stop on 1a, 1b, 1c, etc

If more than one xDR server in the pool: set corresponding storage to MAINTENANCE state

 
2 Stop oracle database If not ASM:
As root,
service TKLCoracledb stop

If ASM:
As oracle:

sqlplus / as sysdba
shutdown immediate
 
3 Cleanup files (it may take long time to delete the files, be patient) If not ASM:
As root, execute rm -f * in /opt/oracle/ctrl1/IXP, /opt/oracle/ctrl2/IXP, /opt/oracle/oraindex/IXP, /opt/oracle/oradata/IXP and /opt/oracle/redo/IXP

If ASM:
As Linux grid user:

[grid@ixp0101-1z ~]$ asmcmd
ASMCMD> cd DATA
ASMCMD> ls
ASM/
IXP/
ASMCMD> rm –rf IXP
ASMCMD> ls
ASM/
ASMCMD>
as root:
cp /opt/TKLCixp/prod/db/instance/dat/initIXP.ora /opt/TKLCixp/prod/db/instance/dat/initIXP.bak
vi /opt/TKLCixp/prod/db/instance/dat/initIXP.ora
delete the two last lines(example)
*.control_files='+DATA/IXP/CONTROLFILE/Current.256.786535449','+DATA/IXP/CONTROLFILE/Current.257.786535449'
*.SPFILE='+DATA/IXP/PARAMETERFILE/spfile.278.786536557'
 
4 Instance creation script

As oracle, in /opt/TKLCixp/prod/db/instance/cmd, execute ./createIXPfull.sh

Then if ASM all the following:

As grid get controle file name in asmcmd.

ASMCMD> cd DATA/IXP/CONTROLFILE/
ASMCMD> ls
Current.538.943590183
Current.539.943590183

As root update InintIXP.ora with what seen before in asmcmd

vi /opt/TKLCixp/prod/db/instance/dat/initIXP.ora
add at end the right file descriptiopn:
*.control_files='+DATA/IXP/CONTROLFILE/Current.538.943590183','+DATA/IXP/CONTROLFILE/Current.539.943590183'

 

As oracle create spfile

  

[oracle@ixp0010-1a dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Wed May 10 05:54:30 2017

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> create spfile='+DATA' from pfile='/opt/TKLCixp/prod/db/instance/dat/initIXP.ora';

File created.

  

As oracle modify initIXP.ora with what read in asmcmd DATA/IXP/PARAMETERFILE

  

example

as grid

ASMCMD> cd +DATA/IXP/PARAMETERFILE/
ASMCMD> ls
spfile.536.943592903

as Oracle:

vi /opt/oracle11/oracle/product/11.2.0/dbhome_1/dbs/initIXP.ora
SPFILE='+DATA/ixp/parameterfile/spfile.536.943592903'

  

 

 

Duration: Approximately 25 minutes

Warning60.gif if error during redo creation edit ../sql/createIXP_1.sql and remove the creation of last redo (redo23) don't forget "," line before create the redo afterward manually. ALTER DATABASE ADD LOGFILE GROUP 23 ( '/opt/oracle/redo/IXP/redo23.log') SIZE 10240M REUSE;

5 Restart listener As oracle if Not ASM:
lsnrctl stop
lsnrctl start
lsnrctl status
 
Repeat the command until the IXP services are present (may take more than 5 minutes)
6 Create schema IAS4:

As oracle:

/opt/TKLCixp/prod/db/schema/cmd execute ./ReinitDTO_Ee.sh IXP/<password>@<SID> DATA_CONF DATA_LOG

IAS6 or IAS7 or higher:
As oracle:

/opt/TKLCixp/prod/db/schema/cmd execute ./ReinitDTO_Ee.sh IXP/<password>@localhost/<SID> DATA_CONF DATA_LOG
additional steps for IAS 9 and up, verify that Dto package is up to date, if not upgrade it.
chapter 7.3 from http://docs.oracle.com/cd/E52616_01/docs.90/PIC_Major_Upgrade_9.0.pdf
For PIC 10.2, refer to 10.2 Upgrade Guide Chapter 7.8 Upgrade DTO Packag: http://docs.oracle.com/cd/E60682_01/docs.102/E60676_rev_3.pdf
 
7 Add data files and index files (in //) IAS4:
As oracle in /opt/TKLCixp/prod/db/utils/cmd, execute:
nohup ./AddDatafiles.sh SYSTEM/<password>@<SID> -c 70 &
nohup ./AddIndexfiles.sh SYSTEM/<password>@<SID> -c 40 &

IAS6:
As oracle in /opt/TKLCixp/prod/db/utils/cmd, execute:

nohup ./AddDatafiles.sh -c 70 &
nohup ./AddIndexfiles.sh -c 60 &

IAS7 or higher:

As root run:
cd_oracle_utils
nohup ./oracle-postinstall.pl -auto | tee -a /tmp/oracle-postinstall.txt &
For 6.0 or 6.1 use following command:

$ ./AddIndexfiles.sh SYSTEM/MANAGER@localhost/IXP -l Recommended allocation for index table space is 70% of available space. Standard allocation for partition of size 1.4TB is 1TB (490GB for 146GB disks). Each allocation unit is single file of 16GB, meaning that at the end there should be 64 (30 files for 146GB disks) files allocated for data table space. For allocating 70% of standard 1.4TB partition use following command: For 6.6 use following command: $ AddIndexfiles.sh –c 63

 

If following error during oracle-postinstall.pl:

[root@ixp0010-1b cmd]# err | Simple DB checks
err | WARNING: Database state may need further attention.
err | WARNING: Recover files found: 0
err | WARNING: Objects in state other then valid: 1
err | Database check did not finish cleanly
war | There were ERRORS. 1

Recompiling All Invalid Objects:

use correct $Oracle_home, in the folowing exemple its /opt/oracle11/oracle/product/11.2.0/dbhome_1

as oracle

sqlplus / AS SYSDBA
SQL> STARTUP
SQL> SPOOL utlrp.out
SQL> @/opt/oracle11/oracle/product/11.2.0/dbhome_1/rdbms/admin/utlrp.sql
SQL> SPOOL OFF
SQL> select count(*) from dba_objects where not status = 'VALID';
SQL> EXIT

 

 

8 Wait an verify progress
Time needed : 5 minutes per data file (may take 5 to 6 hours for all files to be created)
non IAS7 :
watch “ls -l /opt/oracle/oradata/IXP”

IAS7 or higher :
look on putty output or in /tmp/oracle-postinstall.txt

 
9 Restart IXP on 1c service TKLCixp start  
10 Restart IXP on 1b service TKLCixp start  
11 Restart IXP on 1a service TKLCixp start  

Additional needs :

  • as oracle cd /usr/TKLC/TKLCixp/prod/db/tuning/cmd/
  • ./ManageSpaceAdvisor.sh sys/<password>@localhost/<SID> -d
  • ./SystemStats.sh sys/<password>@localhost/<SID> -i
  • ./TuningPackage.sh ixp/<password>@localhost/<SID> -i
  • ./TuningPackage.sh ixp/<password>@localhost/<SID> -e IXP_NIGHTLY_JOB
  • ./ModifyMaintenanceWindow.sh SYS/<password>@localhost/<SID> 2 4
  • for IAS 7 or higher KM Doc 1941703.1
  • (if IAS 7.1) TN003439 (KM Doc needs to be clarified)

References

<NOTE:1941703.1> - How to Apply Very Large Memory (VLM) Workaround to Resolve Oracle Database Shared Memory (ORA-04031)

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback