Home » Fusion Middleware & Colab Suite » Weblogic & Application Server » Problem starting up infrastructure DB for OAS.(merged)
Problem starting up infrastructure DB for OAS.(merged) [message #411046] Wed, 01 July 2009 12:36 Go to next message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Hello everyone!

I'm not really a DBA but fate wanted me to clean up and repair some broken OAS at the place I work. Unfortunately the "professional" DBA left some time ago and our testing servers are not working correctly.

I'm a complete noob when it comes to this heavy-metal systems so I'm in badly need of some help.

The thing is, when I go to $ORACLE_HOME/opmn/bin and run "opmnctl status" it says that it's unable to connect, and maybe Opmn is down. Ok, then I try doing "opmn startall" and it spits out this error:

opmnctl startall
opmnctl: starting opmn and all managed processes...
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

opmnctl: opmn start failed


So, I go check the log files, right?. Here's what I got in $ORACLE_HOME/opmn/logs:

$ ls
bak       ipm.log   ons.log   opmn.log  states

$ more ipm.log
09/07/01 11:52:05 [4] Create pm state directory: /u01/app/oraias/oraInfra/opmn/logs
/states
09/07/01 11:52:05 [4] PM state file does not exist: /u01/app/oraias/oraInfra/opmn/l
ogs/states/.opmndat
09/07/01 11:52:05 [4] OPMN server ready.  Request handling enabled.

$ more ons.log
09/07/01 11:52:05 [4] ONS server initiated
09/07/01 11:52:05 [2] BIND (Address already in use)
09/07/01 11:52:05 [2] XXX.XXX.XXX.130:406323200 - listener BIND failed
09/07/01 11:52:05 [2] BIND (Address already in use)
09/07/01 11:52:05 [2] 127.0.0.1:399835136 - listener BIND failed
09/07/01 11:52:05 [4] Listener thread 10: 127.0.0.1:399835136 (0x442) terminating
09/07/01 11:52:05 [1] Local listener terminated
09/07/01 11:52:05 [4] Listener thread 11: XXX.XXX.XXX.130:406323200 (0x401) terminating


And that's it. "opmn.log" is empty as is the "states" directory.

I see that it says the address is already in use, so I do a lsnrctl status and I get:
$ lsnrctl status

LSNRCTL for Solaris: Version 10.1.0.4.2 - Production on 01-JUL-2009 12:04:10

Copyright (c) 1991, 2004, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=host-name.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Solaris: Version 10.1.0.4.2 - Production
Start Date                29-JUN-2009 13:37:11
Uptime                    1 days 22 hr. 26 min. 59 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Log File         /u01/app/oraias/oraBiaf/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=host-name.com)(PORT=1521)))
Services Summary...
Service "OEXTEST" has 1 instance(s).
  Instance "OEXTEST", status READY, has 1 handler(s) for this service...
Service "iasdb.host-name.com" has 1 instance(s).
  Instance "iasdb", status READY, has 3 handler(s) for this service...
The command completed successfully


As I understand it, there are 2 Oracle DBs, one for the infrastructure and one for the applications themselves. Right now, "oraInfra" is the instance for the infrastructure (the one I think is causing problems), but the Listener lists "oraBiaf" even when my environment variables are all pointing to the "oraInfra" instance, like this:

$ env | sort
_=/usr/bin/env
_INIT_NET_STRATEGY=none
_INIT_PREV_LEVEL=S
_INIT_RUN_LEVEL=3
_INIT_RUN_NPREV=0
_INIT_UTS_ISA=sparc
_INIT_UTS_MACHINE=sun4u
_INIT_UTS_NODENAME=crm-rp
_INIT_UTS_PLATFORM=SUNW,Sun-Fire-V490
_INIT_UTS_RELEASE=5.8
_INIT_UTS_SYSNAME=SunOS
_INIT_UTS_VERSION=Generic_108528-29
DISPLAY=appsprod:11.0
HOME=/export/home/oraias
HZ=100
LC_COLLATE=en_US.ISO8859-15
LC_CTYPE=en_US.ISO8859-1
LC_MESSAGES=en_US.ISO8859-1
LC_MONETARY=en_US.ISO8859-15
LC_NUMERIC=en_US.ISO8859-15
LC_TIME=en_US.ISO8859-15
LOGNAME=oraias
MAIL=/usr/mail/oraias
ORACLE_HOME=/u01/app/oraias/oraInfra
ORACLE_SID=iasdb
PATH=/usr/j2se/bin:/usr/bin::/usr/sbin:/usr/openwin/bin:/usr/sfw/bin:/u01/app/oraias/oraInfra/bin
PS1=[CRM-RP @ $PWD]$ 
PWD=/u01/app/oraias/oraInfra/ldap/bin
SHELL=/usr/bin/ksh
TERM=dtterm
TZ=Mexico/General


If I run ldapcheck I get:

$ ldapcheck
 
Checking Oracle Internet Directory Processes ...ALL
 
Process oidmon is Alive as PID 14070 
Not Running ---- Process oidldapd
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 24746 


So I believe the ldapd is causing trouble as it's not running...but I can't get it to start.

If I do:

$ oidctl connect=iasdb server=oidldapd instance=1 start


It does not send any errors. It just executes, however, after I do an ldapcheck again I get:

$ ldapcheck
 
Checking Oracle Internet Directory Processes ...ALL
 
Process oidmon is Alive as PID 14070 
Not Running ---- Process oidldapd
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 24746 


So, where could the problem be? I've been trying to learn and understand as much as I can about this but still, I'm obviously no expert and the configurarion was really messy from the start up...and you know it's always hard to fix things you didn't break in the first place :/.

So, I can get to the Oracle Enterprise Manager and it says my infrastructure DB is not running...and that's the overall objective...to be able to start it up...


What other info should I post to help you out understand what's happening?


Maybe I'm asking for too much, but any ideas would be greately appreciated, as right now I'm just poking things here and there to see what happens...and maybe it's just a little obvious-to-a-dba kind of thing that I'm just not seeing. Sad

PS. Also, I'm sorry for masking the IP addresses, but since it's a machine from work, I'm not sure my boss would be very happy if I boradcasted the address all over the Internet, right? Smile

Thanks!
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411050 is a reply to message #411046] Wed, 01 July 2009 13:08 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
We can start with one problem at a time Smile.
Check metalink note 444411.1
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411053 is a reply to message #411046] Wed, 01 July 2009 13:36 Go to previous messageGo to next message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Ahhh....ok..let me ask my boss for the CSI number or the Metalink account info and I'll get back to you.

Thanks!
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411076 is a reply to message #411046] Wed, 01 July 2009 17:15 Go to previous messageGo to next message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Ok....got my Metalink account. I read the note 444411.1.

However, I think the problem might be worse than I thought..

I'm in the *first* step : "Clear Down of Log Files and Increasing Log Levels", and it says I have to stop all OAS processes first, and it refers me to the documentation.

The docs say to do the following:

"emctl stop iasconsole" - done
"opmnctl stopall" - done (although I had to do it as root for some reason...if I used the user configured to handle the OAS instance it said permission denied..)

Then Stop the OracleAS Metadata Repository instance as follows:

ORACLE_HOME/bin/sqlplus /nolog
SQL> connect SYS as SYSDBA
SQL> shutdown
SQL> quit


- done

"ORACLE_HOME/bin/lsnrctl stop" - done
"emctl stop dbconsole" - done

Also had to do

"/etc/init.d/init.cssd stop" since the process "ocssd.bin" was running and the docs state that I have to check with ps -ef if any processes are still running for the ias-owner user.

Ok. Now I'm in the part where I already cleared all logs and I'm ready to startup everything again, and that's where I'm stuck.

I do

"ORACLE_HOME/opmn/bin/opmnctl start" and everything is fine, then I do

"ORACLE_HOME/opmn/bin/opmnctl status -l"

and I get:

Processes in Instance: host-name.com
-------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component      | process-type       |     pid | status   |        uid |  memused |    uptime | ports
-------------------+--------------------+---------+----------+------------+----------+-----------+------
LogLoader          | logloaderd         |     N/A | Down     |        N/A |      N/A |       N/A | N/A
dcm-daemon         | dcm-daemon         |     N/A | Down     |        N/A |      N/A |       N/A | N/A
DSA                | DSA                |     N/A | Down     |        N/A |      N/A |       N/A | N/A
OC4J               | OC4J_SECURITY      |     N/A | Down     |        N/A |      N/A |       N/A | N/A
HTTP_Server        | HTTP_Server        |     N/A | Down     |        N/A |      N/A |       N/A | N/A
OID                | OID                |     N/A | Down     |        N/A |      N/A |       N/A | N/A


Then the docs state that I should start each component independently. I do component "OID" first, as i believe the other ones need it to be up first.

So I do

"opmnctl startproc ias-component=OID"

and after a *long* time I get:

opmnctl: starting opmn managed processes...
================================================================================
opmn id=crm-rp:6200
    0 of 1 processes started.

ias-instance id=instas1.host-name.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    OID/OID/OID

Error
--> Process (pid=12306)
    failed to start a managed process after the maximum retry limit
    Log:
    /u01/app/oraias/oraInfra/opmn/logs/OID~1


Then I go look at the log file and it only has this:

$ more /u01/app/oraias/oraInfra/opmn/logs/OID~1

--------
09/07/01 15:44:58 Stop process
--------


And that's where I stop because I have no idea what else to do. The note 444411.1 states that I should increase the log levels, but I need to use oidadmin to increase the OID component log level...and I can't do it because it's not running, so I can't connect to it :(.

Any idea how can I proceed?

Thanks!

Re: Problem starting up infrastructure DB for OAS.(merged) [message #411077 is a reply to message #411046] Wed, 01 July 2009 18:26 Go to previous messageGo to next message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Ok, I managed to increase the log level of the OPMN since I can't do it with for the OID and this is what I got, again, after several minutes of waiting (maybe 15 to 20):

$ opmnctl startproc ias-component=OID
opmnctl: starting opmn managed processes...
================================================================================
opmn id=crm-rp:6200
    0 of 1 processes started.

ias-instance id=instas1.host-name.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    OID/OID/OID

Error
--> Process (pid=0)
    database dependency failed
    iasdb
    failed to start a managed process because a dependency check failed
    Log:
    none



What does "database dependency failed" means?

Thanks again for your time on this very much likely noob issue :)
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411086 is a reply to message #411077] Wed, 01 July 2009 19:19 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
Seems you are following note# 744433.1.
Note# 444411.1 will address the specific issue reported in ons.log
https://metalink2.oracle.com/metalink/plsql/f?p=130:3:2052172406991419814::::p3_database_id,p3_docid,p3_show_header,p3_show_help,p3_b lack_frame,p3_font:NOT,444411.1,1,1,1,helvetica
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411089 is a reply to message #411046] Wed, 01 July 2009 19:40 Go to previous messageGo to next message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Wow....I'm very sorry fot that mistake...I did a search on "note 444411.1" and that's the document I got...apparently I'm also noob enough on this Metalink thing to not be able to check for the "Doc ID" entry... :S

I'll try what this document says tomorrow morning...Thanks again Mahesh....I'm impressed that you either know by heart every Doc ID there is...or I'm not the first unexperienced guy to ask this...

Thanks again! I relaly appreciate your help.

:)
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411235 is a reply to message #411046] Thu, 02 July 2009 11:28 Go to previous messageGo to next message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Ok. I followed the instructions on Doc 444411.1 and now I don't know what happened. I'm still getting an error. If I clear all the files, and then do "opmnctl start" all the log files I get are an empty "opmn.log" one, "ipm.log" with:
$ more ipm.log
09/07/02 11:05:03 [4] PM state directory exists: /u01/app/oraias/oraInfra/opmn/logs/states
09/07/02 11:05:03 [4] OPMN server ready.  Request handling enabled.

and finally "ons.log" with:
$ more ons.log
09/07/02 11:05:03 [5] Config: ORACLE_HOME: /u01/app/oraias/oraInfra
09/07/02 11:05:03 [5] Config: ORACLE_CONFIG_HOME: /u01/app/oraias/oraInfra
09/07/02 11:05:03 [5] Config: Instance Name: instas1.host-name.com
09/07/02 11:05:03 [5] Config: Instance Id: XXX.XXX.XXX.130.1b9240e.109986db3e8.-8000
09/07/02 11:05:03 [5] Config: Cluster Name: C_XXX.XXX.XXX.130.1b9240e.109986db3e8.-8000
09/07/02 11:05:03 [5] Config: Cluster Id: 1004
09/07/02 11:05:03 [5] Config: ONS node: crm-rp.host-name.com:6301
09/07/02 11:05:03 [6] Passive connection init 5 0,0.0.0.0,0
09/07/02 11:05:03 [7] Connection 2,0.0.0.0,0 connectionGetMsg
09/07/02 11:05:03 [4] ONS server initiated
09/07/02 11:05:03 [6] Worker Thread 6 started
09/07/02 11:05:03 [6] Worker Thread 6 checking for idle condition: 0 == 0?
09/07/02 11:05:03 [5] Starting listener thread: 127.0.0.1:6302 (0x62)
09/07/02 11:05:03 [6] Worker Thread 7 started
09/07/02 11:05:03 [6] Worker Thread 7 checking for idle condition: 0 == 0?
09/07/02 11:05:03 [6] Worker Thread 8 started
09/07/02 11:05:03 [6] Worker Thread 8 checking for idle condition: 0 == 0?
09/07/02 11:05:03 [5] Starting listener thread: XXX.XXX.XXX.130:6300 (0x21)
09/07/02 11:05:03 [5] Starting listener thread: XXX.XXX.XXX.130:6003 (0x24)
09/07/02 11:05:03 [2] BIND (Address already in use)
09/07/02 11:05:03 [2] 127.0.0.1:413007872 - listener BIND failed
09/07/02 11:05:03 [2] BIND (Address already in use)
09/07/02 11:05:03 [2] XXX.XXX.XXX.130:412876800 - listener BIND failed
09/07/02 11:05:03 [4] Listener thread 11: XXX.XXX.XXX.130:412876800 (0x401) terminating
09/07/02 11:05:03 [1] Local listener terminated
09/07/02 11:05:03 [4] Listener thread 10: 127.0.0.1:413007872 (0x442) terminating
09/07/02 11:05:03 [1] Local listener terminated


What I understand is that, for some weird reason, it's trying to open a listener on port "412876800", which I think must be an error. As far as I can tell, ports only go up to 65536, don't they?

Any ideas? What else can I do to narrow down the possible error/fix?

Thanks again!
Re: Problem starting up infrastructure DB for OAS.(merged) [message #411254 is a reply to message #411046] Thu, 02 July 2009 17:23 Go to previous message
acapulco
Messages: 7
Registered: July 2009
Junior Member
Now I reset the port numbers to the previous values, and after some time of testing different things (bring down the listener again, bringing it up, change the ports again, etc) I get this log file:
$ more OIDCTL.log

--------
09/07/02 16:51:13 Start process
--------

--------
09/07/02 16:51:13 Start process
--------
[gsdsiConnect] ORA-12514, ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Could not connect to the Database.


I've searched in $ORACLE_HOME/opmn/conf/ for a file that has a connect descriptor to a database (any database) to see if it's correct or not and I can't find nothing to tell the OPMN which database to connect. Where can I look this up?
Previous Topic: Server Failure. Cannot run application
Next Topic: adcfgclone in appstier failed with error rc:50014
Goto Forum:
  


Current Time: Thu Mar 28 05:33:57 CDT 2024