Recentemente, e olha que ele já existe há um bom tempo, desde que foi lançado o 11gR2, eu estava dando uma olhada na documentação do Oracle, porque queria fazer algumas automações de startup e shutdown. Porque nunca pensei em usar o Oracle RESTART para isso? Usando uma base de dados em cima do ASM você precisa instalar o Grid infrastructure e nessa instalação está o Oracle RESTART. Daí fui ler a documentação do Oracle RESTART.
O Oracle RESTART foi desenhado para aumentar a disponibilidade do seus databases, no geral ele consegue monitorar recursos como:
• Database instances
• Listeners
• Database Services
• ASM Instances
• Diskgroups
• Oracle Notification Services (ONS)
Além de simplificar o gerenciamento dos recursos, ele consegue reinicializar componentes após falhas, além disso, é integrado com sqlplus, asmcmd e outras ferramentas Oracle. Isso significa que se você baixar um banco de dados conectado pelo sqlplus, ele consegue entender esse evento e não vai reinicializar o banco novamente. Porém se algo anormal ocorrer e uma instance cair por que um administrador de sistemas ou DBA acidentalmente deu kill -9 em um processo de background vital, ele automaticamente reinicializa a instance afetada. Uma outra coisa muito legal é que se tem diversos databases em uma mesma máquina, poderá baixar e subir todos usando o Oracle RESTART e com isso simplificar muito o gerenciamento desses bancos. Atualmente, não é incomum ter duas, três, quatro instances rodando em uma única máquina e o gerenciamento dessas dependencias, principalmente com discos ASM e outros, dá um certo trabalhinho para gerenciar o startup e shutdown e um outro benefício é a familiaridade com comandos para gerenciamento de ambientes Oracle RAC 11g em diante.
A primeira coisa que devemos fazer, depois de instalar o Oracle Grid Infrastructure é ajustar um .bash_profile para ele ou ajustar o .bash_profile do usuário do Oracle, em caso de usuários separados (role separation, grid e oracle) ajuste os .bash_profile. Particularmente, eu gosto de usar .bash_profile e .bash_grid e configuro um alias (isso eu aprendi com o nosso amigo Tim Hall do Oracle-Base). Depois de criado os arquivos com as variáveis de ambiente, basta colocar no final dos dois arquivos as linhas abaixo:
alias grid_env='. /home/oracle/.bash_grid' alias db_env='. /home/oracle/.bash_profile'
Se for usar apenas um arquivo de variáveis (.bash_profile), basta incluir a váriável GRID_HOME e apontar para o caminho de instalação onde foi feita a instalação do Oracle Grid Infrastructure. Como esse é um exemplo, em uma máquina virtual de testes, vou usar apenas .bash_profile que contém as linhas abaixo, atenção as linhas em negrito, elas irão permitir usar os binários do Grid Infra:
export PATH=$PATH:$HOME/bin # Oracle Settings export TMP=/tmp export TMPDIR=$TMP export ORACLE_HOSTNAME=ora112.mufalani.com export ORACLE_UNQNAME=ora112 export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1 export ORACLE_SID=orcl export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib export GRID_HOME=/u01/app/11.2.0/grid export PATH=$PATH:$GRID_HOME/bin
Depois da instalação do Grid Infrastructure, vamos olhar o funcionamento do RESTART na prática. Para quem já tem familiaridade com o Oracle RAC desde a versão 10g, com certeza conhece o comando crs_stat –t, ele mostra os recursos geridos pelo Oracle clusterware (até poderíamos chamar o Oracle RESTART de um clusterware reduzido para um single instance, não me crucifiquem, claro que o clusterware é bem mais robusto e tem mais componentes). Vale ressaltar que ele não existe mais no Oracle 12c.
[oracle@ora112 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....ER.lsnr ora....er.type ONLINE ONLINE ora112 ora.cssd ora.cssd.type OFFLINE OFFLINE ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE ora112 ora.ons ora.ons.type OFFLINE OFFLINE ora.orcl.db ora....se.type ONLINE ONLINE ora112
Como podemos ver aí em cima, o banco de dados orcl está sendo monitorado através do Oracle RESTART, também estou monitorando o listener que tem o nome LISTENER. Ao criarmos um database usando o DBCA ou um listener usando o NETCA eles já automaticamente são monitorados pelo Oracle RESTART se o Grid infra estiver instalado. Do mesmo jeito que se criar serviços usando dbms_service, porém se criarmos um serviço apenas ajustando o parametro service_names ele não vai ser automaticamente monitorado e a Oracle nem recomenda fazer isso.
Vamos nos familiarizar com os comandos, vamos remover o banco orcl do monitoramento do Oracle RESTART e para isso, usamos o comando abaixo:
[oracle@ora112 ~]$ srvctl remove database -d orcl PRKO-3141 : Database orcl could not be removed because it was running
Note que o Oracle RESTART não deixou remover o banco porque ele está rodando, vamos parar o banco de dados então antes de remover o mesmo.
[oracle@ora112 ~]$ srvctl stop database -d orcl
Obs.: Quando a operação é realizada com sucesso o comando não retorna nada. Na minha humilde opinião, eu acho que deveria retornar pelo menos um OK.
Vamos remover o banco de dados novamente, ele irá perguntar se quer realmente remover o banco do monitoramento. É importante reforçar que o banco de dados não foi removido, apenas o monitoramento dele.
[oracle@ora112 ~]$ srvctl remove database -d orcl Remove the database orcl? (y/[n]) y
Agora veremos como ficou o nosso monitoramento dos recursos do Oracle RESTART:
[oracle@ora112 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....ER.lsnr ora....er.type ONLINE ONLINE ora112 ora.cssd ora.cssd.type OFFLINE OFFLINE ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE ora112 ora.ons ora.ons.type OFFLINE OFFLINE
Não temos mais o database orcl monitorado pelo Oracle RESTART.
[oracle@ora112 ~]$ ls -ltrh /u01/app/oracle/oradata/orcl/
total 2.1G
-rw-r----- 1 oracle oinstall 101M Nov 17 20:28 redo02b.log
-rw-r----- 1 oracle oinstall 101M Nov 17 20:28 redo02a.log
-rw-r----- 1 oracle oinstall 101M Nov 17 20:28 redo01b.log
-rw-r----- 1 oracle oinstall 101M Nov 17 20:28 redo01a.log
-rw-r----- 1 oracle oinstall 21M Nov 17 20:28 temp01.dbf
-rw-r----- 1 oracle oinstall 501M Nov 17 21:08 users01.dbf
-rw-r----- 1 oracle oinstall 201M Nov 17 21:08 undotbs01.dbf
-rw-r----- 1 oracle oinstall 326M Nov 17 21:08 system01.dbf
-rw-r----- 1 oracle oinstall 326M Nov 17 21:08 sysaux01.dbf
-rw-r----- 1 oracle oinstall 101M Nov 17 21:08 redo03b.log
-rw-r----- 1 oracle oinstall 101M Nov 17 21:08 redo03a.log
-rw-r----- 1 oracle oinstall 9.1M Nov 17 21:08 control02.ctl' -- Invalid DBA, criando controlfile com aspas (')... Rsrsrsrs
-rw-r----- 1 oracle oinstall 9.1M Nov 17 21:08 control01.ctl
Só para termos certeza que o banco de dados existe, eu vou subir ele manualmente o banco de dados orcl, pelo sqlplus, para vermos que apenas o monitoramento não existe mais.
[oracle@ora112 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Fri Nov 17 21:19:30 2017 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 1068937216 bytes Fixed Size 2235208 bytes Variable Size 620758200 bytes Database Buffers 440401920 bytes Redo Buffers 5541888 bytes Database mounted. Database opened. SQL>
Agora vamos adicionar novamente o banco orcl para garantir que o Oracle RESTART irá monitorá-lo.
[oracle@ora112 ~]$ srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/db_1
Note que embora o serviço esteja monitorado, o target está offline e state também offline
[oracle@ora112 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....ER.lsnr ora....er.type ONLINE ONLINE ora112 ora.cssd ora.cssd.type OFFLINE OFFLINE ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE ora112 ora.ons ora.ons.type OFFLINE OFFLINE ora.orcl.db ora....se.type OFFLINE OFFLINE [oracle@ra112 ~]$ srvctl stop database -d orcl PRCC-1016 : orcl was already stopped [oracle@ora112 ~]$ srvctl start database -d orcl [oracle@ora112 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....ER.lsnr ora....er.type ONLINE ONLINE ora112 ora.cssd ora.cssd.type OFFLINE OFFLINE ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE ora112 ora.ons ora.ons.type OFFLINE OFFLINE ora.orcl.db ora....se.type ONLINE ONLINE ora112
Agora vamos fazer um teste, matando o processo do SMON, pelo S.O com um kill -9 PID
[oracle@ora112 ~]$ ps -ef | grep smon| grep -v grep oracle 3708 1 0 21:19 ? 00:00:00 ora_smon_orcl [oracle@ora112 ~]$ kill -9 3708 [oracle@ora112 ~]$ ps -ef | grep smon | grep -v grep oracle 4160 1 0 21:27 ? 00:00:00 ora_smon_orcl
Note que mesmo sem eu reiniciar o banco de dados o Oracle RESTART fez o seu trabalho e automaticamente inicializou o banco de dados.
O trecho no alert que mostra a instance sendo abortada e inicialida
Fri Nov 17 21:27:30 2017 PMON (ospid: 3684): terminating the instance due to error 474 Fri Nov 17 21:27:30 2017 ORA-1092 : opitsk aborting process Fri Nov 17 21:27:30 2017 License high water mark = 1 Instance terminated by PMON, pid = 3684 USER (ospid: 4083): terminating the instance Instance terminated by USER, pid = 4083 Fri Nov 17 21:27:31 2017
Vamos dar uma olhada no alert do cluster /u01/app/11.2.0/grid/log//alert_.log
2017-11-17 21:27:30.610 [/u01/app/11.2.0/grid/bin/oraagent.bin(2859)]CRS-5011:Check of resource "orcl" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/ora112/agent/ohasd/oraagent_oracle/oraagent_oracle.log" 2017-11-17 21:27:30.612 [ohasd(2521)]CRS-2765:Resource 'ora.orcl.db' has failed on server 'ora112'.
E também uma olhada mais no log que informado acima, do ohasd:
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] (:CLSN00007:)DbAgent::check failed gimh state 0
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] clsnDbAgent:checkCbk clsagfw_res_status 22
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] InstAgent::check db/asm 2clsagfw_res_status 5 poolState 2
2017-11-17 21:27:30.610: [ USRTHRD][1121712448] {0:0:142} Gimh::destructor gimh_dest_query_ctx rc=0
2017-11-17 21:27:30.610: [ USRTHRD][1121712448] {0:0:142} Gimh::destructor gimh_dest_inst_ctx rc=0
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] ConnectionPool::stopConnection
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:30.610: [ora.orcl.db][1121712448] {0:0:142} [check] InstAgent::check checkCounter 36 prev clsagfw_res_status 0 current clsagfw_res_status 5
2017-11-17 21:27:30.610: [ AGFW][1117509952] {0:0:142} ora.orcl.db 1 1 state changed from: ONLINE to: FAILED
2017-11-17 21:27:30.610: [ AGFW][1117509952] {0:0:142} ora.orcl.db 1 1 state details has changed from: Open to: Abnormal Termination
2017-11-17 21:27:30.610: [ AGFW][1117509952] {0:4:12} Generating new Tint for unplanned state change. Original Tint: {0:0:142}
2017-11-17 21:27:30.611: [ AGFW][1117509952] {0:4:12} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:8319
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] ClusterSubscriber::SubscriberWorker::InternalClusterSubscriber::handleEventCBexecuting for reason 1
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] event type is CRS_RESOURCE_STATE_CHANGE
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] bodylen = 735
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} -----------BodyBlock----------
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} CLS_TINT='{0:4:12}'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} COMPOSITE_RESOURCE_STATUS='LAST_OFFLINE'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} CURRENT_STATE='OFFLINE'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} DATABASE_TYPE='SINGLE'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} DB_UNIQUE_NAME='orcl'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} DEGREE='1'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} DERIVED_STATE='OFFLINE'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} ID='369500182'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} INSTANCE_NAME='orcl'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} INTERMEDIATE_INSTANCE_COUNT='0'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} NAME='ora.orcl.db'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} ONLINE_INSTANCE_COUNT='0'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} PREVIOUS_STATE='ONLINE'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} PREV_STATE_DETAILS='Open'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} REASON='FAILURE'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} RESOURCE_CLASS='database'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} RESOURCE_INCARNATION_NUMBER='1'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} RESOURCE_INSTANCE_ID='ora.orcl.db 1 1'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} RESOURCE_LOCATION='ora112'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} SEQUENCE_NUMBER='200020'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} STATE_DETAILS='Abnormal Termination'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} TIMESTAMP='2017-11-17 21:27:30'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} TOTAL_INSTANCE_COUNT='1'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} TYPE='ora.database.type'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} UNKNOWN_INSTANCE_COUNT='0'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} USER='SYSTEM'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} Version=''
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} CLUSTER_NAME='ora112'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} DB_UNIQUE_NAME='orcl'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} ORACLE_CLUSTERWARE.SUBCOMPONENT='CRSD'
2017-11-17 21:27:30.615: [ USRTHRD][1097230656] {0:4:12} RESOURCE_CLASS='database'
2017-11-17 21:27:30.615: [ AGFW][1117509952] {0:4:12} Agent received the message: RESOURCE_CLEAN[ora.orcl.db 1 1] ID 4100:672
2017-11-17 21:27:30.615: [ AGFW][1117509952] {0:4:12} Preparing CLEAN command for: ora.orcl.db 1 1
2017-11-17 21:27:30.615: [ AGFW][1117509952] {0:4:12} ora.orcl.db 1 1 state changed from: FAILED to: CLEANING
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] (:CLSN00106:) clsn_agent::clean {
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] clean {
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop_option stop mode immediate option 1
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop {
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop original reason system do shutdown abort
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::resetConnection sid orcl status 2
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] Gimh::check OH /u01/app/oracle/product/11.2.0/db_1 SID orcl
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] (:CLSN00007:)DbAgent::check failed gimh state 0
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] clsnDbAgent:checkCbk clsagfw_res_status 22
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] (:CLSN00007:)InstAgent::check: return unplanned offline
2017-11-17 21:27:30.617: [ USRTHRD][1121712448] {0:4:12} Gimh::destructor gimh_dest_query_ctx rc=0
2017-11-17 21:27:30.617: [ USRTHRD][1121712448] {0:4:12} Gimh::destructor gimh_dest_inst_ctx rc=0
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::stopConnection
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::check checkCounter 0 prev clsagfw_res_status 5 current clsagfw_res_status 1
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop shutdown mode: 4
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] DbAgent::preStopCbk {
2017-11-17 21:27:30.617: [ora.orcl.db][1121712448] {0:4:12} [clean] DbAgentEventModule::stop Forcing stop because of clean
2017-11-17 21:27:30.618: [ USRTHRD][1121712448] {0:4:12} Thread:DedicatedThreadstop {
2017-11-17 21:27:30.618: [ USRTHRD][1121712448] {0:4:12} Thread:DedicatedThreadstop }
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] DbAgent::preStopCbk }
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::getConnection 160 ERROR sid orcl pool status 1
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::getConnection 260 pConnxn 00000000
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop pool pConnxn 00000000
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/oracle/product/11.2.0/db_1 Crshome /u01/app/11.2.0/grid
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] clsnInstConnection::makeConnectStr LIBRARY_PATH1 ,LD_LIBRARY_PATH=
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/oracle/product/11.2.0/db_1/bin/oracle)(ARGV0=oracleorcl)(ENVS='ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1,ORACLE_SID=orcl,LD_LIBRARY_PATH=')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=orcl)))
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop non pool pConnxn 1 174d0100
2017-11-17 21:27:30.618: [ora.orcl.db][1121712448] {0:4:12} [clean] InstConnection::connectInt: server not attached
2017-11-17 21:27:30.631: [ USRTHRD][1123813696] {0:4:12} EVM EonsProxyEventForwarder::postMyEvent Event posted
2017-11-17 21:27:30.631: [ USRTHRD][1123813696] {0:4:12} EVM UsrcoEventForwarder::postMyEvent posting event "INSTANCE VERSION=1.0 service=orcl database=orcl instance=orcl host=ora112 status=down reason=FAILURE timestamp=2017-11-17 21:27:30 "
2017-11-17 21:27:30.631: [ USRTHRD][1123813696] {0:4:12} EVM UsrcoEventForwarder::postMyEvent posting event "DATABASE VERSION=1.0 service=orcl database=orcl instance= host=ora112 status=down reason=FAILURE timestamp=2017-11-17 21:27:30 "
2017-11-17 21:27:30.632: [ USRTHRD][1128016192] clsnUsrco: path=/u01/app/11.2.0/grid/racg/usrco/
2017-11-17 21:27:30.632: [ USRTHRD][1119611200] clsnUsrco: path=/u01/app/11.2.0/grid/racg/usrco/
2017-11-17 21:27:30.640: [ora.orcl.db][1121712448] {0:4:12} [clean] ORA-01092: ORACLE instance terminated. Disconnection forced
Process ID: 0
Session ID: 0 Serial number: 0
2017-11-17 21:27:30.640: [ora.orcl.db][1121712448] {0:4:12} [clean] InstConnection::connectInt (2) Exception OCIException
2017-11-17 21:27:30.640: [ora.orcl.db][1121712448] {0:4:12} [clean] InstConnection:connect:excp OCIException OCI error 1092
2017-11-17 21:27:30.640: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop: connect1 errcode 1092
2017-11-17 21:27:30.640: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop: connect2 oracleHome /u01/app/oracle/product/11.2.0/db_1 oracleSid orcl
2017-11-17 21:27:30.640: [ora.orcl.db][1121712448] {0:4:12} [clean] InstConnection::connectInt: server not attached
2017-11-17 21:27:30.655: [ora.orcl.db][1121712448] {0:4:12} [clean] InstConnection::shutdown mode 4
2017-11-17 21:27:31.684: [ora.orcl.db][1121712448] {0:4:12} [clean] DbAgent::stopCbk: {
2017-11-17 21:27:31.685: [ora.orcl.db][1121712448] {0:4:12} [clean] DbAgent::stopCbk {
2017-11-17 21:27:31.685: [ USRTHRD][1121712448] {0:4:12} DbAgentEventModule::stop Forcing stop because of clean
2017-11-17 21:27:31.685: [ USRTHRD][1121712448] {0:4:12} Thread:DedicatedThreadstop {
2017-11-17 21:27:31.685: [ USRTHRD][1121712448] {0:4:12} Thread:DedicatedThreadstop }
2017-11-17 21:27:31.685: [ora.orcl.db][1121712448] {0:4:12} [clean] DbAgent::stopCbk }
2017-11-17 21:27:31.685: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::releaseConnection InstConnection 174d0100
2017-11-17 21:27:31.685: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection connection count 1
2017-11-17 21:27:31.685: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection sid orcl, InstConnection 174d0100
2017-11-17 21:27:31.685: [ USRTHRD][1121712448] {0:4:12} InstConnection::breakCall pConnxn:174d0100 DetachLock:00a82418 m_pSvcH:173fff50
2017-11-17 21:27:31.685: [ USRTHRD][1121712448] {0:4:12} InstConnection:~InstConnection: this 174d0100
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection delete InstConnection 174d0100
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection freed 1
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::stopConnection
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop db/asm
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::stopConnection
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop Container->stop, isShutdown = 0
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] InstAgent::stop: }
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] clean }
2017-11-17 21:27:31.686: [ora.orcl.db][1121712448] {0:4:12} [clean] (:CLSN00106:) clsn_agent::clean }
2017-11-17 21:27:31.686: [ AGFW][1121712448] {0:4:12} Command: clean for resource: ora.orcl.db 1 1 completed with status: SUCCESS
2017-11-17 21:27:31.686: [ AGFW][1117509952] {0:4:12} Agent sending reply for: RESOURCE_CLEAN[ora.orcl.db 1 1] ID 4100:672
2017-11-17 21:27:31.687: [ora.orcl.db][1115408704] {0:4:12} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
2017-11-17 21:27:31.687: [ora.orcl.db][1115408704] {0:4:12} [check] ConnectionPool::resetConnection sid orcl status 2
2017-11-17 21:27:31.687: [ora.orcl.db][1115408704] {0:4:12} [check] Gimh::check OH /u01/app/oracle/product/11.2.0/db_1 SID orcl
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,1 exists
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] (:CLSN00007:)DbAgent::check failed gimh state 0
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] clsnDbAgent:checkCbk clsagfw_res_status 22
2017-11-17 21:27:31.688: [ USRTHRD][1115408704] {0:4:12} Thread:DedicatedThreadstop {
2017-11-17 21:27:31.688: [ USRTHRD][1115408704] {0:4:12} Thread:DedicatedThreadstop }
2017-11-17 21:27:31.688: [ USRTHRD][1115408704] {0:4:12} Gimh::destructor gimh_dest_query_ctx rc=0
2017-11-17 21:27:31.688: [ USRTHRD][1115408704] {0:4:12} Gimh::destructor gimh_dest_inst_ctx rc=0
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] ConnectionPool::stopConnection
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:31.688: [ora.orcl.db][1115408704] {0:4:12} [check] InstAgent::check checkCounter 0 prev clsagfw_res_status 2 current clsagfw_res_status 2
2017-11-17 21:27:31.688: [ AGFW][1117509952] {0:4:12} ora.orcl.db 1 1 state changed from: CLEANING to: PLANNED_OFFLINE
2017-11-17 21:27:31.688: [ AGFW][1117509952] {0:4:12} ora.orcl.db 1 1 state details has changed from: Abnormal Termination to: Instance Shutdown
2017-11-17 21:27:31.688: [ AGFW][1117509952] {0:4:12} Agent sending last reply for: RESOURCE_CLEAN[ora.orcl.db 1 1] ID 4100:672
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] ClusterSubscriber::SubscriberWorker::InternalClusterSubscriber::handleEventCBexecuting for reason 1
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] event type is CRS_RESOURCE_STATE_CHANGE
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] bodylen = 737
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} -----------BodyBlock----------
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} CLS_TINT='{0:4:12}'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} COMPOSITE_RESOURCE_STATUS=''
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} CURRENT_STATE='OFFLINE'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} DATABASE_TYPE='SINGLE'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} DB_UNIQUE_NAME='orcl'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} DEGREE='1'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} DERIVED_STATE='OFFLINE'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} ID='386281495'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} INSTANCE_NAME='orcl'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} INTERMEDIATE_INSTANCE_COUNT='0'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} NAME='ora.orcl.db'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} ONLINE_INSTANCE_COUNT='0'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} PREVIOUS_STATE='OFFLINE'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} PREV_STATE_DETAILS='Abnormal Termination'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} REASON='FAILURE'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} RESOURCE_CLASS='database'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} RESOURCE_INCARNATION_NUMBER='1'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} RESOURCE_INSTANCE_ID='ora.orcl.db 1 1'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} RESOURCE_LOCATION='ora112'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} SEQUENCE_NUMBER='200021'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} STATE_DETAILS='Instance Shutdown'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} TIMESTAMP='2017-11-17 21:27:31'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} TOTAL_INSTANCE_COUNT='1'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} TYPE='ora.database.type'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} UNKNOWN_INSTANCE_COUNT='0'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} USER='SYSTEM'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} Version=''
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} CLUSTER_NAME='ora112'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} DB_UNIQUE_NAME='orcl'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} ORACLE_CLUSTERWARE.SUBCOMPONENT='CRSD'
2017-11-17 21:27:31.693: [ USRTHRD][1097230656] {0:4:12} RESOURCE_CLASS='database'
2017-11-17 21:27:31.694: [ AGFW][1117509952] {0:4:12} Agent received the message: RESOURCE_START[ora.orcl.db 1 1] ID 4098:677
2017-11-17 21:27:31.694: [ AGFW][1117509952] {0:4:12} Preparing START command for: ora.orcl.db 1 1
2017-11-17 21:27:31.694: [ AGFW][1117509952] {0:4:12} ora.orcl.db 1 1 state changed from: PLANNED_OFFLINE to: STARTING
2017-11-17 21:27:31.694: [ora.orcl.db][1115408704] {0:4:12} [start] (:CLSN00107:) clsn_agent::start {
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] InstAgent::start {
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::stopConnection
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:31.695: [ USRTHRD][1115408704] {0:4:12} ConnectionPool::~ConnectionPool m_oracleHome:/u01/app/oracle/product/11.2.0/db_1, m_oracleSid:orcl, m_usrOraEnv:
2017-11-17 21:27:31.695: [ USRTHRD][1115408704] {0:4:12} Thread:DedicatedThreadstop {
2017-11-17 21:27:31.695: [ USRTHRD][1115408704] {0:4:12} Thread:DedicatedThreadstop }
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent:getOracleSid 13 oracle_sid = orcl
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent:getOracleSid oracle_sid = orcl
2017-11-17 21:27:31.695: [ora.orcl.db][1115408704] {0:4:12} [start] getOracleHomeAttrib: oracle_home = /u01/app/oracle/product/11.2.0/db_1
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/app/oracle/product/11.2.0/db_1, m_oracleSid:orcl, m_usrOraEnv:
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent::setOracleSidAttrib updating GEN_USR_ORA_INST_NAME to orcl
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] clsnUtils::setResAttrib nonPerX current value GEN_USR_ORA_INST_NAME value orcl
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] setResAttrib clsagfw_modify_attribute attr GEN_USR_ORA_INST_NAME value orcl retCode 0
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] sModifyConfig for orcl
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] getOracleHomeAttrib: oracle_home = /u01/app/oracle/product/11.2.0/db_1
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] sclsnInstAgent::sUpdateOratab oratab is /etc/oratab
2017-11-17 21:27:31.696: [ora.orcl.db][1115408704] {0:4:12} [start] sclsnInstAgent::sUpdateOratab CssLock
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent:getOracleSid 13 oracle_sid = orcl
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent:getOracleSid oracle_sid = orcl
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] ConfigFile::getValue name orcl value
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] ConfigFile::updateInPlace file /etc/oratab is updated
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] sclsnInstAgent::sUpdateOratab file updated with dbName orcl value /u01/app/oracle/product/11.2.0/db_1:N
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] sclsnInstAgent::sUpdateOratab CSS unlock
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] SPFILE attribute is not set.
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] crsHome = /u01/app/11.2.0/grid
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] oracleHome = /u01/app/oracle/product/11.2.0/db_1
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] command = '/u01/app/11.2.0/grid/bin/setasmgidwrap oracle_binary_path=/u01/app/oracle/product/11.2.0/db_1/bin/oracle'
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] start dependency = weak(type:ora.listener.type,uniform:ora.ons)
2017-11-17 21:27:31.697: [ora.orcl.db][1115408704] {0:4:12} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:31.698: [ora.orcl.db][1115408704] {0:4:12} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/oracle/product/11.2.0/db_1 Crshome /u01/app/11.2.0/grid
2017-11-17 21:27:31.698: [ora.orcl.db][1115408704] {0:4:12} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:31.698: [ora.orcl.db][1115408704] {0:4:12} [start] clsnInstConnection::makeConnectStr LIBRARY_PATH1 ,LD_LIBRARY_PATH=
2017-11-17 21:27:31.698: [ora.orcl.db][1115408704] {0:4:12} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/oracle/product/11.2.0/db_1/bin/oracle)(ARGV0=oracleorcl)(ENVS='ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1,ORACLE_SID=orcl,LD_LIBRARY_PATH=')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=orcl)))
2017-11-17 21:27:31.698: [ora.orcl.db][1115408704] {0:4:12} [start] Container:start oracle home /u01/app/oracle/product/11.2.0/db_1
2017-11-17 21:27:31.698: [ora.orcl.db][1115408704] {0:4:12} [start] InstConnection::connectInt: server not attached
2017-11-17 21:27:31.710: [ora.orcl.db][1115408704] {0:4:12} [start] InstAgent::startup
2017-11-17 21:27:31.806: [ USRTHRD][1123813696] {0:4:12} EVM EonsProxyEventForwarder::postMyEvent Event posted
2017-11-17 21:27:32.261: [ora.orcl.db][1115408704] {0:4:12} [start] InstConnection::connectInt: server not attached
2017-11-17 21:27:32.306: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent::startCbk: {
2017-11-17 21:27:32.307: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent::startCbk: {
2017-11-17 21:27:32.307: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgentEventModule::keepRunning starting event bridge
2017-11-17 21:27:32.309: [ USRTHRD][1115408704] {0:4:12} In DedicatedThread::generateLocalListener
2017-11-17 21:27:32.309: [ USRTHRD][1115408704] {0:4:12} m_crsHome = /u01/app/11.2.0/grid
2017-11-17 21:27:32.333: [ USRTHRD][1115408704] {0:4:12} Add endpoint from ora.LISTENER.lsnr to LOCAL_LISTENER
2017-11-17 21:27:32.333: [ USRTHRD][1115408704] {0:4:12} CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
2017-11-17 21:27:32.357: [ USRTHRD][1115408704] {0:4:12} CrsCmd::ClscrsCmdData::stat 2-ClscrsCmdData::check clscrs_res_get_attr clscrsret:5 attribute:ENDPOINTS@SERVERNAME(ora112) value:null
2017-11-17 21:27:32.357: [ USRTHRD][1115408704] {0:4:12} CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2017-11-17 21:27:32.388: [ USRTHRD][1115408704] {0:4:12} ORA-01405: fetched column value is NULL
2017-11-17 21:27:32.388: [ USRTHRD][1115408704] {0:4:12} Value of LOCAL_LISTENER is
2017-11-17 21:27:32.388: [ USRTHRD][1115408704] {0:4:12} sqlStmt = ALTER SYSTEM SET LOCAL_LISTENER='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))))' SCOPE=MEMORY SID='orcl' /* db agent *//* {0:4:12} */
2017-11-17 21:27:32.389: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent::startCbk option 2 mount ALTER DATABASE MOUNT /* db agent *//* {0:4:12} */
2017-11-17 21:27:36.502: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent::startCbk open ALTER DATABASE OPEN /* db agent *//* {0:4:12} */
2017-11-17 21:27:37.143: [ USRTHRD][1115408704] {0:4:12} DbAgentEventModule::notifyDbOpen database is now open
2017-11-17 21:27:37.143: [ USRTHRD][1115408704] {0:4:12} DbAgentEventModule::notifyDbOpen starting AQ
2017-11-17 21:27:37.143: [ USRTHRD][1115408704] {0:4:12} DbAgentEventModule::notifyDbOpen starting RLB
2017-11-17 21:27:37.143: [ AGENT][1115408704] {0:4:12} ClusterPublisher::connect connecting
2017-11-17 21:27:37.144: [ AGENT][1115408704] {0:4:12} ClusterPublisher::connect Connected
2017-11-17 21:27:37.144: [ora.orcl.db][1115408704] {0:4:12} [start] DbAgent::startCbk: }
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] InstAgent::start: startConnection pool
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::stopConnection
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::removeConnection connection count 0
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::removeConnection freed 0
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::stopConnection sid orcl status 1
2017-11-17 21:27:37.145: [ USRTHRD][1115408704] {0:4:12} ConnectionPool::~ConnectionPool m_oracleHome:/u01/app/oracle/product/11.2.0/db_1, m_oracleSid:orcl, m_usrOraEnv:
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/app/oracle/product/11.2.0/db_1, m_oracleSid:orcl, m_usrOraEnv:
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::startConnection s_statusOfConnectionMap 00a87d90
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::startConnection sid orcl status 2
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] InstAgent::start release pConnxn 0081e5f0 pConnxn
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] ConnectionPool::releaseConnection InstConnection 17404450
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] InstAgent::start: }
2017-11-17 21:27:37.145: [ora.orcl.db][1115408704] {0:4:12} [start] (:CLSN00107:) clsn_agent::start }
2017-11-17 21:27:37.145: [ AGFW][1115408704] {0:4:12} Command: start for resource: ora.orcl.db 1 1 completed with status: SUCCESS
2017-11-17 21:27:37.146: [ AGFW][1117509952] {0:4:12} Agent sending reply for: RESOURCE_START[ora.orcl.db 1 1] ID 4098:677
2017-11-17 21:27:37.148: [ora.orcl.db][1115408704] {0:4:12} [check] Gimh::check OH /u01/app/oracle/product/11.2.0/db_1 SID orcl
2017-11-17 21:27:37.148: [ora.orcl.db][1115408704] {0:4:12} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 9 exists
2017-11-17 21:27:37.148: [ora.orcl.db][1115408704] {0:4:12} [check] updating GEN_START_OPTIONS from to open
2017-11-17 21:27:37.148: [ USRTHRD][1115408704] {0:4:12} Thread:DedicatedThreadstart {
2017-11-17 21:27:37.148: [ USRTHRD][1115408704] {0:4:12} Thread:DedicatedThreadstart }
2017-11-17 21:27:37.148: [ AGFW][1117509952] {0:4:12} ora.orcl.db 1 1 state changed from: STARTING to: ONLINE
2017-11-17 21:27:37.148: [ AGFW][1117509952] {0:4:12} Started implicit monitor for [ora.orcl.db 1 1] interval=1000 delay=1000
2017-11-17 21:27:37.149: [ AGFW][1117509952] {0:4:12} ora.orcl.db 1 1 state details has changed from: Instance Shutdown to: Open
2017-11-17 21:27:37.149: [ AGFW][1117509952] {0:4:12} Agent sending last reply for: RESOURCE_START[ora.orcl.db 1 1] ID 4098:677
2017-11-17 21:27:37.155: [ USRTHRD][1128016192] {0:4:12} DbAgent::Dedicated Thread {
2017-11-17 21:27:37.155: [ USRTHRD][1128016192] {0:4:12} CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
2017-11-17 21:27:37.202: [ USRTHRD][1128016192] {0:4:12} setResAttrib: attr GEN_START_OPTIONS clsagfw attribVal open clscrs tmpAttrValue open
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} setResAttrib clsagfw_modify_attribute attr GEN_START_OPTIONS value open retCode 0
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} DbAgent::DThread getConnxn
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/oracle/product/11.2.0/db_1 Crshome /u01/app/11.2.0/grid
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} clsnInstConnection::makeConnectStr LIBRARY_PATH1 ,LD_LIBRARY_PATH=
2017-11-17 21:27:37.203: [ USRTHRD][1128016192] {0:4:12} makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/oracle/product/11.2.0/db_1/bin/oracle)(ARGV0=oracleorcl)(ENVS='ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1,ORACLE_SID=orcl,LD_LIBRARY_PATH=')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=orcl)))
2017-11-17 21:27:37.204: [ USRTHRD][1128016192] {0:4:12} InstConnection::connectInt: server not attached
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] ClusterSubscriber::SubscriberWorker::InternalClusterSubscriber::handleEventCBexecuting for reason 1
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] event type is CRS_RESOURCE_STATE_CHANGE
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] bodylen = 731
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} -----------BodyBlock----------
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} CLS_TINT='{0:4:12}'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} COMPOSITE_RESOURCE_STATUS='FIRST_ONLINE'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} CURRENT_STATE='ONLINE'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} DATABASE_TYPE='SINGLE'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} DB_UNIQUE_NAME='orcl'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} DEGREE='1'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} DERIVED_STATE='ONLINE'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} ID='403083288'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} INSTANCE_NAME='orcl'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} INTERMEDIATE_INSTANCE_COUNT='0'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} NAME='ora.orcl.db'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} ONLINE_INSTANCE_COUNT='1'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} PREVIOUS_STATE='OFFLINE'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} PREV_STATE_DETAILS='Instance Shutdown'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} REASON='FAILURE'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} RESOURCE_CLASS='database'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} RESOURCE_INCARNATION_NUMBER='2'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} RESOURCE_INSTANCE_ID='ora.orcl.db 1 1'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} RESOURCE_LOCATION='ora112'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} SEQUENCE_NUMBER='200022'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} STATE_DETAILS='Open'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} TIMESTAMP='2017-11-17 21:27:37'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} TOTAL_INSTANCE_COUNT='1'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} TYPE='ora.database.type'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} UNKNOWN_INSTANCE_COUNT='0'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} USER='SYSTEM'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} Version=''
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} CLUSTER_NAME='ora112'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} DB_UNIQUE_NAME='orcl'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} ORACLE_CLUSTERWARE.SUBCOMPONENT='CRSD'
2017-11-17 21:27:37.243: [ USRTHRD][1097230656] {0:4:12} RESOURCE_CLASS='database'
A falha foi detectada 2017-11-17 21:27:30 e 2017-11-17 21:27:37 a instance orcl estava online novamente para atender aos usuários. Vale muito apena usar o Oracle RESTART e eu sugiro uma lida na documentação oficial do produto e precisando de um help sobre o comando, basta usar o srvctl –h:
[oracle@ora112 ~]$ srvctl -h
Usage: srvctl [-V]
Usage: srvctl add database -d -o [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s ] [-t ] [-n ] [-i ] [-y {AUTOMATIC | MANUAL | NORESTART}] [-a ""]
Usage: srvctl config database [-d [-a]] [-v]
Usage: srvctl start database -d [-o ]
Usage: srvctl stop database -d [-o ] [-f]
Usage: srvctl status database -d [-f] [-v]
Usage: srvctl enable database -d
Usage: srvctl disable database -d
Usage: srvctl modify database -d [-n ] [-i ] [-o ] [-u ] [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s ] [-t ] [-y {AUTOMATIC | MANUAL | NORESTART}] [-a ""|-z] [-f]
Usage: srvctl remove database -d [-f] [-y] [-v]
Usage: srvctl getenv database -d [-t ""]
Usage: srvctl setenv database -d {-t =[,=,...] | -T =}
Usage: srvctl unsetenv database -d -t ""
Usage: srvctl upgrade database -d -o
Usage: srvctl downgrade database -d -o -t
Usage: srvctl add service -d -s [-l [PRIMARY][,PHYSICAL_STANDBY][,LOGICAL_STANDBY][,SNAPSHOT_STANDBY]] [-y {AUTOMATIC | MANUAL}][-q {true|false}] [-j {SHORT|LONG}] [-B {NONE|SERVICE_TIME|THROUGHPUT}][-e {NONE|SESSION|SELECT}] [-m {NONE|BASIC}][-z ] [-w ] [-t ] [-f]
Usage: srvctl config service -d [-s ] [-v]
Usage: srvctl start service -d [-s ""] [-o ]
Usage: srvctl stop service -d [-s ""] [-f]
Usage: srvctl status service -d [-s ""] [-f] [-v]
Usage: srvctl enable service -d -s ""
Usage: srvctl disable service -d -s ""
Usage: srvctl modify service -d -s [-l [PRIMARY][,PHYSICAL_STANDBY][,LOGICAL_STANDBY][,SNAPSHOT_STANDBY]] [-y {AUTOMATIC | MANUAL}][-q {true|false}] [-j {SHORT|LONG}] [-B {NONE|SERVICE_TIME|THROUGHPUT}] [-e {NONE|SESSION|SELECT}] [-m {NONE|BASIC}] [-z ] [-w ] [-t ]
Usage: srvctl remove service -d -s [-f]
Usage: srvctl add asm [-l ] [-p ] [-d ]
Usage: srvctl config asm [-a]
Usage: srvctl start asm [-o ]
Usage: srvctl stop asm [-o ] [-f]
Usage: srvctl status asm [-a] [-v]
Usage: srvctl enable asm
Usage: srvctl disable asm
Usage: srvctl modify asm [-l ] [-p ] [-d ]
Usage: srvctl remove asm [-f]
Usage: srvctl getenv asm [-t [, ...]]
Usage: srvctl setenv asm -t "= [,...]" | -T "="
Usage: srvctl unsetenv asm -t "[, ...]"
Usage: srvctl start diskgroup -g
Usage: srvctl stop diskgroup -g [-f]
Usage: srvctl status diskgroup -g [-a] [-v]
Usage: srvctl enable diskgroup -g
Usage: srvctl disable diskgroup -g
Usage: srvctl remove diskgroup -g [-f]
Usage: srvctl add listener [-l ] [-s] [-p "[TCP:][, ...][/IPC:][/NMP:][/TCPS:] [/SDP:]"] [-o ]
Usage: srvctl config listener [-l ]
Usage: srvctl start listener [-l ]
Usage: srvctl stop listener [-l ] [-f]
Usage: srvctl enable listener [-l ]
Usage: srvctl disable listener [-l ]
Usage: srvctl modify listener [-l ] [-o ] [-p "[TCP:][, ...][/IPC:][/NMP:][/TCPS:] [/SDP:]"]
Usage: srvctl remove listener [-l | -a] [-f]
Usage: srvctl getenv listener [-l ] [-t [, ...]]
Usage: srvctl setenv listener [-l ] -t "= [,...]" | -T "="
Usage: srvctl unsetenv listener [-l ] -t "[, ...]"
Usage: srvctl start home -o -s
Usage: srvctl stop home -o -s [-t ] [-f]
Usage: srvctl status home -o -s
Usage: srvctl add ons [-e ] [-l ] [-r ] [-t [:][,[:]...]] [-v]
Usage: srvctl remove ons [-f] [-v]
Usage: srvctl enable ons [-v]
Usage: srvctl disable ons [-v]
Usage: srvctl modify ons [-e ] [-l ] [-r ] [-t [:][,[:]...]] [-v]
Usage: srvctl config ons
Usage: srvctl status ons [-v]
Usage: srvctl start ons [-v]
Usage: srvctl stop ons [-v]
E você também pode filtrar a opção, exemplo:
[oracle@ora112 ~]$ srvctl add database -h
Adds a database configuration to be managed by Oracle Restart.
Usage: srvctl add database -d -o [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s ] [-t ] [-n ] [-i ] [-y {AUTOMATIC | MANUAL | NORESTART}] [-a ""]
-d Unique name for the database
-o ORACLE_HOME path
-m Domain for database. Must be set if database has DB_DOMAIN set.
-p Server parameter file path
-r Role of the database (primary, physical_standby, logical_standby, snapshot_standby)
-s Startup options for the database. Examples of startup options are OPEN, MOUNT, or 'READ ONLY'.
-t Stop options for the database. Examples of shutdown options are NORMAL, TRANSACTIONAL, IMMEDIATE, or ABORT.
-n Database name (DB_NAME), if different from the unique name given by the -d option
-i Instance name
-y Management policy for the database (AUTOMATIC, MANUAL, or NORESTART)
-a "" Comma separated list of disk groups
-h Print usage
Caso precise fazer alguma manutenção, como aplicação de um patch ou algo que exija a parada de todos os processos oracle em execução, basta seguir com os comandos abaixo:
Parando todo o HAS – High Availability Services
[oracle@ora112]$ crsctl disable has CRS-4621: Oracle High Availability Services autostart is disabled.
[oracle@ora112]$ crsctl stop has CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora112 ' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ora112 ' CRS-2673: Attempting to stop 'ora.orcl2.db' on 'ora112 ' CRS-2673: Attempting to stop 'ora.MUFALANI.lsnr' on 'ora112 ' CRS-2673: Attempting to stop 'ora.orcl.db' on 'ora112 ' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ora112 ' succeeded CRS-2677: Stop of 'ora.orcl2.db' on 'ora112 ' succeeded CRS-2677: Stop of 'ora.orcl.db' on 'ora112 ' succeeded CRS-2677: Stop of 'ora.MUFALANI.lsnr' on 'ora112 ' succeeded CRS-2673: Attempting to stop 'ora.evmd' on 'ora112 ' CRS-2677: Stop of 'ora.evmd' on 'ora112 ' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora112 ' has completed CRS-4133: Oracle High Availability Services has been stopped.
Depois de realizar a manutenção, podemos subir o HAS, seguindo os comandos abaixo:
Para habilitar o HAS – High Availability Services
[oracle@ora112]$ crsctl enable has CRS-4622: Oracle High Availability Services autostart is enabled.
[oracle@ora112]$ crsctl start has CRS-4123: Oracle High Availability Services has been started.
[oracle@ora112]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....ER.lsnr ora....er.type ONLINE ONLINE ora112 ora....NI.lsnr ora....er.type ONLINE ONLINE ora112 ora.cssd ora.cssd.type OFFLINE OFFLINE ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE ora112 ora.ons ora.ons.type OFFLINE OFFLINE ora.orcl.db ora....se.type ONLINE ONLINE ora112 ora.orcl2.db ora....se.type ONLINE ONLINE ora112
Espero que esse artigo tenha lhe ajudado.
Att,
Rodrigo Mufalani