cluster下 包无法正常切换

weixiaoma0

cluster下 包无法正常切换
跪求解，什么问题，怎么分析日志

# ls
ora_start.sh   ora_stop.sh    oracle.asc     oracle.sh      oracle.sh.log
# cat oracle.sh.log

        ########### Node "mq1": Starting package at Tue Aug 19 18:58:39 EAT 2008 ###########
Aug 19 18:58:39 - Node "mq1": Activating volume group vgmqm with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgmqm" has been successfully changed.
Aug 19 18:58:40 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Aug 19 18:58:40 - Node "mq1": Checking filesystems:
   /dev/vgmqm/lvol1
   /dev/vgoradata/lvol1
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Aug 19 18:58:40 - Node "mq1": Mounting /dev/vgmqm/lvol1 at /var/mqm
Aug 19 18:58:40 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /ora_data
Aug 19 18:58:40 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.


SQL*Plus: Release 10.2.0.3.0 - Production on 星期二 8月 19 18:58:43 2008

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

SQL> SQL> Connected to an idle instance.
SQL> ORACLE instance started.

Total System Global Area 1610612736 bytes
Fixed Size                  2045752 bytes
Variable Size             385878216 bytes
Database Buffers         1207959552 bytes
Redo Buffers               14729216 bytes
Database mounted.
# cd ../oracle1
# ls
ora_start.sh   ora_stop.sh    oracle.asc     oracle.sh      oracle.sh.log
# cat oracle.sh.log

        ########### Node "mq1": Starting package at Mon Jun  9 13:24:51 EAT 2008 ###########
Jun  9 13:24:51 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Jun  9 13:24:51 - Node "mq1": Checking filesystems:
   /dev/vgoradata/lvol1
   /dev/vgoradata/lvol2
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Jun  9 13:24:51 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /oradata01
Jun  9 13:24:52 - Node "mq1": Mounting /dev/vgoradata/lvol2 at /oradata02
Jun  9 13:24:52 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
.ICEauthority
.TTauthority
.Xauthority
.dt
.dtprofile
.firefox-license
.fonts.cache-1
.mozilla
.profile
.rhosts
.sh_history
.ssh
.sw
bin
cdrom
dev
etc
home
lib
lost+found
net
opt
oracle
oradata01
oradata02
sbin
stand
tmp
tmpho
usr
var

        ########### Node "mq1": Package start completed at Mon Jun  9 13:24:52 EAT 2008 ###########

        ########### Node "mq1": Halting package at Mon Jun  9 13:26:21 EAT 2008 ###########
.ICEauthority
.TTauthority
.Xauthority
.dt
.dtprofile
.firefox-license
.fonts.cache-1
.mozilla
.profile
.rhosts
.sh_history
.ssh
.sw
bin
cdrom
dev
etc
home
lib
lost+found
net
opt
oracle
oradata01
oradata02
sbin
stand
tmp
tmpho
usr
var
Jun  9 13:26:21 - Node "mq1": Remove IP address 172.16.2.194 from subnet 172.16.2.0
Jun  9 13:26:21 - Node "mq1": Unmounting filesystem on /dev/vgoradata/lvol2
Jun  9 13:26:21 - Node "mq1": Unmounting filesystem on /dev/vgoradata/lvol1
Jun  9 13:26:21 - Node "mq1": Deactivating volume group vgoradata
Deactivated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.

        ########### Node "mq1": Package halt completed at Mon Jun  9 13:26:24 EAT 2008 ###########

        ########### Node "mq1": Starting package at Mon Jun  9 14:13:43 EAT 2008 ###########
Jun  9 14:13:43 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Jun  9 14:13:43 - Node "mq1": Checking filesystems:
   /dev/vgoradata/lvol1
   /dev/vgoradata/lvol2
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Jun  9 14:13:43 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /oradata01
Jun  9 14:13:43 - Node "mq1": Mounting /dev/vgoradata/lvol2 at /oradata02
Jun  9 14:13:44 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
.ICEauthority
.TTauthority
.Xauthority
.dt
.dtprofile
.firefox-license
.fonts.cache-1
.mozilla
.profile
.rhosts
.sh_history
.ssh
.sw
bin
cdrom
dev
etc
home
lib
lost+found
net
opt
oracle
oradata01
oradata02
sbin
stand
tmp
tmpho
usr
var

        ########### Node "mq1": Package start completed at Mon Jun  9 14:13:44 EAT 2008 ###########

        ########### Node "mq1": Halting package at Mon Jun  9 16:44:26 EAT 2008 ###########
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.

Processing Database instance "mbcenter": log file /ora_app/ora10g/product/10.2.0//shutdown.log
logout
Jun  9 16:44:26 - Node "mq1": Remove IP address 172.16.2.194 from subnet 172.16.2.0
Jun  9 16:44:27 - Node "mq1": Unmounting filesystem on /dev/vgoradata/lvol1
Jun  9 16:44:28 - Node "mq1": Deactivating volume group vgoradata
Deactivated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.

        ########### Node "mq1": Package halt completed at Mon Jun  9 16:44:30 EAT 2008 ###########

        ########### Node "mq1": Starting package at Mon Jun  9 17:19:31 EAT 2008 ###########
Jun  9 17:19:31 - Node "mq1": Activating volume group vgoradata with exclusive option.
vgchange: Activation mode requested for the volume group "/dev/vgoradata" conflicts with configured mode.
        ERROR:  Function activate_volume_group
        ERROR:  Failed to activate vgoradata
Jun  9 17:19:31 - Node "mq1": Deactivating volume group vgoradata
Volume group "vgoradata" has been successfully changed.

        ########### Node "mq1": Package start failed at Mon Jun  9 17:19:34 EAT 2008 ###########

        ########### Node "mq1": Starting package at Mon Jun  9 17:20:19 EAT 2008 ###########
Jun  9 17:20:19 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Jun  9 17:20:20 - Node "mq1": Checking filesystems:
   /dev/vgoradata/lvol1
   /dev/vgoradata/lvol2
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Jun  9 17:20:20 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /var/mqm
Jun  9 17:20:21 - Node "mq1": Mounting /dev/vgoradata/lvol2 at /oradata01
Jun  9 17:20:21 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.

Processing Database instance "mbcenter": log file /ora_app/ora10g/product/10.2.0//startup.log
logout

        ########### Node "mq1": Package start completed at Mon Jun  9 17:20:25 EAT 2008 ###########

     

lbseraph

1. 时间没错吧？日志里怎么是2008年的？
2. 上面日志看到8月19日的应该是到启Oracle那边了吧，那后面的应该看Oracle的日志吧？cmviewcl -v看一下package的状态是什么样的？

weixiaoma0

# cmviewcl -v

CLUSTER        STATUS      
cluster1       up           
  
  NODE           STATUS       STATE        
  mq1            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c4t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2         

    PACKAGE        STATUS        STATE         AUTO_RUN     NODE        
    oracle         up            running       enabled      mq1         
      
      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual
      
      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS NAME
      Service    unknown  0             0        service1
      Subnet     up                              172.16.2.0
      
      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      mq1 (current)
      Alternate    up           enabled      mq2
  
  NODE           STATUS       STATE        
  mq2            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c8t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2  

weixiaoma0

回复 2# lbseraph

# cmviewcl -v  双机应该没有问题啊

CLUSTER        STATUS      
cluster1       up           
  
  NODE           STATUS       STATE        
  mq1            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c4t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2         

    PACKAGE        STATUS        STATE         AUTO_RUN     NODE        
    oracle         up            running       enabled      mq1         
      
      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual
      
      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS NAME
      Service    unknown  0             0        service1
      Subnet     up                              172.16.2.0
      
      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      mq1 (current)
      Alternate    up           enabled      mq2
  
  NODE           STATUS       STATE        
  mq2            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c8t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2  


   

lbseraph

回复 4# weixiaoma0

那就是说双机切包机制正常，下一步你应该看应用的日志来查你的问题了。
   

weixiaoma0

回复 5# lbseraph

嗯，但是包切换的话，另一个节点不行，如果VG有问题的话，会在oracle.sh.log会显示吗？

我从那边下手找故障点呢？谢谢！！！
   

lbseraph

如果是VG有问题导致没法mount LV之类，你连package都没法启动起来的。但你现在cmviewcl的输出可以看到这个package已经是up and running的状态了。如果包没起来，package的log可以看出VG的问题（如果有的话，比如无法激活VG之类）。