论坛徽章:: 0

电梯直达

1楼 [收藏(0)] [报告]

发表于 2012-09-05 11:01 |只看该作者 |倒序浏览

本帖最后由 xc5sm 于 2012-09-05 11:15 编辑

NBU master server + VCS
二台AIX ,分别上 ERP, ORACLE +vvr
VVR服务偶尔会影响oracle上的cpu wait 很高，现在已经影响到客户的erp业务了，小弟不才，在下面的信息中找不到问题所在，请各位帮忙分析下，谢谢

补充下，如果把VVR暂停了，cpu wait马上就降下来了。VVR影响cpu wait是偶尔会出现，大部分时候是正常的。

# vxrlink -g oradisk -i5 status rlk_p550-dr_oradata_rvg

Wed Sep  5 10:25:59 GMT+08:00 2012
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1846 outstanding writes, occupying 20386 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1820 outstanding writes, occupying 20057 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1819 outstanding writes, occupying 19746 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1819 outstanding writes, occupying 19746 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1819 outstanding writes, occupying 19746 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1819 outstanding writes, occupying 19746 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1819 outstanding writes, occupying 19746 Kbytes (0%) on

the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink rlk_p550-dr_oradata_rvg has 1819 outstanding writes, occupying 19746 Kbytes (0%) on

the SRL
^C# iostat 2 5

System configuration: lcpu=4 drives=7 paths=18 vdisks=0

tty:    tin       tout avg-cpu: % user % sys % idle % iowait
      0.0       383.0             2.1 1.1 0.0    96.8

Disks:       % tm_act    Kbps    tps Kb_read Kb_wrtn
hdisk0          1.0    16.0    4.0       12       20
hdisk1          1.0    12.0    3.0       4       20
hdisk4          0.0    0.0    0.0       0       0
hdisk2          0.0    0.0    0.0       0       0
hdisk3          0.0    0.0    0.0       0       0
hdisk5          0.0    0.0    0.0       0       0
cd0             0.0    0.0    0.0       0       0

tty:    tin       tout avg-cpu: % user % sys % idle % iowait
      0.0       707.6             1.2 0.9 0.0    97.9

Disks:       % tm_act    Kbps    tps Kb_read Kb_wrtn
hdisk0          0.0    0.0    0.0       0       0
hdisk1          0.0    0.0    0.0       0       0
hdisk4          0.0    0.0    0.0       0       0
hdisk2          0.0    0.0    0.0       0       0
hdisk3          0.0    0.0    0.0       0       0
hdisk5          0.0    0.0    0.0       0       0
cd0             0.0    0.0    0.0       0       0

tty:    tin       tout avg-cpu: % user % sys % idle % iowait
      0.0       664.6             1.9 0.8 0.0    97.3

Disks:       % tm_act    Kbps    tps Kb_read Kb_wrtn
hdisk0          0.0    0.0    0.0       0       0
hdisk1          0.0    0.0    0.0       0       0
hdisk4          0.0    0.0    0.0       0       0
hdisk2          0.5    3.9    0.5       8       0
hdisk3          0.0    0.0    0.0       0       0
hdisk5          0.0    0.0    0.0       0       0
cd0             0.0    0.0    0.0       0       0

tty:    tin       tout avg-cpu: % user % sys % idle % iowait
      0.0       716.0             1.4 0.8 0.0    97.8

Disks:       % tm_act    Kbps    tps Kb_read Kb_wrtn
hdisk0          0.0    0.0    0.0       0       0
hdisk1          0.0    0.0    0.0       0       0
hdisk4          0.0    0.0    0.0       0       0
hdisk2          0.0    0.0    0.0       0       0
hdisk3          0.0    0.0    0.0       0       0
hdisk5          0.0    0.0    0.0       0       0
cd0             0.0    0.0    0.0       0       0

tty:    tin       tout avg-cpu: % user % sys % idle % iowait
      0.0       520.5             1.3 0.8 0.0    97.9

Disks:       % tm_act    Kbps    tps Kb_read Kb_wrtn
hdisk0          0.0    0.0    0.0       0       0
hdisk1          0.0    0.0    0.0       0       0
hdisk4          0.0    0.0    0.0       0       0
hdisk2          0.0    0.0    0.0       0       0
hdisk3          0.0    0.0    0.0       0       0
hdisk5          0.0    0.0    0.0       0       0
cd0             0.0    0.0    0.0       0       0
# vmstat 2 5

System configuration: lcpu=4 mem=15680MB

kthr memory             page             faults       cpu
----- ----------- ------------------------ ------------ -----------
r  b avm fre  re  pi  po  fr sr  cy  in sy  cs us sy id wa
0  1 4693879 27320 0 0 0 0 0 0  78 1975 716  2  1  0 97
0  1 4693880 27319 0 0 0 0 0 0  36 1771 659  1  1  0 98
0  1 4693880 27319 0 0 0 0 0 0  39 1598 669  1  1  0 98
0  1 4693880 27319 0 0 0 0 0 0  22 1559 644  1  1  0 98
1  1 4693880 27319 0 0 0 0 0 0 162 1541 721  1  2  0 97
# vmstat 2 10

System configuration: lcpu=4 mem=15680MB

kthr memory             page             faults       cpu
----- ----------- ------------------------ ------------ -----------
r  b avm fre  re  pi  po  fr sr  cy  in sy  cs us sy id wa
1  1 4693903 27296 0 0 0 0 0 0  48 36559 718 16  3  0 81
0  1 4693901 27298 0 0 0 0 0 0  20 1467 686  1  1  0 98
12  0 4689311 31756 0 0 0 0 0 0 1542 25191 3353 76 10  0 15
4  0 4688458 32528 0 0 0 0 0 0 1280 14188 3196 53  5 41  1
1  0 4688382 32592 0 0 0 0 0 0 275 4044 1089 10  2 88  0
1  0 4688311 32606 0 0 0 0 0 0 624 6509 1757 26  3 71  0
1  0 4688239 32620 0 0 0 0 0 0 334 3227 1200  4  2 93  1
1  0 4688266 32586 0 0 0 0 0 0 465 6286 1424 12  2 86  0
1  0 4688209 32634 0 0 0 0 0 0 188 2524 960  4  2 94  0
1  0 4689350 31492 0 0 0 0 0 0  45 36082 636  9  3 89  0
# vmstat 2 10

System configuration: lcpu=4 mem=15680MB

kthr memory             page             faults       cpu
----- ----------- ------------------------ ------------ -----------
r  b avm fre  re  pi  po  fr sr  cy  in sy  cs us sy id wa
1  0 4693176 27607 0 0 0 0 0 0  31 38141 770 11  3 43 43
1  0 4693178 27605 0 0 0 0 0 0  35 41532 910 11  6 42 42
1  0 4693179 27604 0 0 0 0 0 0  33 1405 656  1  1 49 49
1  0 4693180 27603 0 0 0 0 0 0  29 34866 728  3  2 47 48
1  0 4693180 27603 0 0 0 0 0 0  42 1619 728  2  1 49 49
1  0 4693180 27603 0 0 0 0 0 0  30 1801 674  1  1 49 49
1  0 4693180 27603 0 0 0 0 0 0  20 1389 646  1  1 49 49
0  0 4693179 27604 0 0 0 0 0 0  49 1886 746  2  1 49 49
0  0 4693179 27604 0 0 0 0 0 0  28 1496 694  1  1 49 49
1  0 4693179 27604 0 0 0 0 0 0  22 1431 689  1  1 49 49
# vxprint -VPl
Disk group: oradisk

Rlink: rlk_p550-dr_oradata_rvg
info:    timeout=500 rid=0.1857
      latency_high_mark=1000000 latency_low_mark=999500
      bandwidth_limit=none
state: state=ACTIVE
      synchronous=off latencyprot=override srlprot=dcm
assoc: rvg=oradata_rvg
      remote_host=p550-dr IP_addr=192.168.100.10 port=4145
      remote_dg=oradisk
      remote_dg_dgid=1335900509.22.p550-dr
      remote_rvg_version=30
      remote_rlink=rlk_erpdb_oradata_rvg
      remote_rlink_rid=0.1097
      local_host=erpdb IP_addr=192.168.10.15 port=4145
protocol: TCP/IP
flags: write enabled attached consistent connected asynchronous

Rvg:    oradata_rvg
info:    rid=0.1121 version=2 rvg_version=30 last_tag=1
state: state=ACTIVE kernel=ENABLED
assoc: datavols=oradata
      srl=srl
      rlinks=rlk_p550-dr_oradata_rvg
      exports=(none)
      vsets=(none)
att:    rlinks=rlk_p550-dr_oradata_rvg
flags: closed primary enabled attached
device: minor=7002 bdev=47/7002 cdev=47/7002 path=/dev/vx/dsk/oradisk/oradata_rvg
perms: user=root group=system mode=0600

# vradmin -g oradisk repstatus oradata_rvg
Replicated Data Set: oradata_rvg
Primary:
  Host name:                erpdb
  RVG name:                oradata_rvg
  DG name:                   oradisk
  RVG state:                enabled for I/O
  Data volumes:             1
  VSets:                   0
  SRL name:                srl
  SRL size:                40.00 G
  Total secondaries:       1

Secondary:
  Host name:                p550-dr
  RVG name:                oradata_rvg
  DG name:                   oradisk
  Data status:             consistent, behind
  Replication status:       replicating (connected)
  Current mode:             asynchronous
  Logging to:                SRL (16150 Kbytes behind, 0% full)
  Timestamp Information:    behind by 0h 9m 11s

# vxprint -lp
Disk group: oradisk

Plex:    oradata-01
info:    len=83886080
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=oradata sd=oradisk01-01
flags: busy complete
Site:    n3400_1_oradata1
mediatype: hdd
Plex:    oradata-02
info:    len=83886080
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=oradata sd=oradisk02-01
flags: busy complete
Site:    n3400_2_oradata2
mediatype: hdd
Plex:    oradata-03
info:    len=0
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=oradata sd=(none)
flags:
Site:    n3400_2_oradata2
logging:  logsd=oradisk02-05 (enabled)
mediatype: unknown
Plex:    oradata-04
info:    len=0
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=oradata sd=(none)
flags:
Site:    n3400_1_oradata1
logging:  logsd=oradisk01-05 (enabled)
mediatype: unknown
Plex:    oradata_dcl-01
info:    len=6304
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=oradata_dcl sd=oradisk01-02
flags: complete
Site:    n3400_1_oradata1
mediatype: hdd
Plex:    oradata_dcl-02
info:    len=6304
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=oradata_dcl sd=oradisk02-02
flags: complete
Site:    n3400_2_oradata2
mediatype: hdd
Plex:    srl-01
info:    len=83886080
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=srl sd=oradisk01-03
flags: complete
Site:    n3400_1_oradata1
mediatype: hdd
Plex:    srl-02
info:    len=83886080
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=srl sd=oradisk02-03
flags: complete
Site:    n3400_2_oradata2
mediatype: hdd
Plex:    srl_dcl-01
info:    len=6304
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=srl_dcl sd=oradisk01-04
flags: complete
Site:    n3400_1_oradata1
mediatype: hdd
Plex:    srl_dcl-02
info:    len=6304
type:    layout=CONCAT
state: state=ACTIVE kernel=ENABLED io=read-write
assoc: vol=srl_dcl sd=oradisk02-04
flags: complete
Site:    n3400_2_oradata2
mediatype: hdd

文库|博客

无牙

广告杀手

论坛徽章:: 1

2楼 [报告]

发表于 2012-09-05 11:51 |只看该作者

要具体看看是那个进程占CPU过高。另外CPU过高的时候是否刚有大量的写发生？

VVR的版本？

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

xc5sm

白手起家

论坛徽章:: 0

3楼 [报告]

发表于 2012-09-05 12:18 |只看该作者

回复 2# 无牙

上面的iostat 2 5结果中基本上没有读写，但wait就很高了。
请教下，vvr版本用什么命令查看{:3_186:}

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

无牙

广告杀手

论坛徽章:: 1

4楼 [报告]

发表于 2012-09-05 14:58 |只看该作者

用lslpp 看一下VRTSvvr包的信息。

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

xc5sm

白手起家

论坛徽章:: 0

5楼 [报告]

发表于 2012-09-06 09:06 |只看该作者

本帖最后由 xc5sm 于 2012-09-06 09:11 编辑

回复 4# 无牙

lslpp -d  VRTSvvr 结果提示没有这个包
lslpp -l |grep  VRTSvvr  没有结果

lslpp -l |grep VRT*
  VRTSamf                5.1.100.0  COMMITTED  Veritas AMF by Symantec
  VRTSaslapm          5.1.100.200  COMMITTED  Array Support Libraries and
  VRTSat.client          5.0.32.0  COMMITTED  Symantec Product
  VRTSat.server          5.0.32.0  COMMITTED  Symantec Product
  VRTScps                5.1.100.0  APPLIED Veritas Co-ordination Point
  VRTSdbed                5.1.100.0  APPLIED Veritas Storage Foundation for
  VRTSfssdk             5.1.100.0  COMMITTED  Veritas Libraries and Header
  VRTSgab                5.1.100.0  APPLIED Veritas Group Membership and
  VRTSllt                5.1.100.0  APPLIED Veritas Low Latency Transport
  VRTSob                3.4.235.54  APPLIED Veritas Enterprise
  VRTSodm                5.1.101.0  APPLIED Veritas Extension for Oracle
  VRTSpbx                   1.5.0.6  COMMITTED  Symantec Private Branch
  VRTSperl                5.10.0.9  COMMITTED  Perl 5.10.0 for Veritas
  VRTSsfmh             4.0.1598.0  COMMITTED  Veritas Operations Manager
  VRTSspt                   5.5.0.5  COMMITTED  Veritas Support Tools by
  VRTSvcs                5.1.100.0  APPLIED Veritas Cluster Server by
  VRTSvcsag             5.1.100.0  APPLIED Veritas Cluster Server Bundled
  VRTSvcsea             5.1.100.0  APPLIED Veritas High Availability
  VRTSveki                5.1.100.0  APPLIED Veritas Kernel Interface by
  VRTSvlic                3.2.51.10  COMMITTED  Symantec License Utilities
  VRTSvxfen             5.1.100.0  APPLIED Veritas I/O Fencing by
  VRTSvxfs                5.1.101.0  APPLIED Veritas File System by
  VRTSvxvm                5.1.101.0  APPLIED Veritas Volume Manager by
  VRTSamf                5.1.100.0  COMMITTED  Veritas AMF by Symantec
  VRTSaslapm          5.1.100.200  COMMITTED  Array Support Libraries and
  VRTSat.server          5.0.32.0  COMMITTED  Symantec Product
  VRTScps                5.1.100.0  APPLIED Veritas Co-ordination Point
  VRTSdbed                5.1.100.0  APPLIED Veritas Storage Foundation for
  VRTSgab                5.1.100.0  APPLIED Veritas Group Membership and
  VRTSllt                5.1.100.0  APPLIED Veritas Low Latency Transport
  VRTSob                3.4.235.54  APPLIED Veritas Enterprise
  VRTSodm                5.1.101.0  APPLIED Veritas Extension for Oracle
  VRTSpbx                   1.5.0.6  COMMITTED  Symantec Private Branch
  VRTSperl                5.10.0.9  COMMITTED  Perl 5.10.0 for Veritas
  VRTSsfmh             4.0.1598.0  COMMITTED  Veritas Operations Manager
  VRTSvcs                5.1.100.0  APPLIED Veritas Cluster Server by
  VRTSvcsea             5.1.100.0  APPLIED Veritas High Availability
  VRTSveki                5.1.100.0  APPLIED Veritas Kernel Interface by
  VRTSvxfen             5.1.100.0  APPLIED Veritas I/O Fencing by
  VRTSvxfs                5.1.101.0  APPLIED Veritas File System by
  VRTSvxvm                5.1.101.0  APPLIED Veritas Volume Manager by