一、测试环境
名称 | 值 |
---|---|
cpu | Intel® Core™ i5-1035G1 CPU @ 1.00GHz |
操作系统 | CentOS Linux release 7.9.2009 (Core) |
内存 | 4G |
逻辑核数 | 3 |
Gbase-8a节点1-IP | 192.168.142.10 |
Gbase-8a节点2-IP | 192.168.142.11 |
Gbase-8a数据库版本 | 8.6.2.43-R33.132743 |
二、C3安装
c3这个工具需要各节点免密才可以正常使用,免密步骤大家可以百度一下,这里就不写了,嘿嘿。
1、安装rpm包
[root@xdw0 pkg]# rpm -ivh c3-5.1.2-1.noarch.rpm --force
准备中... ################################# [100%]
正在升级/安装...
1:c3-5.1.2-1 ################################# [100%]
2、c3.conf
[gbase@xdw0 ~]$ cat /etc/c3.conf
cluster coor {
192.168.142.10:127.0.0.1
dead remove-index-00
192.168.142.10
}
cluster data {
192.168.142.10:127.0.0.1
dead remove-index-00
192.168.142.11
192.168.142.10
}
cluster all {
192.168.142.10:127.0.0.1
dead remove-index-00
192.168.142.11
192.168.142.10
}
三、常用命令
1、查看集群状态
[gbase@xdw0 ~]$ gcadmin
CLUSTER STATE: ACTIVE
CLUSTER MODE: NORMAL
=====================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
=====================================================================
| NodeName | IpAddress |gcware |gcluster |DataState |
---------------------------------------------------------------------
| coordinator1 | 192.168.142.10 | OPEN | OPEN | 0 |
---------------------------------------------------------------------
=================================================================
| GBASE DATA CLUSTER INFORMATION |
=================================================================
|NodeName | IpAddress |gnode |syncserver |DataState |
-----------------------------------------------------------------
| node1 | 192.168.142.10 | OPEN | OPEN | 0 |
-----------------------------------------------------------------
| node2 | 192.168.142.11 | OPEN | OPEN | 0 |
-----------------------------------------------------------------
2、查看各节点执行的sql
(1)查看管理节点的sql
[gbase@xdw0 ~]$ cexec coor: 'gccli -uroot -pqwer1234 -e"show full processlist"|grep -iv Sleep'
************************* coor *************************
--------- 192.168.142.10---------
Warning: Permanently added '192.168.142.10' (ECDSA) to the list of known hosts.
Id Tid User Host db Command Time State Info
1 32124 event_scheduler localhost NULL Daemon 30160 Waiting for next activation NULL
77 60202 root localhost NULL Query 0 NULL show full processlist
(2)查看数据节点的sql
[gbase@xdw0 ~]$ cexec data: 'gncli -uroot -pqwer1234 -e"show full processlist"|grep -iv Sleep'
************************* data *************************
--------- 192.168.142.11---------
Warning: Permanently added '192.168.142.11' (ECDSA) to the list of known hosts.
Id Tid User Host db Command Time State Info
150 34537 root localhost NULL Query 0 NULL show full processlist
--------- 192.168.142.10---------
Warning: Permanently added '192.168.142.10' (ECDSA) to the list of known hosts.
Id Tid User Host db Command Time State Info
59 31544 root localhost NULL Query 0 NULL show full processlist
3、查看数据节点最大trace文件
有时trace文件并不是在当前管理节点生成的(有,但不是完整版),而是在某个数据节点生成的完整版。
通过大小来判断哪个是最完整的日志。
cexec data: 'ls -lrt /opt/gnode/log/gbase/*.trc|tail -n1 '
4、查看数据分布策略
(1)方式一
[gbase@xdw0 ~]$ gcadmin showdistribution
Distribution ID: 5 | State: new | Total segment num: 2
Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 192.168.142.11 | 1 | 192.168.142.10 |
------------------------------------------------------------------------------------------------------------------------
| 192.168.142.10 | 2 | 192.168.142.11 |
========================================================================================================================
(2)方式二
[gbase@xdw0 ~]$ gcadmin showdistribution node
Distribution ID: 5 | State: new | Total segment num: 2
============================================================================================
|nodes | 192.168.142.11 | 192.168.142.10 |
--------------------------------------------------------------------------------------------
|primary | 1 | 2 |
|segments | | |
--------------------------------------------------------------------------------------------
|duplicate | 2 | 1 |
|segments 1| | |
============================================================================================
(3)JSON方式
[gbase@xdw0 ~]$ gcadmin showdistribution f
<?xml version='1.0' encoding="utf-8"?>
<distributions>
<distribution>
<distribution_id>5</distribution_id>
<distribution_state>new</distribution_state>
<total_segment_num>2</total_segment_num>
<segments>
<segment>
<primarynode ip="192.168.142.11"/>
<duplicatenodes>
<duplicatenode ip="192.168.142.10"/>
</duplicatenodes>
</segment>
<segment>
<primarynode ip="192.168.142.10"/>
<duplicatenodes>
<duplicatenode ip="192.168.142.11"/>
</duplicatenodes>
</segment>
</segments>
</distribution>
</distributions>
5、查看集群锁状态
[gbase@xdw0 ~]$ gcadmin showlock
+===============================================+
| GCLUSTER LOCK |
+===============================================+
+---------+-----+-------+-----------+------+----+
|Lock name|owner|content|create time|orphan|type|
+---------+-----+-------+-----------+------+----+
Total : 0
6、启停集群(86版本)
(1)启动
[root@xdw0 ~]# service gcware start
Starting GCWARE (gcwexec): [ 确定 ]
Starting GBASED : [ 确定 ]
Starting GCSYNC : [ 确定 ]
Starting GCLUSTERD : [ 确定 ]
Starting GCRECOVER : [ 确定 ]
Starting GCMonit success!
(2)停止
[root@xdw0 ~]# service gcware stop
Stopping GCMonit success!
Signaling GCRECOVER (gcrecover) to terminate: [ 确定 ]
Waiting for gcrecover services to unload:.... [ 确定 ]
Signaling GCSYNC (gc_sync_server) to terminate: [ 确定 ]
Waiting for gc_sync_server services to unload: [ 确定 ]
Signaling GCLUSTERD to terminate: [ 确定 ]
Waiting for gclusterd services to unload:......... [ 确定 ]
Signaling GBASED to terminate: [ 确定 ]
Waiting for gbased services to unload:.... [ 确定 ]
Signaling GCWARE (gcware) to terminate: [ 确定 ]
Waiting for gcware services to unload:.. [ 确定 ]
7、查看各节点错误日志
(1)数据节点
express.log
[gbase@xdw0 ~]$ cexec data: 'tail -10 /opt/gnode/log/gbase/express.log|grep ERROR '
************************* data *************************
--------- 192.168.142.11---------
Warning: Permanently added '192.168.142.11' (ECDSA) to the list of known hosts.
--------- 192.168.142.10---------
Warning: Permanently added '192.168.142.10' (ECDSA) to the list of known hosts.
system.log
[gbase@xdw0 ~]$ cexec data: 'tail -1000 /opt/gnode/log/gbase/system.log|grep ERROR '
************************* data *************************
--------- 192.168.142.11---------
Warning: Permanently added '192.168.142.11' (ECDSA) to the list of known hosts.
--------- 192.168.142.10---------
Warning: Permanently added '192.168.142.10' (ECDSA) to the list of known hosts.
(2)管理节点
express.log
[gbase@xdw0 ~]$ cexec coor: 'tail -10 /opt/gcluster/log/gcluster/express.log|grep ERROR '
************************* coor *************************
--------- 192.168.142.10---------
Warning: Permanently added '192.168.142.10' (ECDSA) to the list of known hosts.
2022-08-17 09:00:00.496 [SQLDISP][ERROR][S:2][Q:0]:CAsyncSqlDispatcherArgs have not target node information.
2022-08-17 09:00:00.496 [SQLDISP][ERROR][S:2][Q:0]:Fail to execute a CEventSqlExecutor.
2022-08-17 09:00:01.313 [SQLDISP][ERROR][S:10][Q:40]:Query failed, THD(0x5876800) HOST(::ffff:192.168.142.10:5050->8) reason: (GBA-01EX-700) Gbase general error: (gns_host: ::ffff:192.168.142.10) source table and destination table are not same: src:( localhost:gclusterdb.import_audit_log_errors_n1 ) dst:( ::ffff:192.168.142.11:gclusterdb.import_audit_log_errors_n1 ) column type: [1]type: 1, precision: 256, scale: 0, is_lookup: 0, nulls_mode: 2, internal_size: 256, display_size: 64 -- type: 1, precision: 256, scale: 0, is_lookup: 0, nulls_mode: 2, internal_size: 256, display_size: 85 SQL(SELECT * from `gclusterdb`.`import_audit_log_errors_n1` limit 0 target into server (HOST '::ffff:192.168.142.10,::ffff:192.168.142.11',PORT '5050', USER 'root', PASSWORD '',DATABASE 'gclusterdb',TABLE 'import_audit_log_errors_n1',COMMENT 'table_host 0 0 1, scn 917506, gc_check '))
2022-08-17 09:00:01.457 [INSERT][ERROR][S:10][Q:40]:Commit failed
ERROR NO: 1733ERROR DETAIL: (GBA-01EX-700) Gbase general error: (gns_host: ::ffff:192.168.142.10) source table and destination table are not same: src:( localhost:gclusterdb.import_audit_log_errors_n1 ) dst:( ::ffff:192.168.142.11:gclusterdb.import_audit_log_errors_n1 ) column type: [1]type: 1, precision: 256, scale: 0, is_lookup: 0, nulls_mode: 2, internal_size: 256, display_size: 64 -- type: 1, precision: 256, scale: 0, is_lookup: 0, nulls_mode: 2, internal_size: 256, display_size: 85
system.log
[gbase@xdw0 ~]$ cexec coor: 'tail -10 /opt/gcluster/log/gcluster/system.log|grep ERROR '
************************* coor *************************
--------- 192.168.142.10---------
Warning: Permanently added '192.168.142.10' (ECDSA) to the list of known hosts.
220817 8:58:26 [ERROR] (GBA-03OT-0003) Failed to get mode of gcluster, having error: GC_AIS_ERR_LIBRARY
8、查看是否是固态盘
SSD是非转动盘,如果返回结果为0说明是SSD硬盘,如果返回结果为1,说明是转动盘HDD类的硬盘。
[gbase@czg1 ~]$ lsblk -d -o name,rota
NAME ROTA
sda 1
sr0 1
9、打印堆栈
(1)打印某管理节点服务的堆栈
[gbase@czg0 ~]$ gstack `pidof gclusterd`
Thread 29 (Thread 0x7f4e22c77700 (LWP 5340)):
#0 0x00007f4e2287b3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x0000000000fe51df in signal_hand ()
#2 0x00007f4e22873ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f4e1e145b0d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f4e1d5a6700 (LWP 5341)):
#0 0x00007f4e22877de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000ef2774 in async_detect_fn(void*) ()
#2 0x00007f4e22873ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f4e1e145b0d in clone () from /lib64/libc.so.6
(2)打印某数据节点服务的堆栈
[gbase@czg0 ~]$ gstack `pidof gbased`
Thread 527 (Thread 0x7fece7305700 (LWP 2188)):
#0 0x00007fece67353c1 in sigwait () from /lib64/libpthread.so.0
#1 0x00000000007ce15f in signal_hand ()
#2 0x00007fece672dea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fece3827b0d in clone () from /lib64/libc.so.6
Thread 526 (Thread 0x7fec47808700 (LWP 2246)):
#0 0x00007fece37ee9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fece381f2d4 in usleep () from /lib64/libc.so.6
#2 0x0000000000becd4f in ExpressThread::SleepNow(int) ()
#3 0x0000000000ea3b57 in prioritylist_manager(void*) ()
#4 0x00007fece672dea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fece3827b0d in clone () from /lib64/libc.so.6
Thread 525 (Thread 0x7fec477c7700 (LWP 2247)):
#0 0x00007fece37ee9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fece381f2d4 in usleep () from /lib64/libc.so.6
#2 0x0000000000becd4f in ExpressThread::SleepNow(int) ()
#3 0x0000000000ea38b6 in locklist_manager(void*) ()
#4 0x00007fece672dea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fece3827b0d in clone () from /lib64/libc.so.6
(3)gcore生成core,并采集堆栈信息
创建目录,并生成core文件
[gbase@czg1 ~]$ mkdir corefile
[gbase@czg1 ~]$ cd corefile/
[gbase@czg1 corefile]$ gcore `pidof gclusterd`
[New LWP 29030]
[New LWP 15294]
[New LWP 15293]
[New LWP 15292]
[New LWP 15184]
[New LWP 15183]
[New LWP 3847]
[New LWP 3846]
[New LWP 3822]
[New LWP 3821]
[New LWP 3820]
[New LWP 3818]
[New LWP 3808]
[New LWP 3807]
[New LWP 3803]
[New LWP 3801]
[New LWP 3799]
[New LWP 3798]
[New LWP 3797]
[New LWP 3795]
[New LWP 3794]
[New LWP 3793]
[New LWP 3792]
[New LWP 3791]
[New LWP 3790]
[New LWP 3787]
[New LWP 3763]
[New LWP 3762]
[New LWP 3719]
[New LWP 3718]
[New LWP 3717]
[New LWP 3714]
[New LWP 3664]
[New LWP 3636]
[New LWP 3635]
[New LWP 3634]
[New LWP 3632]
[New LWP 3630]
[New LWP 3629]
[New LWP 3628]
[New LWP 3627]
[New LWP 3626]
[New LWP 3625]
[New LWP 3405]
[New LWP 3401]
[New LWP 3399]
[New LWP 3371]
[New LWP 3090]
[New LWP 3087]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fb03916ab43 in select () from /lib64/libc.so.6
Saved corefile core.3015
[Inferior 1 (process 3015) detached]
[gbase@czg1 corefile]$ ll
总用量 2177224
-rw-rw-r-- 1 gbase gbase 2229473544 11月 8 09:35 core.3015
采集堆栈信息
gdb /opt/gbase/192.168.142.11/gcluster/server/bin/gclusterd /home/gbase/corefile/core.3015 << EOF > /home/gbase/corefile/gclusterd.3015.dump 2>&1
thread apply all bt
EOF
[gbase@czg1 corefile]$ gdb /opt/gbase/192.168.142.11/gcluster/server/bin/gclusterd /home/gbase/corefile/core.3015 << EOF > /home/gbase/corefile/gclusterd.3015.dump 2>&1
> thread apply all bt
> EOF
[gbase@czg1 corefile]$ ll
总用量 2177248
-rw-rw-r-- 1 gbase gbase 2229473544 11月 8 09:35 core.3015
-rw-rw-r-- 1 gbase gbase 24143 11月 8 09:56 gclusterd.3015.dump
查看堆栈信息
[gbase@czg1 corefile]$ cat gclusterd.3015.dump
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/gbase/192.168.142.11/gcluster/server/bin/gclusterd...(no debugging symbols found)...done.
[New LWP 3087]
[New LWP 3090]
[New LWP 3371]
[New LWP 3399]
[New LWP 3401]
[New LWP 3405]
[New LWP 3625]
[New LWP 3626]
[New LWP 3627]
[New LWP 3628]
[New LWP 3629]
[New LWP 3630]
[New LWP 3632]
[New LWP 3634]
[New LWP 3635]
[New LWP 3636]
[New LWP 3664]
[New LWP 3714]
[New LWP 3717]
[New LWP 3718]
[New LWP 3719]
[New LWP 3762]
[New LWP 3763]
[New LWP 3787]
[New LWP 3790]
[New LWP 3791]
[New LWP 3792]
[New LWP 3793]
[New LWP 3794]
[New LWP 3795]
[New LWP 3797]
[New LWP 3798]
[New LWP 3799]
[New LWP 3801]
[New LWP 3803]
[New LWP 3807]
[New LWP 3808]
[New LWP 3818]
[New LWP 3820]
[New LWP 3821]
[New LWP 3822]
[New LWP 3846]
[New LWP 3847]
[New LWP 15183]
[New LWP 15184]
[New LWP 15292]
[New LWP 15293]
[New LWP 15294]
[New LWP 29030]
[New LWP 3015]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/opt/gbase/192.168.142.11/gcluster/server/bin/gclusterd'.
#0 0x00007fb03dfae3c1 in sigwait () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-54.el7_9.x86_64 libcom_err-1.42.9-19.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 nss-softokn-freebl-3.67.0-3.el7_9.x86_64 pcre-8.32-17.el7.x86_64 python-libs-2.7.5-92.el7_9.x86_64 sssd-client-1.16.5-10.el7_9.13.x86_64
(gdb)
Thread 50 (Thread 0x7fb03e3aebc0 (LWP 3015)):
#0 0x00007fb03916ab43 in select () from /lib64/libc.so.6
#1 0x00000000011c5290 in handle_connections_sockets ()
#2 0x00000000008ff578 in main ()
Thread 49 (Thread 0x7fafd711a700 (LWP 29030)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x25d24700) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 48 (Thread 0x7fafd3512700 (LWP 15294)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x25d26a00) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 47 (Thread 0x7fafd3913700 (LWP 15293)):
#0 0x00007fb03dfad75d in read () from /lib64/libpthread.so.0
#1 0x0000000000abd306 in ?? ()
#2 0x0000000000abe7d1 in my_net_read ()
#3 0x0000000000b14855 in do_command(THD*) ()
#4 0x0000000000ad0321 in handle_one_connection ()
#5 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 46 (Thread 0x7fafd4114700 (LWP 15292)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x25d25c00) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 45 (Thread 0x7fafd4915700 (LWP 15184)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x25d25500) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 44 (Thread 0x7fafdf356700 (LWP 15183)):
#0 0x00007fb03dfad75d in read () from /lib64/libpthread.so.0
#1 0x0000000000abd306 in ?? ()
#2 0x0000000000abe7d1 in my_net_read ()
#3 0x0000000000b14855 in do_command(THD*) ()
#4 0x0000000000ad0321 in handle_one_connection ()
#5 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 43 (Thread 0x7fafda922700 (LWP 3847)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x4e6aa00) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 42 (Thread 0x7fafd7d1c700 (LWP 3846)):
#0 0x00007fb03dfad75d in read () from /lib64/libpthread.so.0
#1 0x0000000000abd306 in ?? ()
#2 0x0000000000abe7d1 in my_net_read ()
#3 0x0000000000b14855 in do_command(THD*) ()
#4 0x0000000000ad0321 in handle_one_connection ()
#5 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 41 (Thread 0x7fafd5116700 (LWP 3822)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000000000f0a870 in ?? ()
#2 0x0000000000f1262e in CTableCacheScheduler::ThreadFunction(void*) ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 40 (Thread 0x7fafd5917700 (LWP 3821)):
#0 0x00007fb03916ab43 in select () from /lib64/libc.so.6
#1 0x0000000000f11a41 in TableAccessRecorder::FlushThread(void*) ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 39 (Thread 0x7fafd6919700 (LWP 3820)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x25d26300) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 38 (Thread 0x7fafd6118700 (LWP 3818)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00007fb03a2b9ac8 in GBCCS::Condition::tryWait (
this=this@entry=0x25d267d0, timeout=timeout@entry=0x7fafd6114650)
at concurrent.c:123
#2 0x00007fb03a2b9be2 in GBCCS::Condition::tryWait (
this=this@entry=0x25d267d0, ms=ms@entry=1000) at concurrent.c:119
#3 0x00007fb03a2b39b5 in MessageLinear::PopResponse (this=0x25d266b0,
buffer=std::vector of length 0, capacity 0, seq=1,
timeout=timeout@entry=1000) at coroipcc.c:1090
#4 0x00007fb03a2b3dcc in CConnection::ReceiveLinearsemanticCallBack (
this=<optimized out>, buffer=std::vector of length 0, capacity 0,
seq=<optimized out>, timeout=timeout@entry=1000) at coroipcc.c:1921
#5 0x00007fb03a2b3e8e in coroipcc_dispatch_get (handle=4804996157048619019,
data=0x7fafd61147c8, timeout=1000) at coroipcc.c:3432
#6 0x00007fb03a6f60a1 in gcCrmDispatch (crmHandle=2604745833635119111,
dispatchFlags=GC_DISPATCH_ALL) at crm.c:11084
#7 0x0000000000ee8356 in CMViewWatcher::WatcherThread(void*) ()
#8 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 37 (Thread 0x7fafd791b700 (LWP 3808)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x25d24000) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 36 (Thread 0x7fafd891e700 (LWP 3807)):
#0 0x00007fb03dfad75d in read () from /lib64/libpthread.so.0
#1 0x0000000000abd306 in ?? ()
#2 0x0000000000abe7d1 in my_net_read ()
#3 0x0000000000b14855 in do_command(THD*) ()
#4 0x0000000000ad0321 in handle_one_connection ()
#5 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fafd851d700 (LWP 3803)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x4e6b800) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fafd911f700 (LWP 3801)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000122594b in ExpressCond::TimedWait(long) ()
#2 0x00000000012829ca in HdfsTokenManager::RenewTokenLoop() ()
#3 0x0000000001282a81 in HdfsTokenManager::RenewThread(void*) ()
#4 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fafd9920700 (LWP 3799)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x4e6b100) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fafda121700 (LWP 3798)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00000000010434cb in ?? ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fafdb123700 (LWP 3797)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x0000000000f9f45e in ?? ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fafdc125700 (LWP 3795)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000000000f3ee47 in Detect_fn(void*) ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fafdc926700 (LWP 3794)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00000000011c05c6 in connection_status_monitor ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fafdcd27700 (LWP 3793)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000000000d6d85e in Event_queue::get_top_for_execution_if_time(THD*, Event_queue_element_for_exec**) ()
#2 0x0000000000d66c93 in Event_scheduler::run_internal(THD*) ()
#3 0x0000000000d66d66 in Event_scheduler::run(THD*) ()
#4 0x0000000000d65e90 in event_scheduler_thread ()
#5 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fafdd528700 (LWP 3792)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x0000000000d6737a in LeaderMonitor(void*) ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fafddd29700 (LWP 3791)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x0000000000d67bca in Event_gc_scheduler::run(THD*) ()
#3 0x0000000000d65e90 in event_scheduler_thread ()
#4 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fafde52a700 (LWP 3790)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x0000000000e7f73a in flush_fn ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fafe6dd6700 (LWP 3787)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00000000011dc617 in CDiskSpaceManager::FlushToUserTableThread(void*) ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fafe75d7700 (LWP 3763)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x2598ae00) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fafe7dd8700 (LWP 3762)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x2598a700) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fafe85d9700 (LWP 3719)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x2598a000) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fafe8dda700 (LWP 3718)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x39ef800) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fafe95db700 (LWP 3717)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x39ef100) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fafe9ddc700 (LWP 3714)):
#0 0x00007fb039168ddd in poll () from /lib64/libc.so.6
#1 0x00007fb03a2b6ad5 in CConnection::run (this=0x39eea00) at coroipcc.c:2037
#2 0x00007fb03a2b936a in mmThreadRun (p=<optimized out>) at concurrent.c:30
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fafea5dd700 (LWP 3664)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x000000000121c8dd in CpuUsageManager::run() ()
#3 0x0000000001239f7f in _run_thread ()
#4 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fafeeddf700 (LWP 3636)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00007fb03a2b9ac8 in GBCCS::Condition::tryWait (
this=this@entry=0x2598a4d0, timeout=timeout@entry=0x7fafeeddb670)
at concurrent.c:123
#2 0x00007fb03a2b9be2 in GBCCS::Condition::tryWait (
this=this@entry=0x2598a4d0, ms=ms@entry=1000) at concurrent.c:119
#3 0x00007fb03a2b39b5 in MessageLinear::PopResponse (this=0x2598a3b0,
buffer=std::vector of length 0, capacity 0, seq=1,
timeout=timeout@entry=1000) at coroipcc.c:1090
#4 0x00007fb03a2b3dcc in CConnection::ReceiveLinearsemanticCallBack (
this=<optimized out>, buffer=std::vector of length 0, capacity 0,
seq=<optimized out>, timeout=timeout@entry=1000) at coroipcc.c:1921
#5 0x00007fb03a2b3e8e in coroipcc_dispatch_get (handle=2030151274686578691,
data=0x7fafeeddb7d8, timeout=1000) at coroipcc.c:3432
#6 0x00007fb03a90017a in gcClmDispatch (clmHandle=6868983574935109633,
dispatchFlags=GC_DISPATCH_ALL) at clm.c:343
#7 0x0000000001dcd5eb in ?? ()
#8 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fafef5e0700 (LWP 3635)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00007fb03a2b9ac8 in GBCCS::Condition::tryWait (
this=this@entry=0x39ef5d0, timeout=timeout@entry=0x7fafef5dc410)
at concurrent.c:123
#2 0x00007fb03a2b9be2 in GBCCS::Condition::tryWait (
this=this@entry=0x39ef5d0, ms=ms@entry=1000) at concurrent.c:119
#3 0x00007fb03a2b39b5 in MessageLinear::PopResponse (this=0x39ef4b0,
buffer=std::vector of length 0, capacity 0, seq=1,
timeout=timeout@entry=1000) at coroipcc.c:1090
#4 0x00007fb03a2b3dcc in CConnection::ReceiveLinearsemanticCallBack (
this=<optimized out>, buffer=std::vector of length 0, capacity 0,
seq=<optimized out>, timeout=timeout@entry=1000) at coroipcc.c:1921
#5 0x00007fb03a2b3e8e in coroipcc_dispatch_get (handle=6192282104168972289,
data=0x7fafef5dc588, timeout=1000) at coroipcc.c:3432
#6 0x00007fb03a6f60a1 in gcCrmDispatch (crmHandle=1334614688700301312,
dispatchFlags=GC_DISPATCH_ALL) at crm.c:11084
#7 0x0000000001dcedc9 in ?? ()
#8 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fafefde1700 (LWP 3634)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00007fb03a2b9ac8 in GBCCS::Condition::tryWait (
this=this@entry=0x39efcd0, timeout=timeout@entry=0x7fafefddd870)
at concurrent.c:123
#2 0x00007fb03a2b9be2 in GBCCS::Condition::tryWait (
this=this@entry=0x39efcd0, ms=ms@entry=1000) at concurrent.c:119
#3 0x00007fb03a2b39b5 in MessageLinear::PopResponse (this=0x39efbb0,
buffer=std::vector of length 0, capacity 0, seq=1,
timeout=timeout@entry=1000) at coroipcc.c:1090
#4 0x00007fb03a2b3dcc in CConnection::ReceiveLinearsemanticCallBack (
this=<optimized out>, buffer=std::vector of length 0, capacity 0,
seq=<optimized out>, timeout=timeout@entry=1000) at coroipcc.c:1921
#5 0x00007fb03a2b3e8e in coroipcc_dispatch_get (handle=6566694452268630018,
data=0x7fafefddd9e8, timeout=1000) at coroipcc.c:3432
#6 0x00007fb03a4d32bd in gcLckDispatch (ullLckHandle=3562028287964217344,
ullDispatchFlags=GC_DISPATCH_ALL) at lck.c:513
#7 0x0000000001dcdbf1 in ?? ()
#8 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7faff05e2700 (LWP 3632)):
#0 0x00007fb03dfaaa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000123b04c in ExpressThreadInPool::run() ()
#2 0x0000000001239f7f in _run_thread ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7faff0de3700 (LWP 3630)):
#0 0x00007fb03dfaaa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000123b04c in ExpressThreadInPool::run() ()
#2 0x0000000001239f7f in _run_thread ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7faff15e4700 (LWP 3629)):
#0 0x00007fb03dfaaa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000123b04c in ExpressThreadInPool::run() ()
#2 0x0000000001239f7f in _run_thread ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7faff1de5700 (LWP 3628)):
#0 0x00007fb03dfaaa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000123b04c in ExpressThreadInPool::run() ()
#2 0x0000000001239f7f in _run_thread ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7faff25e6700 (LWP 3627)):
#0 0x00007fb03dfaaa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000123b04c in ExpressThreadInPool::run() ()
#2 0x0000000001239f7f in _run_thread ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7faff2de7700 (LWP 3626)):
#0 0x00007fb03dfaaa35 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000000000123b04c in ExpressThreadInPool::run() ()
#2 0x0000000001239f7f in _run_thread ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7faff35e8700 (LWP 3625)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x0000000001593312 in ExpressEngine::TrashManager::TrashManagerThread(void*) ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7faff5793700 (LWP 3405)):
#0 0x00007fb03916ab43 in select () from /lib64/libc.so.6
#1 0x000000000196769f in my_sleep ()
#2 0x0000000000e624f4 in HAEventHandler::HaLeftEventHandleThreadProc(void*) ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7faff5f94700 (LWP 3401)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03913a894 in sleep () from /lib64/libc.so.6
#2 0x0000000000e60f0a in HAEventHandler::HaEventMonitorThreadProc(void*) ()
#3 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7faff6395700 (LWP 3399)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03916b2d4 in usleep () from /lib64/libc.so.6
#2 0x0000000001225b7f in ExpressThread::SleepNow(int) ()
#3 0x0000000001572076 in locklist_manager(void*) ()
#4 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7faff6796700 (LWP 3371)):
#0 0x00007fb03913a9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007fb03916b2d4 in usleep () from /lib64/libc.so.6
#2 0x0000000001225b7f in ExpressThread::SleepNow(int) ()
#3 0x0000000001566017 in prioritylist_manager(void*) ()
#4 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb0381d3700 (LWP 3090)):
#0 0x00007fb03dfaade2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000000001076814 in async_detect_fn(void*) ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb0385d4700 (LWP 3087)):
#0 0x00007fb03dfae3c1 in sigwait () from /lib64/libpthread.so.0
#1 0x00000000011bff1f in signal_hand ()
#2 0x00007fb03dfa6ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb039173b0d in clone () from /lib64/libc.so.6
(gdb) quit
10、C3推文件
coor:表示推到管理节点,如果换成data就是推到数据节点。
haha.txt:表示你需要上传的文件。
/opt:表示你需要上传的目录。
[gbase@czg0 ~]$ cpush coor: haha.txt /opt
coor: haha.txt /opt
11、C3范围执行命令
例如我们的集群是1-40,但我们只想看38、39、40,就可以参考图片中的命令。