出现这个问题-bash: [: : integer expression expected，没有找到真正原因是什么，因为从监控没看到来自外部的攻击，下面的脚本已经验证过，也发现什么异常，但实地问题就出现了.ls、cat、vi、ll等命令全都执行不了
跟踪发现环境变量中文件被删掉了
如果同一个目录都是空的，还可以理解，但是/usr/bin目录下并不是所有都是空的
云厂商工程师，先将系统盘卸载掉，挂靠到没有问题的服务器上，进行跟进。到了新服务器上，这个磁盘就是数据盘了，里面很多东西就可以读写了。的确是个好的办法，之前没接触过，确实是个很好的经验。
由于主机重启进程和链接未见异常，对主机下面的目录进行检查，这个也是个很好的经验，可以看出高级运维工程师的工作思路
crontab -l
var/spool/cron
/etc/init.d
/bin
/usr/bin
/sbin
/etc/password
/usr/local/bin/
/root
/tmp
/var/tmp
文件检查，未见异常，查看主机日志未见异常
bin目录文件被修改时间是15号11:21:24。history没有该时段操作记录，无法判断是如何被修改的
工程师的建议方案是安装atop和audit，进行监控。当下情况只能这样，发现问题并不可怕，因为他会让你成长，你不可能天生什么都会，那么从比你厉害的人身上可以学到丰富的经验。
1 atop
yum install -y atop安装很是简单,这里参考Linux atop 监控系统状态对atop命令的结果做一些说明
它的数据采集主要包括：CPU、内存、磁盘、网络、进程等，并且内容非常的详细，特别是当那一部分存在压力它会以特殊的颜色进行展示，如果颜色是红色那么说明已经非常严重了
当前情况，一切正常，就没什么可看的，以后有问题再追加。
[root@sp5 ~]# atop
ATOP - sp5 2021/01/16 12:22:28 ----x--------- 7d16h41m46s elapsed
PRC | sys 55m35s | user 10h20m | | #proc 147 | #trun 1 | #tslpi 273 | | #tslpu 0 | #zombie 0 | clones 104e5 | | #exit 0 |
CPU | sys 2% | user 7% | irq 0% | idle 790% | wait 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu007 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu001 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu003 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu004 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu000 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu005 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu006 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu002 w 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.60GHz | curscal ?% |
CPL | avg1 0.46 | avg5 0.32 | | avg15 0.27 | | | csw 466495e3 | intr 46034e4 | | | numcpu 8 | |
MEM | tot 31.3G | free 22.1G | cache 6.5G | dirty 0.4M | buff 199.7M | slab 327.3M | slrec 291.5M | shmem 16.6M | shrss 0.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 2.8G | vmlim 15.6G |
LVM | gdata-lvData | busy 0% | read 269 | | write 42349 | KiB/r 21 | KiB/w 79 | MBr/s 0.0 | MBw/s 0.0 | | avq 108.14 | avio 0.33 ms |
DSK | vda | busy 0% | read 16139 | | write 1088e3 | KiB/r 25 | KiB/w 16 | MBr/s 0.0 | MBw/s 0.0 | | avq 4.25 | avio 0.67 ms |
DSK | vdb | busy 0% | read 535 | | write 8394 | KiB/r 21 | KiB/w 400 | MBr/s 0.0 | MBw/s 0.0 | | avq 7.81 | avio 1.60 ms |
NET | transport | tcpi 37938e3 | tcpo 52709e3 | udpi 135222 | udpo 135233 | tcpao 320192 | tcppo 1953 | tcprs 36385 | tcpie 59 | tcpor 2598 | udpnp 10 | udpie 0 |
NET | network | ipi 39069675 | ipo 53290068 | | ipfrw 0 | deliv 3907e4 | | | | | icmpi 996440 | icmpo 8 |
NET | eth0 ---- | pcki 39367e3 | pcko 53292e3 | sp 0 Mbps | si 165 Kbps | so 133 Kbps | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | lo ---- | pcki 7452 | pcko 7452 | sp 0 Mbps | si 0 Kbps | so 0 Kbps | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
*** system and process activity since boot ***
PID SYSCPU USRCPU VGROW RGROW RDDSK WRDSK RUID EUID ST EXC THR S CPUNR CPU CMD 1/5
21902 7m58s 7h47m 616.6M 61012K 0K 576.8M root root N- - 5 S 4 4% python3.8
4692 1m50s 91m00s 404.1M 69204K 0K 707.3M root root N- - 2 S 0 1% python3.8
1552 17m12s 14m39s 1.2G 13440K 1144K 51072K root root N- - 17 S 2 0% hostguard
1151 48.10s 12m49s 581.6M 99560K 0K 349.2M root root N- - 4 S 4 0% python3.8
999 5m33s 6m25s 11.1G 200.6M 16544K 190.1M root root N- - 24 S 5 0% java
746 4m16s 2m08s 119.3M 1436K 8K 40K root root N- - 2 S 2 0% wrapper
9 5m32s 21.45s 0K 0K 0K 0K root root N- - 1 S 7 0% rcu_sched
2407 22.51s 4m41s 575.2M 93052K 0K 29448K root root N- - 4 S 6 0% python3.8
24587 12.62s 3m02s 549.3M 67576K 0K 33244K root root N- - 4 S 0 0% python3.8
486 2m07s 59.96s 55532K 1080K 72K 1.6G root root N- - 2 S 1 0% auditd
658 76.05s 66.91s 1.5G 21852K 7984K 62260K root root N- - 24 S 4 0% uniagent
28779 17.82s 1m55s 293.4M 46480K 0K 1.3G root root N- - 1 S 6 0% scrapyd
148 1m54s 0.00s 0K 0K 0K 0K root root N- - 1 S 2 0% kauditd
3422 11.92s 85.06s 560.7M 77796K 0K 12444K root root N- - 4 S 2 0% python3.8
1856 11.21s 80.11s 558.3M 75352K 0K 11084K root root N- - 4 S 2 0% python3.8
984 11.01s 72.93s 549.7M 67780K 0K 13836K root root N- - 4 S 1 0% python3.8
10347 12.02s 67.66s 636.0M 84040K 0K 10916K root root N- - 5 S 6 0% python3.8
23080 12.03s 67.47s 549.6M 67460K 0K 11516K root root N- - 4 S 0 0% python3.8
22214 11.51s 60.57s 633.1M 78388K 0K 11796K root root N- - 5 S 4 0% python3.8
24172 11.13s 60.22s 548.4M 64368K 0K 10940K root root N- - 4 S 4 0% python3.8
4237 11.17s 60.06s 556.1M 72568K 0K 11820K root root N- - 4 S 1 0% python3.8
1550 31.73s 38.57s 49044K 2052K 8K 2216K root root N- - 1 S 4 0% hostguard
32598 11.07s 57.56s 550.1M 68408K 0K 10956K root root N- - 4 S 0 0% python3.8
2627 11.20s 55.45s 550.7M 67152K 0K 10424K root root N- - 4 S 6 0% python3.8
4344 11.06s 55.03s 550.1M 66776K 0K 10476K root root N- - 4 S 2 0% python3.8
743 9.28s 48.73s 560.8M 17524K 7828K 16K root root N- - 5 S 2 0% tuned
833 16.23s 11.98s 221.4M 8136K 840K 5296K root root N- - 3 S 0 0% rsyslogd
652 21.28s 4.48s 21672K 1252K 104K 0K root root N- - 1 S 2 0% irqbalance
319 20.73s 0.00s 0K 0K 0K 2.3G root root N- - 1 S 6 0% jbd2/vda1-8
33 20.09s 0.00s 0K 0K 0K 0K root root N- - 1 S 5 0% migration/5
23 20.02s 0.00s 0K 0K 0K 0K root root N- - 1 S 3 0% migration/3
43 19.99s 0.00s 0K 0K 0K 0K root root N- - 1 S 7 0% migration/7
13 19.82s 0.00s 0K 0K 0K 0K root root N- - 1 S 1 0% migration/1
2 audit
因为遇到过命令没法用的，故而先将重要的目录做一下审计，这样可以跟踪什么被删除了。
使用 auditd 监控目录变化
# 一开始，你可以看到，什么规则都没有
[root@sp1 ~]# auditctl -l
No rules
# 这个目录下面有很多常规的命令，添加到审计中
auditctl -w /usr/bin -p wxa -k watch_usr_bin
auditctl -w /usr/sbin -p wxa -k wath_usr_sbin
# 查看规则添加进去了
[root@sp1 sbin]# auditctl -l
-w /usr/bin -p wxa -k watch_usr_bin
-w /usr/sbin -p wxa -k wath_usr_sbin
# 查看日志
ausearch -k watch_usr_bin
ausearch -k wath_usr_sbin
#
扫一扫
2073
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
