试题大部分来自网络刷题,牛客网,自己学习总结等,适合于有一些awk,grep,sed和Shell编程基础的同学阅读。。如有哪里写的不对,欢迎评论区提出~
目录
一、文本/文件处理
1、统计一个文本文件 nowcoder.txt中每一行出现的1,2,3,4,5数字个数并且要计算一下整个文档中一共出现了几个1,2,3,4,5数字数字总数。
假设 nowcoder.txt 内容如下:
a12b8
10ccc
2521abc
9asf
你的脚本应当输出:
line1 number: 2
line2 number: 1
line3 number: 4
line4 number: 0
sum is 7
#!/bin/bash
exec<nowcoder.txt
while read line;do
num=`echo $line | tr -dc [1-5] | wc -c`
let col+=1
echo 'line'$col 'number:'$num
let sum+=$num
#sum+=$num是不会做数学运算的,只是简单的数字拼接
done
echo "sum is" $sum
~
--------------------脚本测试-------------------------------
[root@node01 ~]#sh code1.sh
line1 number:2
line2 number:1
line3 number:4
line4 number:0
sum is 7
2、写一个 bash脚本以实现一个需求,去掉输入中含有this的语句,把不含this的语句输出
示例:
假设输入如下:
that is your bag
is this your bag?
to the degree or extent indicated.
there was a court case resulting from this incident
welcome to nowcoder
你的脚本获取以上输入应当输出that is your bag
to the degree or extent indicated.
welcome to nowcoder
$awk '!/this/ {print $0}' nowcoder.txt
$grep -v 'this'
$sed '/this/d' nowcoder.txt
$awk '$0!~/this/ {print $0}' nowcoder.txt
3、输出一个文本文件 nowcoder.txt 中第5行的内容。
示例:
假设 nowcoder.txt 内容如下:
welcome
to
nowcoder
this
is
shell
code
你的脚本应当输出:
is
$awk 'NR==5 {print $1}'
$sed -n '5,5p'
4、输出一个文本文件 nowcoder.txt中空行的行号,可能连续,从1开始
示例:
假设 nowcoder.txt 内容如下:a b c d e f你的脚本应当输出:
3
5
7
9
10
注:cat -n会显示出行号
$cat -n nowcoder.txt|awk '{if($2=="") {print $1}}'
5、去掉一个文本文件 nowcoder.txt中的空行
示例:
假设 nowcoder.txt 内容如下:abc 567 aaa bbb ccc
你的脚本应当输出:
abc
567
aaa
bbb
ccc
$sed '/^$/d'
$awk '{if(!/^$/)print $0}'
$grep -v '^$' #grep -v 不显示匹配的行
6、统计一个文本文件 nowcoder.txt中字母数小于8的单词。
示例:
假设 nowcoder.txt 内容如下:
how they are implemented and applied in computer
你的脚本应当输出:
how
they
areand
applied
in
$awk '{for(i=0;i<=NF;i++){if(length($i)<8) {print $i} }}'
7、实现一个需求,去掉输入中的含有B和b的单词
示例:
假设输入如下:
big
nowcoder
Betty
basic
test
你的脚本获取以上输入应当输出:
nowcoder test
$grep -v '[bB]'
$awk '$0!~/B|b/ {print $0}'
$sed '/[Bb]/d'
8、统计一个文本文件 nowcoder.txt 中每个单词出现的个数。
示例:
假设 nowcoder.txt 内容如下:
welcome nowcoder
welcome to nowcoder
nowcoder
你的脚本应当输出(以词频升序排列):
to 1
welcome 2
nowcoder 3
$awk '{for(i=1;i<=NF;i++)arr[$i]++}END{for(i in arr) print(i,arr[i])}'
9、判断第二列是否有重复
给定一个 nowcoder.txt文件,其中有3列信息,如下实例,编写一个shell脚本来检查文件第二列是否有重复,且有几个重复,并提取出重复的行的第二列信息:
实例:
20201001 python 99
20201002 go 80
20201002 c++ 88
20201003 php 77
20201001 go 88
20201005 shell 89
20201006 java 70
20201008 c 100
20201007 java 88
20201006 go 97
结果:
2 java
3 go
$cat nowcoder.txt |tr -d '[0-9]'|awk '{for(i=1;i<=NF;i++)arr[$i]++}END{for(i in arr)if(arr[i]>1)print(arr[i],i)}'
10、转置文本文件nowcoder.txt中的文件内容。
为了简单起见,你可以假设:
你可以假设每行列数相同,并且每个字段由空格分隔
示例:
假设 nowcoder.txt 内容如下:
job salary
c++ 13
java 14
php 12
你的脚本应当输出(以词频升序排列):
job c++ java php
salary 13 14 12
awk '{printf $1 ""}' nowcoder.txt
awk '{printf $2 ""}' nowcoder.txt
11、将字段逆序输出文件的每行
将字段逆序输出文件nowcoder.txt的每一行,其中每一字段都是用英文冒号: 相分隔。
假设nowcoder.txt内容如下:
nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
你的脚本应当输出
/usr/bin/false:/var/empty:Unprivileged User:-2:-2:*:nobody
/bin/sh:/var/root:System Administrator:0:0:*:root
while read line
do
echo $line| awk -F: '{
for (i=NF;i>0;i--){
if (i==1){
printf ($1 "\n"); break
}
printf ($i ":")
}
}'
done < nowcoder.txt
12、域名进行计数排序处理
假设我们有一些域名,存储在nowcoder.txt里,现在需要你写一个脚本,将域名取出并根据域名进行计数排序处理。
假设nowcoder.txt内容如下:
http://www.nowcoder.com/index.html
http://www.nowcoder.com/1.html
http://m.nowcoder.com/index.html
你的脚本应该输出:
1 m.nowcoder.com
这个问题其实就是先提取出字段再进行排序
[root@node01 ~]#awk -F"/" '{print $3}' nowcoder1.txt | awk '{for(i=1;i<=NF;i++)arr[$i]++}END{for(i in arr)print(arr[i],i)}'|sort -nr
2 www.nowcoder.com
1 m.nowcoder.com
13、打印只有一个数字的行
假设我们有一个nowcoder.txt,现在需要你写脚本,打印只有一个数字的行。
假设nowcoder.txt内容如下
haha
1
2ab
cd
77
那么你的脚本应该输出
1
2ab
awk -F "[0-9]" '{if(NF==2 )print $0}' nowcoder1.txt
14、格式化输出
我们有一个文件nowcoder.txt,里面的每一行都是一个数字串,假设数字串为“123456789”,那么我们要输出为123,456,789。
假设nowcoder.txt内容如下
1
12
123
1234
123456
那么你的脚本输出如下
1
12
123
1,234
123,456
xargs -n1:一次读取一个参数
cat nowcoder.txt | xargs -n1 printf "%'d\n" #给数字的千位加英文样式的分隔符,也就是逗号
15、将文本中奇数行和偶数行合并
4 ./.pyenv/plugins/python-build/test/stubs
4 ./.pyenv/plugins/python-build/test/tmp
128 ./.pyenv/plugins/python-build/test
4540 ./.pyenv/plugins/python-build
4544 ./.pyenv/plugins
12 ./.pyenv/pyenv.d/exec/pip-rehash
16 ./.pyenv/pyenv.d/exec
4 ./.pyenv/pyenv.d/install
8 ./.pyenv/pyenv.d/rehash/conda.d
8 ./.pyenv/pyenv.d/rehash/source.d
24 ./.pyenv/pyenv.d/rehash
44 ./.pyenv/pyenv.d
32 ./.pyenv/src
4 ./.pyenv/test/libexec
f1.txt
sed 高级用法中N:读下一行追加给模式空间,读取后进行替换
$sed '$!N;s/\n/ /g' f1.txt
4 ./.pyenv/plugins/python-build/test/stubs 4 ./.pyenv/plugins/python-build/test/tmp
128 ./.pyenv/plugins/python-build/test 4540 ./.pyenv/plugins/python-build
4544 ./.pyenv/plugins 12 ./.pyenv/pyenv.d/exec/pip-rehash
16 ./.pyenv/pyenv.d/exec 4 ./.pyenv/pyenv.d/install
8 ./.pyenv/pyenv.d/rehash/conda.d 8 ./.pyenv/pyenv.d/rehash/source.d
24 ./.pyenv/pyenv.d/rehash 44 ./.pyenv/pyenv.d
32 ./.pyenv/src 4 ./.pyenv/test/libexec
16、f1内的字符串翻转
cat f1.txt|rev
17、打印文件中的奇数\偶数号\指定行记录
sed :
常用选项:
-n 不自动打印
-e 多点编辑
编辑:
p 打印当前模式空间内容,追加至默认输出之后
d 删除模式空间匹配的行
=模式空间的行打印行号
!模式空间匹配的行取反处理
sed -n '1~2p' file.txt #奇数
sed -n '2~2p' file.txt #偶数
sed -n -e '2p' -e '6p' file.txt #显示第二、第六行
sed -n '2!d' file.txt #只显示第二行,除了第二行的都删掉
二、数字处理
1、写一个bash脚本以实现一个需求,求输入的一个的数组的平均值
第1行 输入的数组长度N
第2~N行 为数组的元素,如以下为:
数组长度为4,数组元素为1 2 9 8
示例:
4
1
2
9
8
那么平均值为:5.000(保留小数点后面3位)
你的脚本获取以上输入应当输出:
5.000
$awk '{if(NR==1) {N=$1} else{sum+=$1}} END{printf ("%.3f",sum/N)}'
这个题变一下,直接给出数组,求平均值,不给数组个数
2、统计所有进程占用内存大小的和
假设 nowcoder.txt 内容如下:
root 2 0.0 0.0 0 0 ? S 9月25 0:00 [kthreadd]
root 4 0.0 0.0 0 0 ? I< 9月25 0:00 [kworker/0:0H]
web 1638 1.8 1.8 6311352 612400 ? Sl 10月16 21:52 test
web 1639 2.0 1.8 6311352 612401 ? Sl 10月16 21:52 test
tangmiao-pc 5336 0.0 1.4 9100240 238544 ?? S 3:09下午 0:31.70 /Applications
以上内容是通过ps aux | grep -v 'RSS TTY' 命令输出到nowcoder.txt文件下面的
请你写一个脚本计算一下所有进程占用内存大小的和:
[root@node01 ~]#awk '{a+=$6} END{print(a)}' nowcoder.txt
1409900
3、ps分析-统计VSZ,RSS各自总和
假设命令运行的结果我们存储在nowcoder.txt里,格式如下:(这是ps aux命令得到的结果,其中VSZ,RSS代表的含义是:
VSZ:虚拟内存大小
RSS:resident size,常驻内存集,这个不可以放在交换分区,必须位于内存空间
)
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 37344 4604 ? Ss 2020 2:13 /sbin/init
root 231 0.0 1.5 166576 62740 ? Ss 2020 15:15 /lib/systemd/systemd-journald
root 237 0.0 0.0 0 0 ? S< 2020 2:06 [kworker/0:1H]
root 259 0.0 0.0 45004 3416 ? Ss 2020 0:25 /lib/systemd/systemd-udevd
root 476 0.0 0.0 0 0 ? S< 2020 0:00 [edac-poller]
root 588 0.0 0.0 276244 2072 ? Ssl 2020 9:49 /usr/lib/accountsservice/accounts-daemon
message+ 592 0.0 0.0 42904 3032 ? Ss 2020 0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 636 0.0 0.0 65532 3200 ? Ss 2020 1:51 /usr/sbin/sshd -D
daemon 637 0.0 0.0 26044 2076 ? Ss 2020 0:00 /usr/sbin/atd -f
root 639 0.0 0.0 29476 2696 ? Ss 2020 3:29 /usr/sbin/cron -f
root 643 0.0 0.0 20748 1992 ? Ss 2020 0:26 /lib/systemd/systemd-logind
syslog 645 0.0 0.0 260636 3024 ? Ssl 2020 3:17 /usr/sbin/rsyslogd -n
root 686 0.0 0.0 773124 2836 ? Ssl 2020 26:45 /usr/sbin/nscd
root 690 0.0 0.0 19472 252 ? Ss 2020 14:39 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
ntp 692 0.0 0.0 98204 776 ? Ss 2020 25:18 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:114
uuidd 767 0.0 0.0 28624 192 ? Ss 2020 0:00 /usr/sbin/uuidd --socket-activation
root 793 0.0 0.0 128812 3148 ? Ss 2020 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 794 0.0 0.2 133376 9120 ? S 2020 630:57 nginx: worker process
www-data 795 0.0 0.2 133208 8968 ? S 2020 633:02 nginx: worker process
www-data 796 0.0 0.2 133216 9120 ? S 2020 634:24 nginx: worker process
www-data 797 0.0 0.2 133228 9148 ? S 2020 632:56 nginx: worker process
web 955 0.0 0.0 36856 2112 ? Ss 2020 0:00 /lib/systemd/systemd --user
web 956 0.0 0.0 67456 1684 ? S 2020 0:00 (sd-pam)
root 1354 0.0 0.0 8172 440 tty1 Ss+ 2020 0:00 /sbin/agetty --noclear tty1 linux
root 1355 0.0 0.0 7988 344 ttyS0 Ss+ 2020 0:00 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220
root 2513 0.0 0.0 0 0 ? S 13:07 0:00 [kworker/u4:1]
root 2587 0.0 0.0 0 0 ? S 13:13 0:00 [kworker/u4:2]
root 2642 0.0 0.0 0 0 ? S 13:17 0:00 [kworker/1:0]
root 2679 0.0 0.0 0 0 ? S 13:19 0:00 [kworker/u4:0]
root 2735 0.0 0.1 102256 7252 ? Ss 13:24 0:00 sshd: web [priv]
web 2752 0.0 0.0 102256 3452 ? R 13:24 0:00 sshd: web@pts/0
web 2753 0.5 0.1 14716 4708 pts/0 Ss 13:24 0:00 -bash
web 2767 0.0 0.0 29596 1456 pts/0 R+ 13:24 0:00 ps aux
root 10634 0.0 0.0 0 0 ? S Nov16 0:00 [kworker/0:0]
root 16585 0.0 0.0 0 0 ? S< 2020 0:00 [bioset]
root 19526 0.0 0.0 0 0 ? S Nov16 0:00 [kworker/1:1]
root 28460 0.0 0.0 0 0 ? S Nov15 0:03 [kworker/0:2]
root 30685 0.0 0.0 36644 2760 ? Ss 2020 0:00 /lib/systemd/systemd --user
root 30692 0.0 0.0 67224 1664 ? S 2020 0:00 (sd-pam)
root 32689 0.0 0.0 47740 2100 ? Ss 2020 0:00 /usr/local/ilogtail/ilogtail
root 32691 0.2 0.5 256144 23708 ? Sl 2020 1151:31 /usr/local/ilogtail/ilogtail
现在需要你统计VSZ,RSS各自的总和(以M兆为统计),输出格式如下
MEM TOTAL
VSZ_SUM:3250.8M,RSS_SUM:179.777M
[root@node01 ~]#awk '{if(NR!=1)sum1+=$5;sum2+=$6}END{printf("MEM TOTAL\nVSZ_SUM:%0.1fM,RSS_SUM:%0.3fM",sum1/1024,sum2/1024)}' nowcoder1.txt
MEM TOTAL
VSZ_SUM:3250.8M,RSS_SUM:179.777M
使用 printf来控制输出格式,灵活方便
4、计算100以内可以被3整除的整数之和
#!/bin/bash
for i in {1..100..3};do
let sum+=i
done
echo sum=$sum
unset sum
5、 统计平均成绩
[root@node01 ~]#cat score.txt
name score sex
lili 90 f
wang 92 m
lura 100 f
jack 91 m
mike 90 m
john 88 f
[root@node01 ~]#awk '{if(NR!=1)num[$NF]++;sum[$NF]+=$2}END{printf "male:%.2f\nfemale:%.2f\n",sum["m"]/num["m"],sum["f"]/num["f"]}' score.txt
male:91.00
female:92.67
三、日志分析
1、nginx日志分析2-统计某个时间段的IP
假设nginx的日志我们存储在nowcoder.txt里,格式如下:
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在你需要统计2020年04月23日20-23点的去重IP访问量,你的脚本应该输出
5
$grep -r '23/Apr/2020:2[0-3]' |cut -c "-12"|uniq|wc -l
$grep -r '23/Apr/2020:2[0-3]' |cut -c "-12"|sort -c|wc -l #sort -c 等于uniq
2、nginx日志分析1-IP统计
假设nginx的日志我们存储在nowcoder.txt里,格式如下:
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要你统计出2020年4月23号的访问ip次数,并且按照次数降序排序。你的脚本应该输出:
5 192.168.1.22
4 192.168.1.21
3 192.168.1.20
2 192.168.1.25
1 192.168.1.24
awk '{if(substr($4,2,11)=="23/Apr/2020") arr[$1]++;}END{for(k in arr){print arr[k] " " k}}' nowcoder2.txt |sort -r
3、nginx日志分析3-统计访问3次以上的IP
假设nginx的日志我们存储在nowcoder.txt里,格式如下:
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要你写脚本统计访问3次以上的IP,你的脚本应该输出
6 192.168.1.22
5 192.168.1.21
4 192.168.1.20
[root@node01 ~]#awk '{arr[$1]++}END{for(k in arr){if(arr[k]>3)print(arr[k],k)}}' nowcoder2.txt | sort -r
6 192.168.1.22
5 192.168.1.21
4 192.168.1.20
4、nginx日志分析4-查询某个IP的详细访问情况
假设nginx的日志我们存储在nowcoder.txt里,格式如下:
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要你查询192.168.1.22的详细访问情况,按访问频率降序排序。你的脚本应该输出
4 /1/index.php
2 /3/index.php
[root@node01 ~]#awk '{if($1=="192.168.1.22")print $7}' nowcoder2.txt | awk '{for(i=1;i<=NF;i++)arr[$i]++}END{for(k in arr)print(arr[k],k)}'|sort -r
4 /1/index.php
2 /3/index.php
5、nginx日志分析5-统计爬虫抓取404的次数
假设nginx的日志我们存储在nowcoder.txt里,格式如下:
192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 301 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台)"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台)"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台)"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 300 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要你统计百度爬虫抓取404的次数,你的脚本应该输出
2
[root@node01 ~]#awk '{if($9==404 && $14=="Baiduspider/2.0;")arr[$14]++}END{for(k in arr)print(arr[k])}' nowcoder2.txt
2
6、nginx日志分析6-统计每分钟的请求数
假设nginx的日志我们存储在nowcoder.txt里,格式如下:
192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Baiduspider"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Baiduspider"
192.168.1.25 - - [23/Apr/2020:16:15:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要你统计每分钟的请求数,并且按照请求数降序排序。你的脚本应该输出
5 20:27
4 15:00
2 22:10
2 14:12
2 10:27
1 23:59
1 21:21
1 16:15
1 15:26
1 09:20
1 08:05
使用substr,先提出来时间字段
[root@node01 ~]#awk '{arr[substr($4,14,5)]++}END{for(k in arr)print(arr[k],k)}' nowcoder2.txt | sort -nr
5 20:27
4 15:00
2 22:10
2 14:12
2 10:27
1 23:59
1 21:21
1 16:15
1 15:26
1 09:20
1 08:05
四、网络有关问题
1、netstat练习1-查看各个状态的连接数
假设netstat命令运行的结果我们存储在nowcoder.txt里,格式如下:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6160 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 172.16.56.200:41856 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49822 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49674 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:42316 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:44076 172.16.240.74:6379 ESTABLISHED
tcp 0 0 172.16.56.200:49656 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58248 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:50108 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41944 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:35548 100.100.32.118:80 TIME_WAIT
tcp 0 0 172.16.56.200:39024 100.100.45.106:443 TIME_WAIT
tcp 0 0 172.16.56.200:41788 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58260 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:41812 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41854 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58252 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:49586 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41754 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50466 120.55.222.235:80 TIME_WAIT
tcp 0 0 172.16.56.200:38514 100.100.142.5:80 TIME_WAIT
tcp 0 0 172.16.56.200:49832 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:52162 100.100.30.25:80 ESTABLISHED
tcp 0 0 172.16.56.200:50372 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50306 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49600 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41908 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:60292 100.100.142.1:80 TIME_WAIT
tcp 0 0 172.16.56.200:37650 100.100.54.133:80 TIME_WAIT
tcp 0 0 172.16.56.200:41938 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49736 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41890 172.16.34.144:3306 ESTABLISHED
udp 0 0 127.0.0.1:323 0.0.0.0:*
udp 0 0 0.0.0.0:45881 0.0.0.0:*
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 172.16.56.200:68 0.0.0.0:*
udp6 0 0 ::1:323 :::*
raw6 0 0 :::58 :::* 7
现在需要你查看系统tcp连接中各个状态的连接数,并且按照连接数降序输出。你的脚本应该输出如下:
ESTABLISHED 22
TIME_WAIT 9
LISTEN 3
[root@node01 ~]#awk '{if($6 != "" && NR!= 1 && $6 ~ /EST|TIME|LIST/)arr[$6]++}END{for(k in arr)print(k,arr[k])}' nowcoder2.txt |sort -nr -k2
ESTABLISHED 22
TIME_WAIT 9
LISTEN 3
2、netstat练习2-查看和3306端口建立的连接
假设netstat命令运行的结果我们存储在nowcoder.txt里,内容和第一题很像,但有一些不同,具体如下:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6160 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 172.16.56.200:41856 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49822 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49674 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:42316 172.16.34.143:3306 ESTABLISHED
tcp 0 0 172.16.56.200:44076 172.16.240.74:6379 ESTABLISHED
tcp 0 0 172.16.56.200:49656 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58248 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:50108 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41944 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:35548 100.100.32.118:80 TIME_WAIT
tcp 0 0 172.16.56.200:39024 100.100.45.106:443 TIME_WAIT
tcp 0 0 172.16.56.200:41788 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58260 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:41812 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41854 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58252 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:49586 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41754 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50466 120.55.222.235:80 TIME_WAIT
tcp 0 0 172.16.56.200:38514 100.100.142.5:80 TIME_WAIT
tcp 0 0 172.16.56.200:49832 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:52162 100.100.30.25:80 ESTABLISHED
tcp 0 0 172.16.56.200:50372 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50306 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49600 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41908 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:60292 100.100.142.1:80 TIME_WAIT
tcp 0 0 172.16.56.200:37650 100.100.54.133:80 TIME_WAIT
tcp 0 0 172.16.56.200:41938 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49736 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41890 172.16.34.144:3306 ESTABLISHED
udp 0 0 127.0.0.1:323 0.0.0.0:*
udp 0 0 0.0.0.0:45881 0.0.0.0:*
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 172.16.56.200:68 0.0.0.0:*
udp6 0 0 ::1:323 :::*
现在需要你查看和本机3306端口建立连接并且状态是established的所有IP,按照连接数降序排序。你的脚本应该输出
10 172.16.0.24
9 172.16.34.144
1 172.16.34.143
[root@node01 ~]#awk '{if(NR!=1 && $6="ESTABLISHED" && $6 != "" && $5 ~ /3306/)arr[$5]++}END{for(k in arr)print(arr[k],k)}' nowcoder2.txt |awk -F":" '{print $1}'|sort -nr
10 172.16.0.24
9 172.16.34.144
1 172.16.34.143
3、netstat练习3-输出每个IP的连接数
假设netstat命令运行的结果我们存储在nowcoder.txt里,格式如下:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6160 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 172.16.56.200:41856 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49822 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49674 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:42316 172.16.34.143:3306 ESTABLISHED
tcp 0 0 172.16.56.200:44076 172.16.240.74:6379 ESTABLISHED
tcp 0 0 172.16.56.200:49656 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58248 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:50108 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41944 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:35548 100.100.32.118:80 TIME_WAIT
tcp 0 0 172.16.56.200:39024 100.100.45.106:443 TIME_WAIT
tcp 0 0 172.16.56.200:41788 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58260 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:41812 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41854 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58252 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:49586 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41754 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50466 120.55.222.235:80 TIME_WAIT
tcp 0 0 172.16.56.200:38514 100.100.142.5:80 TIME_WAIT
tcp 0 0 172.16.56.200:49832 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:52162 100.100.30.25:80 ESTABLISHED
tcp 0 0 172.16.56.200:50372 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50306 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49600 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41908 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:60292 100.100.142.1:80 TIME_WAIT
tcp 0 0 172.16.56.200:37650 100.100.54.133:80 TIME_WAIT
tcp 0 0 172.16.56.200:41938 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49736 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41890 172.16.34.144:3306 ESTABLISHED
udp 0 0 127.0.0.1:323 0.0.0.0:*
udp 0 0 0.0.0.0:45881 0.0.0.0:*
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 172.16.56.200:68 0.0.0.0:*
udp6 0 0 ::1:323 :::*
raw6 0 0 :::58 :::* 7
现在需要你输出每个IP的连接数,按照连接数降序排序。你的脚本应该输出
172.16.0.24 10
172.16.34.144 9
100.100.142.4 3
0.0.0.0 3
172.16.34.143 1
172.16.240.74 1
120.55.222.235 1
100.100.54.133 1
100.100.45.106 1
100.100.32.118 1
100.100.30.25 1
100.100.142.5 1
100.100.142.1 1
[root@node01 ~]#awk '{if($1=="tcp")print($5)}' nowcoder2.txt | sed -r 's/(.*):(.*)/\1/'|awk '{for(i=1;i<=NF;i++)arr[$i]++}END{for(k in arr)print(k,arr[k])}'|sort -nr -k2
172.16.0.24 10
172.16.34.144 9
100.100.142.4 3
0.0.0.0 3
172.16.34.143 1
172.16.240.74 1
120.55.222.235 1
100.100.54.133 1
100.100.45.106 1
100.100.32.118 1
100.100.30.25 1
100.100.142.5 1
100.100.142.1 1
4、netstat练习4-输出和3306端口建立连接总的各个状态的数目
假设netstat命令运行的结果我们存储在nowcoder.txt里,格式如下:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6160 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 172.16.56.200:41856 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49822 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49674 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:42316 172.16.34.143:3306 ESTABLISHED
tcp 0 0 172.16.56.200:44076 172.16.240.74:6379 ESTABLISHED
tcp 0 0 172.16.56.200:49656 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58248 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:50108 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41944 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:35548 100.100.32.118:80 TIME_WAIT
tcp 0 0 172.16.56.200:39024 100.100.45.106:443 TIME_WAIT
tcp 0 0 172.16.56.200:41788 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58260 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:41812 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41854 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:58252 100.100.142.4:80 TIME_WAIT
tcp 0 0 172.16.56.200:49586 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41754 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50466 120.55.222.235:80 TIME_WAIT
tcp 0 0 172.16.56.200:38514 100.100.142.5:80 TIME_WAIT
tcp 0 0 172.16.56.200:49832 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:52162 100.100.30.25:80 ESTABLISHED
tcp 0 0 172.16.56.200:50372 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:50306 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49600 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41908 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:60292 100.100.142.1:80 TIME_WAIT
tcp 0 0 172.16.56.200:37650 100.100.54.133:80 TIME_WAIT
tcp 0 0 172.16.56.200:41938 172.16.34.144:3306 ESTABLISHED
tcp 0 0 172.16.56.200:49736 172.16.0.24:3306 ESTABLISHED
tcp 0 0 172.16.56.200:41890 172.16.34.144:3306 ESTABLISHED
udp 0 0 127.0.0.1:323 0.0.0.0:*
udp 0 0 0.0.0.0:45881 0.0.0.0:*
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 172.16.56.200:68 0.0.0.0:*
udp6 0 0 ::1:323 :::*
raw6 0 0 :::58 :::* 7
现在需要你输出和本机3306端口建立连接的各个状态的数目,按照以下格式输出
TOTAL_IP表示建立连接的ip数目
TOTAL_LINK表示建立连接的总数目
TOTAL_IP 3
ESTABLISHED 20
TOTAL_LINK 20
#!/bin/bash
awk '{
if($1=="tcp" && $5 ~ /3306$/){
if($6=="ESTABLISHED"){
es++}
ans++
arr[$5]++ }
}END{
printf("TOTAL_IP %d\nESTABLISHED %d\nTOTAL_LINK %d\n",length(arr),es,ans)}' nowcoder2.txt
~
[root@node01 ~]#sh answer1.sh
TOTAL_IP 3
ESTABLISHED 20
TOTAL_LINK 20
5、编写脚本,提示输入网络地址,如192.168.0.0,判断当前网段内主机在线个数
#!/bin/bash
read -p "please input your network address:" NETADDR
up=0
down=0
net=`echo $NETADDR|cut -d. -f 1-3`
echo $net
#扫描的是1-253内所有的ip
for i in {1..253};do
ping -c1 -w1 $net.$i &>/dev/null && { echo "$net.$i is UP" >> uphostlist.txt ; let ++up; }||{ echo %net.$i is DOWN &> /dev/null ; let ++down; }
done
wait
echo uphost is $up
echo downhost is $down
························································
[root@node01 ~]#sh ip.sh
please input your network address:192.168.74.0
192.168.74
uphost is 2
downhost is 251
6、在access.log 内给出了网络连接ip和连接次数,对于连接次数大于1000的ip,加入到iptables内
```````````````````````````先取出每个ip的次数``````````````````````````````````````
[root@node01 ~]#awk '{ip[$1]++}END{for(i in ip)print i,ip[i]}' access.log 172.20.111.64 1
192.168.30.1 1280
172.20.111.65 1
172.20.1.24 1
172.20.111.9 919
172.20.61.7 61
::1 654
172.20.111.88 1
172.20.110.34 1
```````````````````````````对超过1000连接数的ip进行限制`````````````````````
[root@node01 ~]#for i in `awk '{ip[$1]++}END{for(i in ip){if(ip[i]>1000)print i}}' access.log`;do iptables -A INPUT -s $i -j REJECT;done
[root@node01 ~]#iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
REJECT all -- 192.168.30.1 anywhere reject-with icmp-port-unreachable
五、业务分析
1、业务分析-提取值
假设我们的日志nowcoder.txt里,内容如下
12-May-2017 10:02:22.789 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:Apache Tomcat/8.5.15
12-May-2017 10:02:22.813 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:May 5 2017 11:03:04 UTC
12-May-2017 10:02:22.813 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:8.5.15.0
12-May-2017 10:02:22.814 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:Windows, OS Version:10
12-May-2017 10:02:22.814 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:x86_64
现在需要你提取出对应的值,输出内容如下
serverVersion:Apache Tomcat/8.5.15
serverName:8.5.15.0
osName:Windows
osVersion:10
[root@node01 ~]#awk -F"[:,]" '{if($0~"Server version"){print "serverVersion:" $4;}if($0~"Server number"){print "serverName:" $4;}if($0~"OS Name"){print "osName:" $4;}if($0~"OS Version"){print "osVersion:" $6}}' nowcoder1.txt
serverVersion: Apache Tomcat/8.5.15
serverName:8.5.15.0
osName:Windows
osVersion:10
六、其他类型
1、判断输入的是否为IP地址
写一个脚本统计文件nowcoder.txt中的每一行是否是正确的IP地址。
如果是正确的IP地址输出:yes
如果是错误的IP地址,四段号码的话输出:no,否则的话输出:error
假设nowcoder.txt内容如下
192.168.1.1
192.168.1.0
300.0.0.0
123
你的脚本应该输出
yes
yes
no
error
分支选择,分成几个条件做判断然后给出结果
awk -F"." '{
if(NF!=4)print "error";
else {
msg="yes"
for(i=1;i<=4;i++){
if($i<0 || $i>255){
msg="no";break
}
}print msg;
}
}'
注意此题,如果使用for来遍历每一行的域的话,会出现重复打印,如下图所示:前几次都是在循环打印,最后一次才输出error
解决此情况,可以使用awk的next功能
awk -F'.' '{if(NF!=4){print "error";next}else{for(i=1;i<=4;i++){if(i==4){print "yes";next}else{if($i<0||$i>255){print "no";next}}}}}' nowcoder.txt
2、求10个随机数的最大最小值
#!/bin/bash
for ((i=0;i<10;i++));do
rand[$i]=$RANDOM
if [ $i -eq 0 ];then
max=${rand[$i]}
min=$max
else
[ $max -lt ${rand[$i]} ]&& { max=${rand[$i]};continue; }
[ $min -gt ${rand[$i]} ]&& min=${rand[$i]}
fi
done
echo ALL rand is ${rand[*]}
echo MAX rand is $max
echo MIN rand is $min
·······································································
[root@node01 ~]#sh random.sh
ALL rand is 29885 9654 26686 16478 29176 21546 17697 10676 27301 225
MAX rand is 29885
MIN rand is 225
3、取出分区利用率
如下,取出df命令得到的分区利用率,并在利用率超过40%时候报警
[root@node01 ~]#df
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 915680 0 915680 0% /dev
tmpfs 931516 0 931516 0% /dev/shm
tmpfs 931516 92308 839208 10% /run
tmpfs 931516 0 931516 0% /sys/fs/cgroup
/dev/sda2 48803552 11905220 36898332 25% /
/dev/sda1 1038336 187684 850652 19% /boot
/dev/sda3 29282336 33052 29249284 1% /data
tmpfs 186304 20 186284 1% /run/user/0
/dev/sr0 9961428 9961428 0 100% /run/media/root/CentOS 7 x86_64
3
#!/bin/bash
df | while read disk;do
if echo $disk | grep "^/dev/sd" > /dev/null;then
used=`echo $disk | sed -r 's/.* ([0-9]+)%.*/\1/'`
device=`echo $disk|cut -d" " -f1`
[ $used -ge 20 ]&&echo "$device will be full.used:$used%"
fi
done
········································································
[root@node01 ~]#sh disk.sh
/dev/sda2 will be full.used:25%
4、添加10个新用户,密码为123456
#!/bin/bash
for I in {1..10};do
if id user$I &> /dev/null;then
echo "user$I is already exist"
else
useradd user$I &>/dev/null
echo "123456" | passwd --stdin user$I &> /dev/null
echo "useradd$I is successful"
fi
done
`````````````````````````````````````````````````````
[root@node01 ~]#sh user.sh
useradd1 is successful
useradd2 is successful
useradd3 is successful
useradd4 is successful
useradd5 is successful
useradd6 is successful
useradd7 is successful
useradd8 is successful
useradd9 is successful
useradd10 is successful
``````````````````````````````````````````````````````
批量删除用户:
[root@node01 ~]#for i in {1..10};do userdel user$i;done
5、将/usr/local/test目录下的大于100K的文件转移到/tmp目录下
#!/bin/bash
for file in `ls /usr/local/test`
do
if [ -f $file ];then
if [ `ls -l $file|awk '{print $5}'` -gt 100 ];then
mv $file /tmp/
fi
fi
done
6、统计网站次数
http://a.domin.com/1.html
http://b.domin.com/1.html
http://a.domin.com/1.html
http://b.domin.com/1.html
http://a.domin.com/1.html
http://b.domin.com/1.html
http://a.domin.com/1.html
http://a.domin.com/1.html
cat list awk -F"/" '{arr[$3]++}END{for(i in arr)print(i,arr[i])}'|sort -r
7、编写shell程序,实现自动删除50个账号的功能,账号名为stud1-stud50
#!/bin/bash
i=1
while( $i -le 50)
do
userdel -r stud${i}
i=$(($i+1))
done
8、编写Shell,添加一个新组为class1,然后添加属于这个组的30个用户,用户名形式为stdxx,其中,xx范围为01-30
#!/bin/bash
i=1
groupadd class1
while [ $i -le 30]
do
if[ $i -le 9];then
USERNAME=stu0$i
else
USERNAME=stu$i
fi
useradd $USERNAME
mkdir /home/$USERNAME
chown -R $USERNAME /home/$USERNAME
chgrp -R class1 /home/$USERNAME
i=$(($i+1))
done
9、设计一个shell程序,在/userdata目录下建立50个目录,user1-user50,并设置每个目录的权限,其中其他用户的权限为r,文件所有者权限为rwx,文件所有者所在组的权限为rx
#!/bin/bash
for i in {1..50};do
if [ -d /userdata ];then
mkdir -p /userdata/user$i
chmod 754 /userdata/user$i
echo "$user$i is created!"
else:
mkdir /userdata
mkdir -p /userdata/user$i
chmod 754 /userdata/user$i
echo "$user$i is created!"
fi
done
crontab题目
1、某系统管理员每天需要做一些重复工作,按以下要求,编写一个解决方案:
下午4:50删除/abc目录下的全部子目录和全部文件
每天早上八点到下午6点,每隔一个小时,将/xyz下的x1文件的第一列数据备份至backup下的bak01.txt文件内
每天下午5点50分,将data下的所有目录和文件打包为backup.tar.gz
下午5点55分将IDE接口的CD-ROM卸载,假设,设备名为hdc
将以下crontab写入文件prgx
50 16 * * * rm -rf /abc/*
* 8-18/1 * * * awk '{print $1}' /xyz/x1 > /backup/bak01.txt
50 17 * * 1 tar zcvf backup.tar.gz /data/*
55 17 * * * umount /dev/hdc
以超级用户的身份登陆,并执行
crontab prgx
2、在每日早晨8点之前开机后即可自动启动crontab
每月第一天备份并压缩/etc下所有内容,存放在/root/bak下,且文件名为yymmdd_etc.shell程序filebak存放在/usr/bin目录下
-----1.编写shell程序fileback--------------
#!/bin/bash
DIRNAME=`ls /root|grep bak`
#判断bak目录是否为空,若为空,则先创建一个目录
if [ -z "$DIRNAME" ];then
mkdir /root/bak
cd /root/bak
fi
YY=`date+%y`
MM=`date+%m`
DD=`date+%d`
BACKETC=$YY$MM$DD_etc.tar.gz
tar zcvf $BACKETC /etc
echo "fileback finished"
-------2.编写并执行定时任务-----------------
echo "* * 1 * * /bin/sh /usr/bin/fileback >;/root/etcbakcron"
crontab /root/etcbakcron
3、有一个普通用户想在每周日凌晨零点零分定期备份/usr/backup到/tmp目录下,应该如何做
#create crontab task
crontab -e "0 0 * * Sun cp -r /usr/backup /tmp"