shell编程之sed&awk

一、sed

sed的用途：行编辑器，一次处理文本中的一行。（常用于查找替换）

sed的语法：

sed 命令选项  '动作'  文件 （动作的引号是一定要有的，经常会出现"//"表开启匹配模式） 

sed常用的内部命令	sed的命令选项
a\：在匹配后面添加	-n：只显示自己指定的内容
i\：在匹配前面添加	-e：编辑多个条件(;)
p：打印	-f：指定命令文件
d：删除	-i：要求修改原文件（此动作不可逆）
s：查找替换	-r：加入正则
c\：行替换
y：字符转换
g：新文本替换现有文本的全部实例
p：表示打印原始内容
w 文件夹：将替换的结果写入文件夹

• sed常用的内部命令的练习：

[root@localhost ~]# cat test       #原文本
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy dog
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

① a\：在匹配后面添加（追加）

[root@localhost ~]# sed '/3 the/ a\hello world' test   

# 双斜杠匹配==>/3 the/ ==> 即第三行。 在第三行后面追加（a\）hello world  文件（test）
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy dog
hello world
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

②  i\：在匹配前面添加

[root@localhost ~]# sed '/3 the/ i\hello world' test
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
hello world
3 the quick brown fox jumps over the lazy dog
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

③ d：删除

 [root@localhost ~]# sed '/2 the/d' test     #把第二行删了
1 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy dog
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

④ s：查找替换

[root@localhost ~]# sed 's/dog/cat/' test
1 the quick brown fox jumps over the lazy cat
2 the quick brown fox jumps over the lazy cat
3 the quick brown fox jumps over the lazy cat
4 the quick brown fox jumps over the lazy cat
5 the quick brown fox jumps over the lazy cat
6 the quick brown fox jumps over the lazy cat

⑤ c\：行替换

[root@localhost ~]# sed '/3 the/ c\hello world' test   #将第三行替换成hello world
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
hello world
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

⑥ y：字符转换

[root@localhost ~]# sed 'y/abcdef/ABCDEF/' test   #a对A，b对B，字符与字符之间一一对应
1 thE quiCk Brown Fox jumps ovEr thE lAzy Dog
2 thE quiCk Brown Fox jumps ovEr thE lAzy Dog
3 thE quiCk Brown Fox jumps ovEr thE lAzy Dog
4 thE quiCk Brown Fox jumps ovEr thE lAzy Dog
5 thE quiCk Brown Fox jumps ovEr thE lAzy Dog
6 thE quiCk Brown Fox jumps ovEr thE lAzy Dog

⑦ p：打印

 [root@localhost ~]# sed '/3 the/p' test    #打印第三行
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy dog
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

[root@localhost ~]# sed '3s/dog/cat/p' test    # 将第三行的dog替换成cat，并打印一行
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy cat
3 the quick brown fox jumps over the lazy cat
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog

⑧ g：新文本替换现有文本的全部实例 

[root@localhost ~]# sed 's/dog/cat/g' test
1 the quick brown fox jumps over the lazy cat
2 the quick brown fox jumps over the lazy cat
3 the quick brown fox jumps over the lazy cat
4 the quick brown fox jumps over the lazy cat
5 the quick brown fox jumps over the lazy cat
6 the quick brown fox jumps over the lazy cat

⑨  w 文件夹：将替换的结果写入文件夹

[root@localhost ~]# sed '2s/dog/cat/w mfile' test   

 #将第二行的dog替换成cat，并将替换结果输出到mfile文件夹上
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy cat
3 the quick brown fox jumps over the lazy dog
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog
 

[root@localhost ~]# cat mfile   #查看mfile文件夹的结果
2 the quick brown fox jumps over the lazy cat

• sed命令选项的练习： 

-n：只显示自己指定的内容

[root@localhost ~]# sed -n '3p' test         #只打印文本中的第3行
3 the quick brown fox jumps over the lazy dog

 -e：编辑多个条件(;)

[root@localhost ~]# sed -e 's/brown/red/;s/dog/cat/' test
1 the quick red fox jumps over the lazy cat
2 the quick red fox jumps over the lazy cat
3 the quick red fox jumps over the lazy cat
4 the quick red fox jumps over the lazy cat
5 the quick red fox jumps over the lazy cat
6 the quick red fox jumps over the lazy cat

 -f：指定命令文件

[root@localhost ~]# vim mfile  #写的是命令文件
s/brown/red/
s/dog/cat/

[root@localhost ~]# sed -f mfile test
1 the quick red fox jumps over the lazy cat
2 the quick red fox jumps over the lazy cat
3 the quick red fox jumps over the lazy cat
4 the quick red fox jumps over the lazy cat
5 the quick red fox jumps over the lazy cat
6 the quick red fox jumps over the lazy cat

-i：要求修改原文件（此动作不可逆）

[root@localhost ~]# sed -i 's/dog/cat/g' test   # -i 修改并不会直接显示 （修改后不可逆）
[root@localhost ~]# cat test
1 the quick brown fox jumps over the lazy cat
2 the quick brown fox jumps over the lazy cat
3 the quick brown fox jumps over the lazy cat
4 the quick brown fox jumps over the lazy cat
5 the quick brown fox jumps over the lazy cat
6 the quick brown fox jumps over the lazy cat

-r：加入正则

[root@localhost ~]# cat selinux.bak      #原文本
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:

#     enforcing - SELinux security policy is enforced.
#

#     permissive - SELinux prints warnings instead of enforcing.
     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
     targeted - Targeted processes are protected,
     minimum - Modification of targeted policy. Only selected processes are protected. 
     mls - Multi Level Security protection.
SELINUXTYPE=targeted 
 

[root@localhost ~]# sed -r '/(^#|^$)/d' selinux.bak    #删除以#开头，以及空行（^$）
     disabled - No SELinux policy is loaded.
SELINUX=enforcing
     targeted - Targeted processes are protected,
     minimum - Modification of targeted policy. Only selected processes are protected. 
     mls - Multi Level Security protection.
SELINUXTYPE=targeted 

注：sed常用：

1、统计文本的行数

[root@localhost ~]# sed -n '$=' selinux.bak   # $统计文本有多少行
16

2、管道符对行替换

[root@localhost ~]# echo Amy is cool | sed 's/Amy/Jane/'
Jane is cool

---

二、awk

awk的用途：擅长取列

awk的语法：

awk  [options]  '{program}'  file ...

• program: [BEGIN]{action statements;...}[END]

program：必选项，如何处理数据流。

BEGIN和END：BEGIN处理数据流前执行，END处理数据流后执行

action statements：对数据进行处理，放在{ }内指明print,printf

 awk的常用选项：

-F：指定分隔符

awk的常用情况：

1、awk对字段（列）进行提取 ==> 打印列

提取字段（列）：提取一个文本中的一列数据并打印输出

$0：整行文本

$1：文本行中的第一列

$2：文本行中的第二列

$N：文本行中的第n列

$NF：文本行中最后一列

[root@localhost ~]# awk '{print $0}' test   #整个文本都打印
1 the quick brown fox jumps over the lazy dog
2 the quick brown fox jumps over the lazy dog
3 the quick brown fox jumps over the lazy dog
4 the quick brown fox jumps over the lazy dog
5 the quick brown fox jumps over the lazy dog
6 the quick brown fox jumps over the lazy dog
[root@localhost ~]# awk '{print $1}' test   #打印第一列
1
2
3
4
5
6
[root@localhost ~]# awk '{print $2}' test    #打印第二列
the
the
the
the
the
the
[root@localhost ~]# awk '{print $NF}' test    #打印第最后一列
dog
dog
dog
dog
dog
dog

2、 awk对记录（行）的提取  ==> 打印行

NR：指定行号

[root@localhost ~]# awk 'NR==3{print $0}' test
3 the quick brown fox jumps over the lazy dog
 

[root@localhost ~]# sed -n '3p' test    #用sed的方式
3 the quick brown fox jumps over the lazy dog

3、综合练习 

[root@localhost ~]# awk -F ":" 'NR==1{print $1,$3,$5}' /etc/passwd   

# 以:为分隔，输出第一行的第1列，第3列和第5列
root 0 root

[root@localhost ~]# awk -F ":" 'NR==1{print $1 "-" $3 "-" $5}' /etc/passwd

#可以根据自己的需要加上相应的东西
root-0-root

[root@localhost ~]# awk -F ":" 'NR==1{print "account:" $1,"UID:"$3,"DESC:"$5}' /etc/passwd
account:root UID:0 DESC:root