文本处理三剑客 sed 、awk 、grep

文本处理三剑客

sed、awk、grep

文本处理三剑客之 sed

sed 基于行的过滤和转换文本的流编辑器

使用sed命令不跟上-i它只是打印输出并不是真的改里面内容，sed命令默认就是打印。

-n, --quiet, --silent  取消自动打印模式空间，一般-n和p是在一起用
 -e 脚本, --expression=脚本   添加“脚本”到程序的运行列表
 -f 脚本文件, --file=脚本文件  添加“脚本文件”到程序的运行列表
 -l N, --line-length=N   指定“l”命令的换行期望长度
 -r, --regexp-extended  在脚本中使用扩展正则表达式
 -a ∶追加。 a 的后面可以接字串，而这些字串会在新的一行出现(目前的下一行)
 -c ∶更改。 c 的后面可以接字串，这些字串可以更改 n1,n2 之间的行！
 -i ∶插入。 i 的后面可以接字串，而这些字串会在新的一行出现(目前的上一行)；
 -p ∶列印。
s 表示替换
g 便是全局
d 是删除
& 用正则表达式匹配的内容进行替换
h 或 H 将模式空间的内容复制或追加到保持空间
g 或 G 将保持空间的内容复制 或追加到模式空间

模式空间：进行文本处理。
保持空间：把数据临时存放在这里

注意：在这下面写的不加 -i 就单单只是把它打印出来，不会真的做任何修改动作。

替换 一般s跟g在一起用

[root@localhost ~]# cat test
Every man
has his own purpose.
In this world,
there are
people who love me,
there are people
who hate me,



打印文本第二行
使用行号
[root@localhost ~]# sed -n '2p' test
has his own purpose.

精确匹配
[root@localhost ~]# sed -n '/has his own purpose./p' test
has his own purpose.


打印第二行两遍，不加-n，-n是取消自动打印，p是打印一遍，然sed命令默认就是打印。所以是两遍
[root@localhost ~]#  sed  '/has his own purpose./p' test
Every man
has his own purpose.
has his own purpose.
In this world,
there are
people who love me,
there are people
who hate me,


使用行号也是会打印两遍
[root@localhost ~]#  sed '2p' test
Every man
has his own purpose.
has his own purpose.
In this world,
there are
people who love me,
there are people
who hate me,

最近数据
[root@localhost ~]# cat test
Every man
has his own has
has this world,
there are has
has who has me,
there are people
who hate me,


使用替换s 、g全局
把has改为hello  ，这里注意不加-i 是打印，不会真的修改里面的has  
[root@localhost ~]# sed 's/has/hello/' test
Every man
hello his own has
hello this world,
there are hello
hello who has me,
there are people
who hate me,
# 注意不跟g的情况下，只修改第一行。

[root@localhost ~]# sed 's/has/hello/g' test
Every man
hello his own hello
hello this world,
there are hello
hello who hello me,
there are people
who hate me,

改第二个为hello
[root@localhost ~]# sed 's/has/hello/2' test
Every man
has his own hello
has this world,
there are has
has who hello me,
there are people
who hate me,

把第二行删除
[root@localhost ~]# sed '2d' test
Every man
has this world,
there are has
has who has me,
there are people
who hate me,

一般用精确查找
[root@localhost ~]# sed '/has his own has has/d' test
Every man
has this world,
there are has
has who has me,
there are people
who hate me,

右斜杠表示转义 \
[root@localhost ~]# echo '/usr/local/src/' | sed 's/\/usr\/local\/src/\/user\/local\/src/'
/user/local/src/

如果需要用右斜杠做转义，就用#号代替，！感叹号也可以
[root@localhost ~]# echo '/usr/local/src/' | sed 's#/usr/local/src#/user/local/src#'
/user/local/src/
[root@localhost ~]# echo '/usr/local/src/' | sed 's!/usr/local/src!/user/local/src!'
/user/local/src/


& 用正则表达式匹配的内容进行替换
有这样一个文本如果我想写123到basedir=123和456到datadir=456。怎么弄?
[root@localhost ~]# cat test
# If you change base dir,
# overwritten by settings

basedir=
datadir=

先把它打印出来看一下是否是你想要的结果，然后再进行修改
[root@localhost ~]# sed -n 's/^basedir=/&123/p' test
basedir=123

使用-i，不加-i只是打印，不会真的修改。^表示行首
[root@localhost ~]# sed -i 's/^basedir=/&123/g' test
[root@localhost ~]# cat test
# If you change base dir,
# overwritten by settings

basedir=123
datadir=

有这样一个文本，
[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc. # 如果想把ORA Associates, Inc. 修改为O'Reilly ORA Associates, Inc. 怎么弄呢？

-r 正则表达式 。(.*) 表示 Associates, Inc.
[root@localhost ~]# sed -r "s/ORA (.*)/O'Reilly &/g" test
.Ah "Major Heading"
O'Reilly ORA Associates, Inc.

// 在这里的& 就会匹配ORA Associates, Inc. 

在ORA 后面添加123。你想一下之前我们部署lamp的时候需要把index.php 写到index.html的前面，就可以用这种方式。
[root@localhost ~]# sed  "s/ORA/& 123/g" test
.Ah "Major Heading"
ORA 123 Associates, Inc.

在这一整行ORA Associates, Inc.添加123
[root@localhost ~]# sed  "s/ORA.*/& 123/g" test
.Ah "Major Heading"
ORA Associates, Inc. 123

使用正则表达式 -r (扩展正则，不需要用右斜杠转义)
[root@localhost ~]# sed -r "s/ORA (.*)/& 123/g" test
.Ah "Major Heading"
ORA Associates, Inc. 123

基本正则，需要用右斜杠转义。
[root@localhost ~]# sed  "s/ORA \(.*\)/& 123/g" test
.Ah "Major Heading"
ORA Associates, Inc. 123

\t 表示tab，把第二个使用tab的修改为>。2表示第二个tab出来的。把2改为g就全部替换为>
[root@localhost ~]# cat test1
hello1	hello2	hello3
[root@localhost ~]# sed 's/\t/>/2' test1
hello1	hello2>hello3

\n 换行
[root@localhost ~]# sed 's/\t/\n/2' test1
hello1	hello2
hello3	hello4 hello5



有这样一个文本.我想改为这样 @A HEAD = Major Heading
[root@localhost ~]# cat test
.Ah "Major Heading"

[root@localhost ~]# sed '/^\.Ah/{
> s/\.Ah */\ # 先匹配.Ah 
> \  # 再换行
> @A HEAD = / # 把.Ah 改为 @A HEAD
> s/"//g # 把 " 双引号改为空
> s/$/\ # $ 表示行尾  。 \ 换行。上面一行下面一行就是两行
> /}' test

@A HEAD = Major Heading

[root@localhost ~]# 

[root@localhost ~]# sed '/^\.Ah/{s/\.Ah */\n@A HEAD = /;s/"//g};s/$/\n/' test

@A HEAD = Major Heading

[root@localhost ~]#


有这样一个文本，
要求，on the UNIX Operating System. 把它改为on the \s-2UNIX\s0 Operating System. 怎么弄？
[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.


在\s-2的前面不是有一个右斜杠吗哪个就是转义。这种需要慢慢品，就跟酒一样得需要慢慢品，才会品出好酒。
[root@localhost ~]# sed 's/UNIX/\\s-2&\\s0/g' test
.Ah "Major Heading"
ORA Associates, Inc.
on the \s-2UNIX\s0 Operating System.


文本这样子
要求：(See Section 1.4)和(See Section 12.9)
[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9

成果
[root@localhost ~]# sed 's/See Section [1-9][0-9]*\.[1-9][0-9]*/(&)/g' test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
(See Section 1.4)
(See Section 12.9)

还是刚才哪个文本要求：
See Section\fB1.4\fB
See Section\fB12.9\fB


[root@localhost ~]# sed -r 's/(See Section) ([1-9][0-9]*\.[1-9][0-9]*)/\1\\fB\2\\fB/g' test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section\fB1.4\fB
See Section\fB12.9\fB

文本内容
[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
first:second
one:two
// 要求把它变成翻转过来
second:first
two:one

成果
[root@localhost ~]# sed -r 's/(.*):(.*)/\2:\1/g' test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
second:first
two:one

追加

 a ∶新增。 a 的后面可以接字串，而这些字串会在新的一行出现(目前的下一行)
 c ∶取代。 c 的后面可以接字串，这些字串可以取代 n1,n2 之间的行
 i ∶插入。 i 的后面可以接字串，而这些字串会在新的一行出现(目前的上一行)；

文本内容
[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
first:second
one:two
需求：在.Ah "Major Heading"行的下一行追加hello

右斜杠在这转义
[root@localhost ~]# sed '/.Ah "Major Heading"/a \hello' test
.Ah "Major Heading"
hello
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
first:second
one:two

如果你需要hello有空格就这样做
[root@localhost ~]# sed '/.Ah "Major Heading"/ a \    hello' test
.Ah "Major Heading"
    hello
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
first:second
one:two


文本内容
[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
See Section 13.6
first:second
one:two
需求：
在这See Section 12.9和See Section 13.6两行的下面加123。

怎么做？
 *     //匹配其前面的任意单个字符任意次
[root@localhost ~]# sed '/See.*[1-9][0-9]\.[0-9]/a 123' test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
123
See Section 13.6
123
first:second
one:two

插入、更改

[root@localhost ~]# cat test
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
See Section 13.6
first:second
one:two
需求：
在.Ah "Major Heading"这一行的前面加上123
\ 转义
[root@localhost ~]# sed '/\.Ah "Major Heading"/i \123' test
123
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
....省略N

还是刚才哪个文本
把.Ah "Major Heading"修改为hello
[root@localhost ~]# sed '/.Ah "Major Heading"/c \hello' test
hello
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
See Section 13.6
first:second
one:two

.Ah "Major Heading"的前面插入两行hello跟haha。\n 换行
[root@localhost ~]# sed '/.Ah "Major Heading"/i hello\nhaha' test
hello
haha
.Ah "Major Heading"
ORA Associates, Inc.
on the UNIX Operating System.
See Section 1.4
See Section 12.9
See Section 13.6
first:second
one:two


还是刚刚的文本
替换第一行到第三行为123
[root@localhost ~]# sed '/\.Ah/,/on/c 123' test
123
See Section 1.4
See Section 12.9
See Section 13.6
first:second
one:two

扩展

文本内容
[root@localhost ~]# cat test1
.Ah "hello world"
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello meimei"
.Ah "hello hehe"
需求就是把前面的.Ah不要把后面的打印到下一行


[root@localhost ~]# sed '/\.Ah/{p;s/"//g;s/\.Ah //}' test1
.Ah "hello world"
hello world
.Ah "hello tom"
hello tom
.Ah "hello jerry"
hello jerry
.Ah "hello meimei"
hello meimei
.Ah "hello hehe"
hello hehe


打印行号
[root@localhost ~]# sed  -n '/hello world/{=;p}' test1
1
.Ah "hello world"
它在第一行

tom在第二行
[root@localhost ~]# sed  -n '/hello tom/{=;p}' test1
2
.Ah "hello tom"


这里的n是打印从hello world的下一行打印
[root@localhost ~]# sed -n  '/hello world/n;p' test1
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello meimei"
.Ah "hello hehe"


文本内容
[root@localhost ~]# cat test1
.Ah "hello world"
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello world"
.Ah "hello meimei"
.Ah "hello world"
.Ah "hello hehe"

需求要在hello world的下面添加.Ah ()
[root@localhost ~]# sed -r '/hello world/{n;s/\.Ah (.*)/\.Ah (\1)/g}' test1
.Ah "hello world"
.Ah ("hello tom")
.Ah "hello jerry"
.Ah "hello world"
.Ah ("hello meimei")
.Ah "hello world"
.Ah ("hello hehe")
[root@localhost ~]#


文本内容
[root@localhost ~]# cat test1
.Ah "hello world"
                     <-----这个是删了的
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello world"
                     <-----这个是删了的
.Ah "hello meimei"
.Ah "hello world"
.Ah "hello hehe"     <-----这个没有删因为没有空行

需求就是hello world 下面有空行则删除，没有不删
[root@localhost ~]# sed '/hello world/{n;/^$/d}' test1
.Ah "hello world"
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello world"
.Ah "hello meimei"
.Ah "hello world"
.Ah "hello hehe"



把上面删除空行的结果写到一个文本里
[root@localhost ~]# sed -n '/hello world/{n;/^$/d};p' test1
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello meimei"
.Ah "hello hehe"

[root@localhost ~]# ls
1  anaconda-ks.cfg  mysqld  test  test1  test10
[root@localhost ~]# cat test10
.Ah "hello tom"
.Ah "hello jerry"
.Ah "hello meimei"
.Ah "hello hehe"

N 用法

文本内容
[root@localhost ~]# cat test
Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.
[root@localhost ~]#

下面的脚本寻找行结尾处的“Operator”,读取下一个输入行，然后进行替换。

这种方式是换行
[root@localhost ~]# sed -n '/Operator$/{N;s/Owner and Operator\nGuide/instation\nGuide/g;p}' test
Consult Section 3.1 in the instation
Guide for a description of the tape drives

这种方式没有换行
[root@localhost ~]# sed -n '/Operator$/{N;s/Owner and Operator\nGuide/instation Guide/g;p}' test
Consult Section 3.1 in the instation Guide for a description of the tape drives


文本内容
[root@apache ~]# cat abc 
Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
avai lable on your system.
Look in the Owner and Operator Guide shipped with your system.
Two manuals are provided inc luding the Owner and
Operator Guide and the User Guide.
The Owner and Operator Guide is shipped with your system.



在再执行这条命令时先一步一步的取
[root@apache ~]# cat test 
s/Owner and Operator Guide/ Installation Guide/  # 先执行这个，出结果之后再之下面的
/Owner/ {
N
s/ *\n/ /
s/Owner and Operator Guide */Instal lation Guide\
/
}



执行 文本
[root@apache ~]# sed -f test abc 
Consult Section 3.1 in the Instal lation Guide
for a description of the tape drives
avai lable on your system.
Look in the  Installation Guide shipped with your system.
Two manuals are provided inc luding the Instal lation Guide
and the User Guide.
The  Installation Guide is shipped with your system.

D：多行删除

[root@apache ~]# cat abc 
hello world

hello world


hello world



hello world




hello world
[root@apache ~]# sed '/^$/{N;/^\n$/d}' abc      
hello world

hello world
hello world

hello world
hello world
[root@apache ~]# sed '/^$/{N;/^\n$/D}' abc  
hello world

hello world

hello world

hello world

hello world

P:多行打印

[root@apache ~]# cat abc 
Here are examples of the UNIX 
System.  Where UNIX
System appears，it should be the UNIX
Operat ing System.
[root@apache ~]# cat test 
/UNIX$/ {
N
/\nSystem/ {
s// Operating &/
P
D
}
}
[root@apache ~]# sed -f test abc 
Here are examples of the UNIX 
System.  Where UNIX Operating 
System appears，it should be the UNIX
Operat ing System.

文本三剑客之 awk

awk 基于列的文本报告工具

默认以空格、换行符、制表符作为分隔符，使用-F可以指定分隔符

有这样一个文本
[root@localhost ~]# cat passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[root@localhost ~]#

打印第一列跟最后一列，它们使用冒号做分隔符，默认打印空格
[root@localhost ~]# awk -F: '{print $1,$7}' passwd|head -3
root /bin/bash
bin /sbin/nologin
daemon /sbin/nologin

跟上面的对比一下
[root@localhost ~]# awk -F: '{print $1":"$7}' passwd|head -3
root:/bin/bash
bin:/sbin/nologin
daemon:/sbin/nologin
    
 //awk默认是用空格做分隔符的
[root@localhost ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             874M     0  874M   0% /dev
tmpfs                893M     0  893M   0% /dev/shm
tmpfs                893M  8.8M  884M   1% /run
tmpfs                893M     0  893M   0% /sys/fs/cgroup
/dev/mapper/cs-root   17G  2.9G   15G  17% /
/dev/sda1           1014M  210M  805M  21% /boot
tmpfs                179M     0  179M   0% /run/user/0
//打印第四列
[root@localhost ~]# df -h | awk '{print $4}' 
Avail
874M
893M
884M
893M
15G
805M
179M
 

 
 
//打印第四列，第六行   （行用NR表示）
[root@localhost ~]# df -h|awk 'NR==6{print $4}'
15G

//打印最后一列                （最后一列用NF表示）
[root@localhost ~]# df -h|awk '{print $NF}'
on
/dev
/dev/shm
/run
/sys/fs/cgroup
/
/boot
/run/user/0
[root@localhost ~]# 

//打印倒数第二列
[root@localhost ~]# df -h|awk '{print $(NF-1)}'
Mounted
0%
0%
1%
0%
19%
21%
0%

//打印倒数第二列，第一行不要
[root@localhost ~]# df -h|awk 'NR!=1{print $(NF-1)}'
0%
0%
1%
0%
19%
21%
0%




有这样一个文本
[root@localhost ~]# cat test
Hello,World


打印Hello，World
[root@localhost ~]# awk '{print "Hello,World"}' test
Hello,World

文本内容
[root@localhost ~]# cat test
John Robinson 666-555-1111

要求：结果要这样子 Robinson John 666-555-1111
[root@localhost ~]# awk '{print $2,$1,$3}' test
Robinson John 666-555-1111



//文本处理三剑客
sed    针对行 编辑文件内容
awk    针对列
grep   过滤

文本处理三剑客之 grep

根据模式搜索文本，并将符合模式的文本行显示出来。
使用基本正则表达式定义的模式来过滤文本的命令。

-i          //忽略大小写
--color     //匹配到的内容高亮显示
-v          //显示没有被模式匹配到的行
-o          //只显示被模式匹配到的字符串
-E          //使用扩展正则表达式。grep -E相当于使用egrep
-q          //静默模式，不输出任何信息
-A 1        //被模式匹配到的内容以及其后面一行的内容都显示出来，\
            //如果把1改成2就表示被模式匹配到的内容以及其后面2行的内容均显示出来
            
-B 1        //被模式匹配到的内容以及其前面一行的内容都显示出来， \
            //如果把1改成2就表示被模式匹配到的内容以及其前面2行的内容均显示出来
            
-C 1        //被模式匹配到的内容以及其前后的行各显示1行，如果把1改成2 \
            //就表示被模式匹配到的内容以及其前后的行各显示2行。

# 文本内容
[root@localhost ~]# cat  abc 
hello world
helo world
helol world
helol runtime
hello runtime
HELLO world

取出带有 hello 的关键字

取出带有 hello 的关键字、不区分大小写

-v //显示没有被模式匹配到的行(取反)

-o //只显示被模式匹配到的字符串

-q //静默模式，不输出任何信息

-E //使用扩展正则表达式。grep -E相当于使用egrep

文本内容

取出helllo和hellol

取出空行

除空行之外的都取出来

-A 1 //被模式匹配到的内容以及其后面一行的内容都显示出来

-B 1 //被模式匹配到的内容以及其前面一行的内容都显示出来

-C 1 //被模式匹配到的内容以及其前后的行各显示1行