linux--shell sed

sed简介

#grep/find  sed awk
sed是一款流编辑工具
针对文本进行过滤，替换操作（vim）
在大批量场景下相当实用

文本来自于本地文件，标准输入（包括键盘输入，文件重定向，字符串，变量，管道文件）

sed工作流程

• 文件------>(模式空间)sed指令集--------->显示屏幕/重定向输出

• sed通过一次仅读一行内容来对某些指令进行处理后输出（更适合于大数据文件）

• 首先，sed通过文件或管道读取文件内容，但 sed不会直接修改源文件，而是将数据读入的内容复制到缓冲区，这部分缓冲区称之为模式空空间(pattern)，所有的指令操作都在模式空间进行，然后sed根据相应的指定对模式空间内容进行相应输出，默认情况下为标准输入(打印到屏幕)

sed使用场景

• 编辑相对交互式文本编辑器而言太大的文件
• 编辑命令过于复杂，在交互式文本编辑器中那一输入的情况
• 对文件扫描一遍，但是需要执行多个编辑函数的情况

sed基本语法

• sed从文件中读取数据，若没有文件，则默认标准输入进行处理

• 语法

格式： sed [选项]....｛脚本指令｝[输入文件] ...
选项：
    --version    显示sed版本信息
    --help        显示帮助信息
    -n    --quiet, --silent   静默输出，默认情况下sed程序在所有的脚本指令            执行完毕后，将自动打印模式空间的内容，-n屏蔽自动打印
    -e script   允许多个脚本指令呗执行    
    -f  script-file    从文件中读取脚本指令，
    -i[SUFFIX], --in-place[=SUFFIX]  将修改内容直接修改为源文件（谨慎使用）    
    -r  在脚本中使用扩展正则表达式
    -s, --separate  默认情况小爱，sed将输入的多个文件作为一个长的连续的输入流，而GNUsed则把他们当作单个文件

2.将sed命令插入脚本文件后通过sed命令调用
        
    格式：sed [选项] -f script-file 输入文件

3.将sed命令插入到脚本文件，让该脚本可执行，直接执行该脚本文件
    ./sed脚本文件 [输入文件]


注意：第三种方式的脚本文件与第二种方式有所不同，第三种脚本文件需要以sha-bang(#!)符号开头，sha-bang后面跟脚本的程序名

若没有指定输入文件，sed将从标准输入中接收，
    -n    不打印所有行到标准输出
    -e 允许多个脚本指令呗执行
    -f    从文件中读取脚本指令

sed简单案例

• sed命令由定位文本行和sed编辑命令俩部分组成

sed编辑命令对定位文本进行操作，sed定位方式主要有2种

• 行号，指定一行或者多行范围
• 使用正则表达式

sed命令定位文本方法：

x    x指定行号
x,y    指定x-y行范围
/pattern/    查看包含的行
/pattern/pattern/    查看包含俩个匹配的行
/pattern/,x        从paeertn包含行到x行之间的行
x,/pattern/        从x行到pattern包含行
x,y!    不包含x，y行

sed编辑方法

• 主要为增删改查打印

p    打印匹配的行
=    打印文件行号
a\    在定位行号之后追加文本信息
i\    在定位行号之前插入文件信息
d    删除定位符
c\    替换定位文本
s    使用替换模式替换相应模式
r    从另一个文本中读入文本
w    将文本写到一个文件
y    变换字符
q    第一个模式匹配完成后退出
g    将保持缓冲区的内容复制到模式缓冲区
G    将保持缓冲去的内容追加到模式缓冲区

sed案例

#源文件
[root@localhost ~]# cp  /etc/sysconfig/network-scripts/ifcfg-eno16777728 test.txt

#第二行后追加TYPE=Ethernet
[root@localhost ~]# sed '2a TYPE=Ethernet' test.txt
#第三行前插入TYPE=Ethernet
sed '3i TYPE=Ethernet' test.txt
#将文本中所有的yes替换为no
[root@localhost ~]# sed 's/yes/no/g' test.txt
#删除第三四行的内容
[root@localhost ~]# sed  '3,4d' test.txt

sed命令集练习

• 案例文件

[root@localhost ~]# cat quote.txt
The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.

使用print显示行

[root@localhost ~]# sed '2p' quote.txt 
The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.

[root@localhost ~]# sed -n '2p' quote.txt 
It was an evening of splendid music and company. 

打印范围

number：指定输入文件的唯一行号
number1，number2：'1,3p' 匹配number1到number2支间的所有行
first~step：指定从first行开始，每个step几行打印
$:代表最后一行
addr1，+N：匹配addr1以及后面的N行内容
/regexp/ 匹配正则

[root@localhost ~]# sed -n '1,3'p quote.txt 
[root@localhost ~]# sed -n '1~2p' quote.txt 
[root@localhost ~]# sed -n '1,+1'p quote.txt 

打印匹配模式

/pattern/:匹配模式
#匹配单词evening，并打印此行
[root@localhost ~]# sed -n '/evening/p' quote.txt 
It was an evening of splendid music and company.
[root@localhost ~]# 

使用模式和行号进行查询

为编辑某个问价时，sed返回指定单词的许多行，为了返回结果更加精确，我们可以将行号和模式进行结合
使用模式与行号混合方法时：格式为：
line_number,/patter/   #行号与匹配逗号隔开
[root@localhost ~]# sed -n '/The/p' quote.txt
[root@localhost ~]# sed -n '4,/The/p' quote.txt

匹配元字符

$
对于元字符，使用转义字符\屏蔽其原有含义
/\$/p

[root@localhost ~]# sed -n '/\$/'p quote.txt

显示整个文件

从第一行匹配到最后一行    1,$

[root@localhost ~]# sed -n '1,$'p quote.txt

显示任意字符

匹配任意字母，
.*ing

[root@localhost ~]# sed -n '/.*ing/'p quote.txt

打印首行

打印文件第一行
sed -n '1p'  quote.txt

打印最后一行

sed -n '$'p  quote.txt

打印行号

/pattern/=

[root@localhost ~]# sed -n '/music/=' quote.txt 
2
[root@localhost ~]# sed -n '/music/p' quote.txt 
It was an evening of splendid music and company.
[root@localhost ~]# sed -n -e '/musci/p' -e '/musci/=' quote.txt 
[root@localhost ~]# sed -n -e '/music/p' -e '/music/=' quote.txt 
It was an evening of splendid music and company.
2
[root@localhost ~]# cat -n quote.txt 
     1    The honeysuckle band played all night long for inly $90.
     2    It was an evening of splendid music and company.
     3    Too bad the disco floor fell through at 23:10.
     4    The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# sed -e '/music/=' quote.txt 
The honeysuckle band played all night long for inly $90.
2
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# 

文本追加

a/  附加文本
可以指定文本一行或多行附加到指定行，若不指定文本放置位置，缺省放置在每一行后面
格式：
    [address]a\text\text\text\
        

    root@localhost ~]# sed "/company/a\I love Beijing Tian'anmen " quote.txt 

    [root@localhost ~]# sed "a\I love Beijing Tian'anmen " quote.txt 

利用脚本进行追加

[root@localhost ~]# vim add.sed
[root@localhost ~]# cat add.sed 
#!/bin/sed -f
/company/a\
I love my gril friend. \
I love myself.
[root@localhost ~]# 

[root@localhost ~]# chmod u+x add.sed 
[root@localhost ~]# ./add.sed 

[root@localhost ~]# ./add.sed  quote.txt 
The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
I love my gril friend. 
I love myself.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# 

插入文本

i/  插入文本
插入文本类似于附加文本，只是在指定行前插入
[root@localhost ~]# vim insert.sed
[root@localhost ~]# cat insert.sed 
#!/bin/sed -f
4 i\
Welcome to Beijing Tu Ling XueYuan 
[root@localhost ~]# 

[root@localhost ~]# chmod u+x insert.sed 
[root@localhost ~]# ./insert.sed quote.txt 
The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
Welcome to Beijing Tu Ling XueYuan 
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# 

修改文本

[address[,address]] c\
text/
text/
………………

[root@localhost ~]# vim cha.sed
[root@localhost ~]# chmod +x cha.sed 
[root@localhost ~]# ./cha.sed quote.txt 
北京图灵学院欢迎您！
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# cat -n quote.txt 
     1    The honeysuckle band played all night long for inly $90.
     2    It was an evening of splendid music and company.
     3    Too bad the disco floor fell through at 23:10.
     4    The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# cat -n cha.sed 
     1    #!/bin/sed -f
     2    /honeysuckle/ c\
     3    北京图灵学院欢迎您！
[root@localhost ~]# 

针对于同一个文件可以对脚本进行混合操作

[root@localhost ~]# vim Supper.sed
[root@localhost ~]# 
[root@localhost ~]# cat Supper.sed 
#!/bin/sed -f
1 c\
北京图灵学院欢迎您~~~


/evening/  i\
欢迎来到北京图灵学院学习~~~


3 a\
我们的课程包含python,java,数据分析，前端，php,linux



$ c\
北京图灵学院祝愿学业有成~~
[root@localhost ~]# chmod u+x Supper.sed 
[root@localhost ~]# ./Supper.sed quote.txt 
北京图灵学院欢迎您~~~
欢迎来到北京图灵学院学习~~~
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
我们的课程包含python,java,数据分析，前端，php,linux
北京图灵学院祝愿学业有成~~
[root@localhost ~]# 

删除文本

格式:
[address[,address]]d   #可以是的行也可以是匹配

删除第一行：
删除最后一行：
使用正则进行删除：


[root@localhost ~]# sed '1d' quote.txt 
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# sed '$d' quote.txt 
The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
[root@localhost ~]# sed '/music/d' quote.txt 
The honeysuckle band played all night long for inly $90.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# 

替换文本

格式：
[address[,address]] s\ x/y/[gpwn]

s  通知sed为替换操作，查询x，替换为y

g    全局替换，默认替换第一次匹配到的
p    标准输出，加p使-n不生效
w    替换输出文件为，另存为


#替换company为COMPANY
[root@localhost ~]# sed 's/company/CONMPANY/' quote.txt

#替换小写the
[root@localhost ~]# sed 's/The/the/' quote.txt 

[root@localhost ~]# sed 's/The/the/' quote.txt 
the The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
the local nurse MIss P.Neave was in attendance.
[root@localhost ~]# sed 's/The/the/g' quote.txt 
the the honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
the local nurse MIss P.Neave was in attendance.
[root@localhost ~]# 


#从$90中删除$符号
[root@localhost ~]# sed 's/\$//' quote.txt 
The The honeysuckle band played all night long for inly 90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# 


#文件另存为
w 存储数据文件
[root@localhost ~]# sed 's/splendid/HELLO/' quote.txt  > sed.out


[root@localhost ~]# sed 's/splendid/HELLO/w' quote.txt
sed: couldn't open file : No such file or directory
[root@localhost ~]# sed 's/splendid/HELLO/w sed.out' quote.txt
The The honeysuckle band played all night long for inly $90.
It was an evening of HELLO music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# cat sed.out 
It was an evening of HELLO music and company.
[root@localhost ~]# 

使用替换修改字符串

如果要附加或修改一个字符串，可以使用(&)命令
&命令保存发现模式，以便重新调用，然后把他放在替换字符串里面
先给出一个被替换模式，然后是一个准备附加在第一个模式后的另一个模式，并且后面带有&，这样修改模式将放在匹配模式之前


[root@localhost ~]# sed -n 's/floor/FLLOR &/p' quote.txt
[root@localhost ~]# sed -n 's/band/ Welcome  TLXY &/'p quote.txt

写入文件

使用 > 文件重定向
[root@localhost ~]# 
[root@localhost ~]# sed '1,2 w file' quote.txt 
The The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# cat fil
file    filedt  filrft  
[root@localhost ~]# cat file
The The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
[root@localhost ~]# 

从文件中读取文本

格式：
    address r  filename

[root@localhost ~]# vim r.txt
[root@localhost ~]# cat r.txt 
I love my girl friend
[root@localhost ~]# sed '/company/r r.txt' quote.txt 
The The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
I love my girl friend
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
[root@localhost ~]# sed '$r r.txt' quote.txt 
The The honeysuckle band played all night long for inly $90.
It was an evening of splendid music and company.
Too bad the disco floor fell through at 23:10.
The local nurse MIss P.Neave was in attendance.
I love my girl friend
[root@localhost ~]# 

匹配后退出

q
有时候需要在模式匹配首次出现后退出，以便执行其他脚本指令
格式：
    address q

[root@localhost ~]# cat q.txt 
Line 1.band
Line 2.bad
Line 3.was
Line 4.was
[root@localhost ~]#
[root@localhost ~]# sed '/.*a.*/q' q.txt 
Line 1.band
[root@localhost ~]# 

去除首行数字

[root@localhost ~]# cat a.txt 
121Line 1.band
3232Line 2.bad
342345234Line 3.was
342342Line 4.was
[root@localhost ~]# 


[root@localhost ~]# sed 's/^[0-9]*/Start /g' a.txt 
Start Line 1.band
Start Line 2.bad
Start Line 3.was
Start Line 4.was
[root@localhost ~]# sed 's/^[0-9]*//g' a.txt 
Line 1.band
Line 2.bad
Line 3.was
Line 4.was
[root@localhost ~]# 

处理报文数据

[root@localhost ~]# cat sql.txt 
Database    Size(MB)    Date Created
…………………………………………………………………………
GOSOUTH        2244        12/11/97
TRISUD        5632        8/9/99
(2 rows affected)
[root@localhost ~]# 



[root@localhost ~]# cat sql.txt | sed 's/……*//g' | sed '/^$/d' | sed '$d' | sed '1d' | awk '{print $0}'
GOSOUTH        2244        12/11/97
TRISUD        5632        8/9/99
[root@localhost ~]#