Sed and Awk 101 Hacks --1-sed 命令学习

Sed and Awk 101 Hacks

(Chr1: sed syntax and basic commands)

1.sed command syntax

$ cat >employee.txt
101,John Doe,CEO
102,Jason Smith,IT Manager
103,Raj Reddy,Sysadmin
104,Anand Ram,Developer
105,Jane Miller,Sales Manager

2.print （P）

#仅打印第二行
$ sed -n '2p' employee.txt
102,Jason Smith,IT Manager
#打印第1-4行
$ sed -n '1,4 p' employee.txt 
101,John Doe,CEO 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer
#打印第二行到最后一行
$ sed -n '2,$ p' employee.txt 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
105,Jane Miller,Sales Manager 
# 按步长 打印奇数行
$ sed -n '1~2 p' employee.txt 
101,John Doe,CEO 
103,Raj Reddy,Sysadmin 
105,Jane Miller,Sales Manager 

3.匹配并结合打印

#字符匹配并打印改行
$ sed -n '/Jane/ p' employee.txt 
105,Jane Miller,Sales Manager 
# 打印匹配的行到第四行（，到）；如果匹配的行在第四行之后，则只打印匹配行
$ sed -n '/Jason/,4 p' employee.txt 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
# 打印两次匹配及中间的行
$ sed -n '/Raj/,/Jane/ p' employee.txt 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
105,Jane Miller,Sales Manager
# 打印匹配行及后面两行
$ sed -n '/Jason/,+2 p' employee.txt 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 

4.删除行（d）

# 删除所有行
sed 'd' employee.txt 
# 以下内容结合 打印（P）命令理解
# , ：表示num1到num2的所有行
# $ ：指文件的尾部
# ~ ：步长
$ sed '2 d' employee.txt 
$ sed '1,4 d' employee.txt
$ sed '2,$ d' employee.txt
$ sed '1~2 d' employee.txt 
$ sed '/Manager/ d' employee.txt
$ sed '/Jason/,4 d' employee.txt 
$ sed '/Raj/,$ d' employee.txt
$ sed '/Raj/,/Jane/ d' employee.txt
$ sed '/Jason/,+2 d' employee.txt
# 删除所有的空行
sed '/^$/ d' employee.txt
# 删除“#”注释行
sed '/^#/ d' employee.txt

5.输出到文件（write，w)

# 输出结果到文件 output.txt,此时并打印结果到了screen，（不打印则加 -n）
$ sed 'w output.txt' employee.txt
$ sed -n 'w output.txt' employee.txt
# 只输出第二行到输出文件中，以下命令结合前面内容理解
$ sed -n '2 w output.txt' employee.txt
$ sed -n '1,4 w output.txt' employee.txt 
$ sed -n '2,$ w output.txt' employee.txt 
$ sed -n '2,$ w output.txt' employee.txt 
$ sed -n '/Jane/ w output.txt' employee.txt
$ sed -n '/Jason/,4 w output.txt' employee.txt
$ sed -n '/Raj/,$ w output.txt' employee.txt
$ sed -n '/Raj/,/Jane/ w output.txt' employee.txt
$ sed -n '/Jason/,+2 w output.txt' employee.txt 

Chr2 . Sed 替换命令

6、sed 替命令语法

sed '[address-range|pattern-range] s/original-string/replacement-string/[substitute-flags]' inputfile
In the above sed substitute command syntax:
• address-range or pattern-range is optional. If you don't specify
one, sed will execute the substitute command on all lines. 
• s – tells Sed to execute the substitute command 
• original-string – this is the string to be searched for in the
input file. The original-string can also be a regular expression. 
• replacement-string – Sed will replace original-string with this
string.
• substitute-flags are optional. More on this in the next section. 

7.全局命令 （global flag ， g flag）

# some commond
$ sed 's/Manager/Director/' employee.txt
# 只替换包含Sales字符的行
$ sed '/Sales/s/Manager/Director/' employee.txt 
# 不加g flag则只替换第一次出现的字符
$ sed 's/a/A/' employee.txt 
101,John Doe,CEO 
102,JAson Smith,IT Manager 
103,RAj Reddy,Sysadmin 
104,AnAnd Ram,Developer 
105,JAne Miller,Sales Manager
$ sed 's/a/A/g' employee.txt 
101,John Doe,CEO 
102,JAson Smith,IT MAnAger 
103,RAj Reddy,SysAdmin 
104,AnAnd RAm,Developer 
105,JAne Miller,SAles MAnAger

8.数字标签（1，2，3，3… flag）

# 替换第二次出现的字符
$ sed 's/a/A/2' employee.txt 
101,John Doe,CEO 
102,Jason Smith,IT MAnager 
103,Raj Reddy,SysAdmin 
104,Anand RAm,Developer 
105,Jane Miller,SAles Manager
#如果没有第二次则不替换，但会打印
$ sed 's/locate/find/2' substitute-locate.txt 
locate command is used to find files 
locate command uses database to find files
locate command can also use regex for searching

9.打印 print flag（P flag）

# 只打印进行了替换的行
$ sed -n 's/John/Johnny/p' employee.txt
$ sed -n 's/locate/find/2p' substitute-locate.txt

10. 输出flag （w flag）

$ sed -n 's/John/Johnny/w output.txt' employee.txt
$ sed 's/locate/find/2w output.txt' substitute-locate.txt 

11. ignore flag （i flag）

$ sed 's/john/Johnny/' employee.txt
# Replace “john” or “John” with Johnny:
$ sed 's/john/Johnny/i' employee.txt

12. Execute flag （e flag）

# 利用替换命令在行首添加内容
$ sed 's/^/ls -l /' files.txt 
ls -l /etc/passwd 
ls -l /etc/group 
# 添加并执行 
$ sed 's/^/ls -l /e' files.txt 
-rw-r--r-- 1 root root 1547 Oct 27 08:11 /etc/passwd 
-rw-r--r-- 1 root root 651 Oct 27 08:11 /etc/group

13. 结合替换命令

$ sed -n 's/Manager/Director/gipw output.txt'
employee.txt 
102,Jason Smith,IT Director 
105,Jane Miller,Sales Director 

14. sed 定界符

$ vi path.txt 
reading /usr/local/bin directory
$ sed 's/\/usr\/local\/bin/\/usr\/bin/' path.txt 
reading /usr/bin directory
# 使用不同的定界符
sed 's|/usr/local/bin|/usr/bin|' path.txt 
sed 's^/usr/local/bin^/usr/bin^' path.txt 
sed 's@/usr/local/bin@/usr/bin@' path.txt 
sed 's!/usr/local/bin!/usr/bin!' path.txt

略

Chr3 正则表达式

20-21

Zero or more Occurrences (*)
One or more Occurrence (\+) 
Zero or one Occurrence (\?)
Escaping the Special Character (\) # 转义符
Character Class ([0-9]) 
OR Operation (|)
Exactly M Occurrences ({m}) 
Word Boundary (\b)



$ sed -n '/127\.0\.0\.1/ p' /etc/hosts 
127.0.0.1 localhost.localdomain localhost
$ sed -n '/[234]/ p' employee.txt 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
$ sed -n '/[2-4]/ p' employee.txt 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer
$ sed -n '/101\|102/ p' employee.txt 
101,John Doe,CEO 
102,Jason Smith,IT Manager 
$ sed -n '/[0-9]/ p' numbers.txt 
1 
12 
123 
1234 
12345
$ sed -n '/^[0-9]\{5\}$/ p' numbers.txt 
12345 
$ sed -n '/^[0-9]\{3,5\}$/ p' numbers.txt 
123 
1234 
12345
# Match lines containing the whole word "the"
$ sed -n '/\bthe\b/ p' words.txt 
word matching using: the 
$ sed -n '/\bthe/ p' words.txt 
word matching using: the 
word matching using: thethe 
word matching using: they

22. 使用正则表达式进行替换

# 替换每行后面最后两个字符
$ sed 's/..$/,Not Defined/' employee.txt 
101,John Doe,C,Not Defined
102,Jason Smith,IT Manag,Not Defined
103,Raj Reddy,Sysadm,Not Defined
104,Anand Ram,Develop,Not Defined
105,Jane Miller,Sales Manag,Not Defined
# 删除manager及后面的字符
$ sed 's/Manager.*//' employee.txt 
101,John Doe,CEO 
102,Jason Smith,IT 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
105,Jane Miller,Sales 
# 删除空行及注释行#
sed -e 's/#.*// ; /^$/ d' employee.txt

$ vi test.html
<html><body><h1>Hello World!</h1></body></html>
$ sed -e 's/<[^>]*>//g' test.html
Hello World!

Chr4 sed 执行

1. Use multiple -e option in the command line
sed -e 'command1' -e 'command2' -e 'command3' inputfile

2. Break-up several sed commands using \ 
sed -e 'command1' \
-e 'command2' \
-e 'command3' inputfile

3. Group multiple commands using { } 
sed '{command1
command2
command3}' inputfile

4.Sed Script Files
$ vi mycommands.sed 
s/\([^,]*\),\([^,]*\),\(.*\).*/\2,\1,\3/g 
s/^.*/<&>/ 
s/Developer/IT Manager/ 
s/Manager/Director/
$ sed -f mycommands.sed employee.txt 
<John Doe,101,CEO> 
<Jason Smith,102,IT Director> 
<Raj Reddy,103,Sysadmin> 
<Anand Ram,104,IT Director> 
<Jane Miller,105,Sales Director>

25-27 略

Chr 5. 文本后面追加内容

28. 行后添加内容（append, a)

# 语法
$ sed '[address] a the-line-to-append' input-file
# 第二行后面添加一行
$ sed '2 a 203,Jack Johnson,Engineer' employee.txt
101,John Doe,CEO 
102,Jason Smith,IT Manager 
203,Jack Johnson,Engineer
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
105,Jane Miller,Sales Manager
# 最后一行后添加内容
$ sed '$ a 106,Jack Johnson,Engineer' employee.txt 
101,John Doe,CEO 
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
105,Jane Miller,Sales Manager 
106,Jack Johnson,Engineer
# 添加两行
$ sed '/Jason/a\ 
203,Jack Johnson,Engineer\ 
204,Mark Smith,Sales Engineer' employee.txt 

29.插入行（insert，i）

$ sed '2 i 203,Jack Johnson,Engineer' employee.txt
101,John Doe,CEO 
203,Jack Johnson,Engineer
102,Jason Smith,IT Manager 
103,Raj Reddy,Sysadmin 
104,Anand Ram,Developer 
105,Jane Miller,Sales Manager
$ sed '$ i 108,Jack Johnson,Engineer' employee.txt 
$ sed '/Jason/i\
203,Jack Johnson,Engineer\ 
204,Mark Smith,Sales Engineer' employee.txt 

30. 改变行（change, c）

$ sed '2 c 202,Jack Johnson,Engineer' employee.txt
101,John Doe,CEO 
202,Jack Johnson,Engineer
103,Raj Reddy,Sysadmin
...
$ sed '/Raj/c\ 
203,Jack Johnson,Engineer\ 
204,Mark Smith,Sales Engineer' employee.txt

31. 结合 a, i 和c 命令

You can also combine the a, i, and c commands. the following sed
example does all these three things: 
• a - Append 'Jack Johnson' after 'Jason' 
• i - Insert 'Mark Smith' before 'Jason' 
• c - Change 'Jason' to 'Joe Mason'

$ sed '/Jason/ { 
a\ 
204,Jack Johnson,Engineer 
i\ 
202,Mark Smith,Sales Engineer 
c\ 
203,Joe Mason,Sysadmin 
}' employee.txt

31-33

# 打印隐藏的符号 （ l ）：\n \t
$ sed -n l tabfile.txt
# 在20th个字符后显示隐藏符号
$ sed -n 'l 20' employee.txt

# 打印出行号
$ sed = employee.txt 
1 
101,John Doe,CEO 
2 
102,Jason Smith,IT Manager 
$ sed '1,3 =' employee.txt
$ sed '/Jane/ =' employee.txt 
$ sed -n '/Raj/ =' employee.txt
$ sed -n '$ =' employee.txt

# 切换大小写 （y）
# 将文本中的a,b,c,d,e 变成 A,B,C,D,E
$ sed 'y/abcde/ABCDE/' employee.txt

34-50 略