linux 中 awk sed cut sort 常规操作

awk

awk语言的最基本功能是在文件或者字符串中基于指定规则浏览和抽取信息，awk抽取信息后，才能进行其他文本操作。完整的awk脚本通常用来格式化文本文件中的信息。

通常，awk是以文件的一行为处理单位的。awk每接收文件的一行，然后执行相应的命令，来处理文本。

默认以空格符作为分隔符将每行切片

awk --help

Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options:		GNU long options: (standard)
	-f progfile		--file=progfile
	-F fs			--field-separator=fs。以 fs 作为分隔符
	-v var=val		--assign=var=val。赋值一个用户定义变量
Short options:		GNU long options: (extensions)
	-b			--characters-as-bytes
	-c			--traditional
	-C			--copyright
	-d[file]		--dump-variables[=file]
	-e 'program-text'	--source='program-text'
	-E file			--exec=file
	-g			--gen-pot
	-h			--help
	-L [fatal]		--lint[=fatal]
	-n			--non-decimal-data
	-N			--use-lc-numeric
	-O			--optimize
	-p[file]		--profile[=file]
	-P			--posix
	-r			--re-interval
	-S			--sandbox
	-t			--lint-old
	-V			--version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
	gawk '{ sum += $1 }; END { print sum }' file
	gawk -F: '{ print $1 }' /etc/passwd

awk 操作实例

实例1

[root@localhost ~]# last -n 5
root     pts/0        172.16.110.179   Tue Aug 10 15:27   still logged in   
root     tty1                          Tue Aug 10 15:27   still logged in   
reboot   system boot  3.10.0-1062.el7. Tue Aug 10 15:26 - 15:29  (00:02)    
root     pts/1        172.16.110.179   Tue Aug  3 17:25 - crash (6+22:00)   
root     pts/0        172.16.110.179   Mon Aug  2 17:49 - crash (7+21:36)

[root@localhost ~]# last -n 5 | awk '{print $1}'
root
root
reboot
root
root

实例2

首先将 /etc/passwd 赋值到当前目录 另存为 pw
[root@localhost ~]# cat pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin

只显示 /etc/passwd 账号
[root@localhost ~]# cat pw | awk -F : '{print $1}'
root
bin
daemon
adm
lp
sync
shutdown
halt
mail
operator
games
ftp
nobody
systemd-network
dbus
polkitd
sshd
postfix
clickhouse
dockerroot

显示账号和账号对应的shell,并且以逗号隔开，且在所有行上添加列名 “name,shell”,并且在最后一行添加 “wayne,/bin/nosh”
[root@localhost ~]# cat pw |awk  -F ':'  'BEGIN {print "name,shell"}  {print $1","$7} END {print "blue,/bin/nossh"}'
name,shell
root,/bin/bash
bin,/sbin/nologin
daemon,/sbin/nologin
adm,/sbin/nologin
lp,/sbin/nologin
sync,/bin/sync
shutdown,/sbin/shutdown
halt,/sbin/halt
mail,/sbin/nologin
operator,/sbin/nologin
games,/sbin/nologin
ftp,/sbin/nologin
nobody,/sbin/nologin
systemd-network,/sbin/nologin
dbus,/sbin/nologin
polkitd,/sbin/nologin
sshd,/sbin/nologin
postfix,/sbin/nologin
clickhouse,/bin/false
dockerroot,/sbin/nolo

实例3

搜索/etc/passwd 有root关键字的所有行
[root@localhost ~]# awk -F: '/root/' /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin

搜索/etc/passwd 有root关键字的所有行，并只显示对应的shell
[root@localhost ~]# awk -F: '/root/{print $7}' pw
/bin/bash
/sbin/nologin
/sbin/nologin

sed

sed是一个很好的文件处理工具，本身是一个管道命令，主要是以行为单位进行处理，可以将数据行进行替换、删除、新增、选取等特定工作

sed --help

Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...

  -n, --quiet, --silent
                 suppress automatic printing of pattern space。只有经过sed特殊处理的那一行（或动作）才会打印出来。
  -e script, --expression=script
                 add the script to the commands to be executed。直接在指令列模式上进行sed的动作编辑。
  -f script-file, --file=script-file
                 add the contents of script-file to the commands to be executed。直接将sed的动作写在一个文档中，-f filename 可以执行filename内的sed动作
  --follow-symlinks
                 follow symlinks when processing in place
  -i[SUFFIX], --in-place[=SUFFIX]
                 edit files in place (makes backup if SUFFIX supplied)。直接修改读取的档案内容，而不是由荧幕输出。
  -c, --copy
                 use copy instead of rename when shuffling files in -i mode
  -b, --binary
                 does nothing; for compatibility with WIN32/CYGWIN/MSDOS/EMX (
                 open files in binary mode (CR+LFs are not treated specially))
  -l N, --line-length=N
                 specify the desired line-wrap length for the `l' command
  --posix
                 disable all GNU extensions.
  -r, --regexp-extended
                 use extended regular expressions in the script.
  -s, --separate
                 consider files as separate rather than as a single continuous
                 long stream.
  -u, --unbuffered
                 load minimal amounts of data from the input files and flush
                 the output buffers more often
  -z, --null-data
                 separate lines by NUL characters
  --help
                 display this help and exit
  --version
                 output version information and exit

常用命令功能：
    a  ∶新增， a 的后面可以接字串，而这些字串会在新的一行出现(目前的下一行)～
    c  ∶取代， c 的后面可以接字串，这些字串可以取代 n1,n2 之间的行！
    d  ∶删除，因为是删除啊，所以 d 后面通常不接任何咚咚；
     i  ∶插入， i 的后面可以接字串，而这些字串会在新的一行出现(目前的上一行)；
     p ∶列印，亦即将某个选择的资料印出。通常 p 会与参数 sed -n 一起运作～
     s ∶取代，可以直接进行取代的工作哩！通常这个 s 的动作可以搭配正规表示法！例如 1,20s/old/new/g 就是啦！

If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret.  All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.

GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed@gnu.org>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.

sed 操作实例

实例1 删除（输出修改，不修改文件）

(首先将 /etc/passwd 赋值到当前目录 另存为 pw,如 awk中操作)
[root@localhost ~]# cat  pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin

删除第一行
[root@localhost ~]# sed '1d' pw
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin

删除最后一行
[root@localhost ~]# sed '$d' pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false

删除第一行到第二行
[root@localhost ~]# sed '1,2d' pw
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
[root@localhost ~]# sed '2,$d' pw
root:x:0:0:root:/root:/bin/bash

删除第二行到最后一行
[root@localhost ~]# sed '2,$d' pw
root:x:0:0:root:/root:/bin/bash

实例2 显示（只显示，不修改文件）

显示第一行 
[root@localhost ~]# sed -n '1p' pw
显示最后一行
[root@localhost ~]# sed -n '$p' pw
显示第一行到第二行
[root@localhost ~]# sed -n '1,2p' pw
显示第二行到最后一行
[root@localhost ~]# sed -n '2,$p' pw

实例3 使用模式进行查询（只查询，不修改文件）

查询包括关键字root所在所有行
[root@localhost ~]# sed -n '/root/p' pw
#查询包括关键字$所在所有行，使用反斜线\屏蔽特殊含义
[root@localhost ~]# sed -n '/\$/p' pw

实例4 增加一行或多行字符串（输出修改，不修改文件）

第一行后增加字符串"drink tea"
[root@localhost ~]# sed '1a drink tea' pw
第一行到第三行后增加字符串"drink tea"
[root@localhost ~]# sed '1,3a drink tea' pw
第一行后增加多行，使用换行符\n
[root@localhost ~]# sed '1a drink tea\nor coffee' pw

实例5 代替一行或多行（输出修改，不修改文件）

第一行代替为 drink tea
[root@localhost ~]# sed '1c drink tea' pw
第一行到第二行代替为 dink tea
[root@localhost ~]# sed '1,2c drink tea' pw

实例6 替换一行中的某部分（输出修改，不修改文件）

格式：sed 's/要替换的字符串/新的字符串/g'   （要替换的字符串可以用正则表达式）
替换 root 为 wayne
[root@localhost ruby] # sed -n '/root/p' pw | sed 's/root/wayne/g'
删除ruby
[root@localhost ruby] # sed -n '/root/p' pw | sed 's/root//g'

实例7 插入（修改文件）

在文件pw中最后一行直接输入"bye"
[root@localhost ~]# sed -i '$a bye' pw

实例8 删除匹配行（修改文件）

sed -i '/匹配字符串/d'  filename  （注：若匹配字符串是变量，则需要“”，而不是‘’。记得好像是）

替换匹配行中的某个字符串
[root@localhost ~]# sed -i '/root/s/root/wayne/g' pw

cut

cut的工作就是剪，具体的说是在文件中负责剪切数据用。cut命令从文件的每一行剪切字节、字符和字段并将这些字节、字符和字段输出。

cut --help

Usage: cut OPTION... [FILE]...
Print selected parts of lines from each FILE to standard output.

Mandatory arguments to long options are mandatory for short options too.
  -b, --bytes=LIST        select only these bytes
  -c, --characters=LIST   select only these characters。指定具体的字符
  -d, --delimiter=DELIM   use DELIM instead of Tpw for field delimiter。指定分隔符分格列，默认分隔符是制表符
  -f, --fields=LIST       select only these fields;  also print any line that contains no delimiter character, unless the -s option is specified。列号，提取第几列
  -n                      with -b: don't split multibyte characters
      --complement        complement the set of selected bytes, characters
                            or fields
  -s, --only-delimited    do not print lines not containing delimiters
      --output-delimiter=STRING  use STRING as the output delimiter
                            the default is to use the input delimiter
      --help     display this help and exit
      --version  output version information and exit

Use one, and only one of -b, -c or -f.  Each LIST is made up of one
range, or many ranges separated by commas.  Selected input is written
in the same order that it is read, and is written exactly once.
Each range is one of:

  N     N'th byte, character or field, counted from 1
  N-    from N'th byte, character or field, to end of line
  N-M   from N'th to M'th (included) byte, character or field
  -M    from first to M'th (included) byte, character or field

With no FILE, or when FILE is -, read standard input.

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
For complete documentation, run: info coreutils 'cut invocation'

cut 操作实例

实例1

(首先将 /etc/passwd 赋值到当前目录 另存为 pw,如 awk中操作)
以 空格 切割 pw 输出第一列
[root@localhost ~]# cut -d " " -f 1
以 冒号 切割 pw 输出第一，七列
[root@localhost ~]# cut -d ":" -f 1,7 pw
在 pw 文件中，切割出包含 root 的行
[root@localhost ~]# cat pw | grep "root" | cut -d " " -f 1

sort

sort命令是在Linux里非常有用，它将文件进行排序，并将排序结果标准输出。

sort --help

Usage: sort [OPTION]... [FILE]...
  or:  sort [OPTION]... --files0-from=F
Write sorted concatenation of all FILE(s) to standard output.

Mandatory arguments to long options are mandatory for short options too.
Ordering options:

  -b, --ignore-leading-blanks  ignore leading blanks
  -d, --dictionary-order      consider only blanks and alphanumeric characters
  -f, --ignore-case           fold lower case to upper case characters
  -g, --general-numeric-sort  compare according to general numerical value
  -i, --ignore-nonprinting    consider only printable characters
  -M, --month-sort            compare (unknown) < 'JAN' < ... < 'DEC'
  -h, --human-numeric-sort    compare human readable numbers (e.g., 2K 1G)
  -n, --numeric-sort          compare according to string numerical value。依照数值的大小排序
  -R, --random-sort           sort by random hash of keys
      --random-source=FILE    get random bytes from FILE
  -r, --reverse               reverse the result of comparisons。以相反的顺序来排序
      --sort=WORD             sort according to WORD:
                                general-numeric -g, human-numeric -h, month -M,
                                numeric -n, random -R, version -V
  -V, --version-sort          natural sort of (version) numbers within text

Other options:

      --batch-size=NMERGE   merge at most NMERGE inputs at once;
                            for more use temp files

  -c, --check, --check=diagnose-first  check for sorted input; do not sort
  -C, --check=quiet, --check=silent  like -c, but do not report first bad line
      --compress-program=PROG  compress temporaries with PROG;
                              decompress them with PROG -d
      --debug               annotate the part of the line used to sort,
                              and warn about questionable usage to stderr
      --files0-from=F       read input from the files specified by
                            NUL-terminated names in file F;
                            If F is - then read names from standard input
  -k, --key=KEYDEF          sort via a key; KEYDEF gives location and type。指定需要排序的列
  -m, --merge               merge already sorted files; do not sort
  -o, --output=FILE         write result to FILE instead of standard output
  -s, --stable              stabilize sort by disabling last-resort comparison
  -S, --buffer-size=SIZE    use SIZE for main memory buffer
  -t, --field-separator=SEP  use SEP instead of non-blank to blank transition。设置排序时所用的分隔符
  -T, --temporary-directory=DIR  use DIR for temporaries, not $TMPDIR or /tmp;
                              multiple options specify multiple directories
      --parallel=N          change the number of sorts run concurrently to N
  -u, --unique              with -c, check for strict ordering;
                              without -c, output only the first of an equal run
  -z, --zero-terminated     end lines with 0 byte, not newline
      --help     display this help and exit
      --version  output version information and exit

KEYDEF is F[.C][OPTS][,F[.C][OPTS]] for start and stop position, where F is a
field number and C a character position in the field; both are origin 1, and
the stop position defaults to the line's end.  If neither -t nor -b is in
effect, characters in a field are counted from the beginning of the preceding
whitespace.  OPTS is one or more single-letter ordering options [bdfgiMhnRrV],
which override global ordering options for that key.  If no key is given, use
the entire line as the key.

SIZE may be followed by the following multiplicative suffixes:
% 1% of memory, b 1, K 1024 (default), and so on for M, G, T, P, E, Z, Y.

With no FILE, or when FILE is -, read standard input.

*** WARNING ***
The locale specified by the environment affects sort order.
Set LC_ALL=C to get the traditional sort order that uses
native byte values.

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
For complete documentation, run: info coreutils 'sort invocation'

sort 操作实例

实例1

[root@localhost ~]# cat pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin

以 冒号 切割 pw 文件，并以第三列倒序排序
[root@localhost ~]# sort -t : -nrk 3 pw
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
halt:x:7:0:halt:/sbin:/sbin/halt
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
sync:x:5:0:sync:/sbin:/bin/sync
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
root:x:0:0:root:/root:/bin/bash

参考：

https://www.cnblogs.com/leedaily/p/8329445.html

https://www.cnblogs.com/kwzblog/p/13652977.html