awk
awk语言的最基本功能是在文件或者字符串中基于指定规则浏览和抽取信息,awk抽取信息后,才能进行其他文本操作。完整的awk脚本通常用来格式化文本文件中的信息。
通常,awk是以文件的一行为处理单位的。awk每接收文件的一行,然后执行相应的命令,来处理文本。
默认以空格符作为分隔符将每行切片
awk --help
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options: (standard)
-f progfile --file=progfile
-F fs --field-separator=fs。以 fs 作为分隔符
-v var=val --assign=var=val。赋值一个用户定义变量
Short options: GNU long options: (extensions)
-b --characters-as-bytes
-c --traditional
-C --copyright
-d[file] --dump-variables[=file]
-e 'program-text' --source='program-text'
-E file --exec=file
-g --gen-pot
-h --help
-L [fatal] --lint[=fatal]
-n --non-decimal-data
-N --use-lc-numeric
-O --optimize
-p[file] --profile[=file]
-P --posix
-r --re-interval
-S --sandbox
-t --lint-old
-V --version
To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.
gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.
Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
awk 操作实例
实例1
[root@localhost ~]# last -n 5
root pts/0 172.16.110.179 Tue Aug 10 15:27 still logged in
root tty1 Tue Aug 10 15:27 still logged in
reboot system boot 3.10.0-1062.el7. Tue Aug 10 15:26 - 15:29 (00:02)
root pts/1 172.16.110.179 Tue Aug 3 17:25 - crash (6+22:00)
root pts/0 172.16.110.179 Mon Aug 2 17:49 - crash (7+21:36)
[root@localhost ~]# last -n 5 | awk '{print $1}'
root
root
reboot
root
root
实例2
首先将 /etc/passwd 赋值到当前目录 另存为 pw
[root@localhost ~]# cat pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
只显示 /etc/passwd 账号
[root@localhost ~]# cat pw | awk -F : '{print $1}'
root
bin
daemon
adm
lp
sync
shutdown
halt
mail
operator
games
ftp
nobody
systemd-network
dbus
polkitd
sshd
postfix
clickhouse
dockerroot
显示账号和账号对应的shell,并且以逗号隔开,且在所有行上添加列名 “name,shell”,并且在最后一行添加 “wayne,/bin/nosh”
[root@localhost ~]# cat pw |awk -F ':' 'BEGIN {print "name,shell"} {print $1","$7} END {print "blue,/bin/nossh"}'
name,shell
root,/bin/bash
bin,/sbin/nologin
daemon,/sbin/nologin
adm,/sbin/nologin
lp,/sbin/nologin
sync,/bin/sync
shutdown,/sbin/shutdown
halt,/sbin/halt
mail,/sbin/nologin
operator,/sbin/nologin
games,/sbin/nologin
ftp,/sbin/nologin
nobody,/sbin/nologin
systemd-network,/sbin/nologin
dbus,/sbin/nologin
polkitd,/sbin/nologin
sshd,/sbin/nologin
postfix,/sbin/nologin
clickhouse,/bin/false
dockerroot,/sbin/nolo
实例3
搜索/etc/passwd 有root关键字的所有行
[root@localhost ~]# awk -F: '/root/' /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
搜索/etc/passwd 有root关键字的所有行,并只显示对应的shell
[root@localhost ~]# awk -F: '/root/{print $7}' pw
/bin/bash
/sbin/nologin
/sbin/nologin
sed
sed是一个很好的文件处理工具,本身是一个管道命令,主要是以行为单位进行处理,可以将数据行进行替换、删除、新增、选取等特定工作
sed --help
Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...
-n, --quiet, --silent
suppress automatic printing of pattern space。只有经过sed特殊处理的那一行(或动作)才会打印出来。
-e script, --expression=script
add the script to the commands to be executed。直接在指令列模式上进行sed的动作编辑。
-f script-file, --file=script-file
add the contents of script-file to the commands to be executed。直接将sed的动作写在一个文档中,-f filename 可以执行filename内的sed动作
--follow-symlinks
follow symlinks when processing in place
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)。直接修改读取的档案内容,而不是由荧幕输出。
-c, --copy
use copy instead of rename when shuffling files in -i mode
-b, --binary
does nothing; for compatibility with WIN32/CYGWIN/MSDOS/EMX (
open files in binary mode (CR+LFs are not treated specially))
-l N, --line-length=N
specify the desired line-wrap length for the `l' command
--posix
disable all GNU extensions.
-r, --regexp-extended
use extended regular expressions in the script.
-s, --separate
consider files as separate rather than as a single continuous
long stream.
-u, --unbuffered
load minimal amounts of data from the input files and flush
the output buffers more often
-z, --null-data
separate lines by NUL characters
--help
display this help and exit
--version
output version information and exit
常用命令功能:
a ∶新增, a 的后面可以接字串,而这些字串会在新的一行出现(目前的下一行)~
c ∶取代, c 的后面可以接字串,这些字串可以取代 n1,n2 之间的行!
d ∶删除,因为是删除啊,所以 d 后面通常不接任何咚咚;
i ∶插入, i 的后面可以接字串,而这些字串会在新的一行出现(目前的上一行);
p ∶列印,亦即将某个选择的资料印出。通常 p 会与参数 sed -n 一起运作~
s ∶取代,可以直接进行取代的工作哩!通常这个 s 的动作可以搭配正规表示法!例如 1,20s/old/new/g 就是啦!
If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret. All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed@gnu.org>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.
sed 操作实例
实例1 删除(输出修改,不修改文件)
(首先将 /etc/passwd 赋值到当前目录 另存为 pw,如 awk中操作)
[root@localhost ~]# cat pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
删除第一行
[root@localhost ~]# sed '1d' pw
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
删除最后一行
[root@localhost ~]# sed '$d' pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
删除第一行到第二行
[root@localhost ~]# sed '1,2d' pw
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
[root@localhost ~]# sed '2,$d' pw
root:x:0:0:root:/root:/bin/bash
删除第二行到最后一行
[root@localhost ~]# sed '2,$d' pw
root:x:0:0:root:/root:/bin/bash
实例2 显示(只显示,不修改文件)
显示第一行
[root@localhost ~]# sed -n '1p' pw
显示最后一行
[root@localhost ~]# sed -n '$p' pw
显示第一行到第二行
[root@localhost ~]# sed -n '1,2p' pw
显示第二行到最后一行
[root@localhost ~]# sed -n '2,$p' pw
实例3 使用模式进行查询(只查询,不修改文件)
查询包括关键字root所在所有行
[root@localhost ~]# sed -n '/root/p' pw
#查询包括关键字$所在所有行,使用反斜线\屏蔽特殊含义
[root@localhost ~]# sed -n '/\$/p' pw
实例4 增加一行或多行字符串(输出修改,不修改文件)
第一行后增加字符串"drink tea"
[root@localhost ~]# sed '1a drink tea' pw
第一行到第三行后增加字符串"drink tea"
[root@localhost ~]# sed '1,3a drink tea' pw
第一行后增加多行,使用换行符\n
[root@localhost ~]# sed '1a drink tea\nor coffee' pw
实例5 代替一行或多行(输出修改,不修改文件)
第一行代替为 drink tea
[root@localhost ~]# sed '1c drink tea' pw
第一行到第二行代替为 dink tea
[root@localhost ~]# sed '1,2c drink tea' pw
实例6 替换一行中的某部分(输出修改,不修改文件)
格式:sed 's/要替换的字符串/新的字符串/g' (要替换的字符串可以用正则表达式)
替换 root 为 wayne
[root@localhost ruby] # sed -n '/root/p' pw | sed 's/root/wayne/g'
删除ruby
[root@localhost ruby] # sed -n '/root/p' pw | sed 's/root//g'
实例7 插入(修改文件)
在文件pw中最后一行直接输入"bye"
[root@localhost ~]# sed -i '$a bye' pw
实例8 删除匹配行(修改文件)
sed -i '/匹配字符串/d' filename (注:若匹配字符串是变量,则需要“”,而不是‘’。记得好像是)
替换匹配行中的某个字符串
[root@localhost ~]# sed -i '/root/s/root/wayne/g' pw
cut
cut的工作就是剪,具体的说是在文件中负责剪切数据用。cut命令从文件的每一行剪切字节、字符和字段并将这些字节、字符和字段输出。
cut --help
Usage: cut OPTION... [FILE]...
Print selected parts of lines from each FILE to standard output.
Mandatory arguments to long options are mandatory for short options too.
-b, --bytes=LIST select only these bytes
-c, --characters=LIST select only these characters。指定具体的字符
-d, --delimiter=DELIM use DELIM instead of Tpw for field delimiter。指定分隔符分格列,默认分隔符是制表符
-f, --fields=LIST select only these fields; also print any line that contains no delimiter character, unless the -s option is specified。列号,提取第几列
-n with -b: don't split multibyte characters
--complement complement the set of selected bytes, characters
or fields
-s, --only-delimited do not print lines not containing delimiters
--output-delimiter=STRING use STRING as the output delimiter
the default is to use the input delimiter
--help display this help and exit
--version output version information and exit
Use one, and only one of -b, -c or -f. Each LIST is made up of one
range, or many ranges separated by commas. Selected input is written
in the same order that it is read, and is written exactly once.
Each range is one of:
N N'th byte, character or field, counted from 1
N- from N'th byte, character or field, to end of line
N-M from N'th to M'th (included) byte, character or field
-M from first to M'th (included) byte, character or field
With no FILE, or when FILE is -, read standard input.
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
For complete documentation, run: info coreutils 'cut invocation'
cut 操作实例
实例1
(首先将 /etc/passwd 赋值到当前目录 另存为 pw,如 awk中操作)
以 空格 切割 pw 输出第一列
[root@localhost ~]# cut -d " " -f 1
以 冒号 切割 pw 输出第一,七列
[root@localhost ~]# cut -d ":" -f 1,7 pw
在 pw 文件中,切割出包含 root 的行
[root@localhost ~]# cat pw | grep "root" | cut -d " " -f 1
sort
sort命令是在Linux里非常有用,它将文件进行排序,并将排序结果标准输出。
sort --help
Usage: sort [OPTION]... [FILE]...
or: sort [OPTION]... --files0-from=F
Write sorted concatenation of all FILE(s) to standard output.
Mandatory arguments to long options are mandatory for short options too.
Ordering options:
-b, --ignore-leading-blanks ignore leading blanks
-d, --dictionary-order consider only blanks and alphanumeric characters
-f, --ignore-case fold lower case to upper case characters
-g, --general-numeric-sort compare according to general numerical value
-i, --ignore-nonprinting consider only printable characters
-M, --month-sort compare (unknown) < 'JAN' < ... < 'DEC'
-h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G)
-n, --numeric-sort compare according to string numerical value。依照数值的大小排序
-R, --random-sort sort by random hash of keys
--random-source=FILE get random bytes from FILE
-r, --reverse reverse the result of comparisons。以相反的顺序来排序
--sort=WORD sort according to WORD:
general-numeric -g, human-numeric -h, month -M,
numeric -n, random -R, version -V
-V, --version-sort natural sort of (version) numbers within text
Other options:
--batch-size=NMERGE merge at most NMERGE inputs at once;
for more use temp files
-c, --check, --check=diagnose-first check for sorted input; do not sort
-C, --check=quiet, --check=silent like -c, but do not report first bad line
--compress-program=PROG compress temporaries with PROG;
decompress them with PROG -d
--debug annotate the part of the line used to sort,
and warn about questionable usage to stderr
--files0-from=F read input from the files specified by
NUL-terminated names in file F;
If F is - then read names from standard input
-k, --key=KEYDEF sort via a key; KEYDEF gives location and type。指定需要排序的列
-m, --merge merge already sorted files; do not sort
-o, --output=FILE write result to FILE instead of standard output
-s, --stable stabilize sort by disabling last-resort comparison
-S, --buffer-size=SIZE use SIZE for main memory buffer
-t, --field-separator=SEP use SEP instead of non-blank to blank transition。设置排序时所用的分隔符
-T, --temporary-directory=DIR use DIR for temporaries, not $TMPDIR or /tmp;
multiple options specify multiple directories
--parallel=N change the number of sorts run concurrently to N
-u, --unique with -c, check for strict ordering;
without -c, output only the first of an equal run
-z, --zero-terminated end lines with 0 byte, not newline
--help display this help and exit
--version output version information and exit
KEYDEF is F[.C][OPTS][,F[.C][OPTS]] for start and stop position, where F is a
field number and C a character position in the field; both are origin 1, and
the stop position defaults to the line's end. If neither -t nor -b is in
effect, characters in a field are counted from the beginning of the preceding
whitespace. OPTS is one or more single-letter ordering options [bdfgiMhnRrV],
which override global ordering options for that key. If no key is given, use
the entire line as the key.
SIZE may be followed by the following multiplicative suffixes:
% 1% of memory, b 1, K 1024 (default), and so on for M, G, T, P, E, Z, Y.
With no FILE, or when FILE is -, read standard input.
*** WARNING ***
The locale specified by the environment affects sort order.
Set LC_ALL=C to get the traditional sort order that uses
native byte values.
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
For complete documentation, run: info coreutils 'sort invocation'
sort 操作实例
实例1
[root@localhost ~]# cat pw
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
以 冒号 切割 pw 文件,并以第三列倒序排序
[root@localhost ~]# sort -t : -nrk 3 pw
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
clickhouse:x:998:996::/nonexistent:/bin/false
dockerroot:x:997:994:Docker User:/var/lib/docker:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
halt:x:7:0:halt:/sbin:/sbin/halt
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
sync:x:5:0:sync:/sbin:/bin/sync
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
root:x:0:0:root:/root:/bin/bash
参考: