awk学习笔记
Awk是什么
Awk、sed与grep,俗称Linux下的三剑客,它们之间有很多相似点,但是同样也各有各的特色,相似的地方是它们都可以匹配文本,其中sed和awk还可以用于文本编辑,而grep则不具备这个功用。sed是一种非交互式且面向字符流的编辑器(a "non-interactive" stream-oriented editor),而awk则是一门模式匹配的编程语言,因为它的主要功能是用于匹配文本并处理,同时它有一些编程语言才有的语法,例如函数、分支循环语句、变量等等,当然比起我们常见的编程语言,Awk相对比较简单。
使用Awk,我们可以做以下事情:
- 将文本文件视为由字段和记录组成的文本数据库;
- 在操作文本数据库的过程中能够使用变量;
- 能够使用数学运算和字符串操作
- 能够使用常见的编程结构,例如条件分支与循环;
- 能够格式化输出;
- 能够自定义函数;
- 能够在awk脚本中执行UNIX命令;
- 能够处理UNIX命令的输出结果;
装备以上功能,awk能够做得事情非常多。但千里之行,始于足下,我们首先从最基本的命令行语法开始,一步一步得走入awk的编程世界。
命令行语法
同sed一样,awk的命令行语法也有两种形式:
<span class="pln">awk </span><span class="pun">[-</span><span class="pln">F ERE</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[-</span><span class="pln">v assignment</span><span class="pun">]</span><span class="pln"> </span><span class="pun">...</span><span class="pln"> program </span><span class="pun">[</span><span class="pln">argument </span><span class="pun">...]</span><span class="pln"> awk </span><span class="pun">[-</span><span class="pln">F ERE</span><span class="pun">]</span><span class="pln"> </span><span class="pun">-</span><span class="pln">f progfile </span><span class="pun">...</span><span class="pln"> </span><span class="pun">[-</span><span class="pln">v assignment</span><span class="pun">]</span><span class="pln"> </span><span class="pun">...[</span><span class="pln">argument </span><span class="pun">...]</span>
这里的program类似sed中的script,因为我们一直强调awk是一门编程语言,所以将awk的脚本视为一段代码。而awk的脚本同样可以写到一个文件中,并通过-f参数指定,这一点和sed是一样的。program一般多个pattern和action序列组成,当读入的记录匹配pattern时,才会执行相应的action命令。这里有一点要注意,在第一种形式中,除去命令行选项外,program参数一定要位于第一个位置。
Awk的输入被解析成多个记录(Record),默认情况下,记录的分隔符是\n,因此可以认为一行就是一个记录,记录的分隔符可以通过内置变量RS
更改。当记录匹配某个pattern时,才会执行后续的action命令。
而每个记录由进一步地被分隔成多个字段(Field),默认情况下字段的分隔符是空白符,例如空格、制表符等等,也可以通过-F ERE
选项或者内置变量FS
更改。在awk中,可以通过$1,$2...来访问对应位置的字段,同时$0存放整个记录,这一点有点类似shell下的命令行位置参数。关于这些内容,我们会在下面详细介绍,这里你只要知道有这些东西就好。
标准的awk命令行参数主要由以下三个:
-F ERE
:定义字段分隔符,该选项的值可以是扩展的正则表达式(ERE);-f progfile
:指定awk脚本,可以同时指定多个脚本,它们会按照在命令行中出现的顺序连接在一起;-v assignment
:定义awk变量,形式同awk中的变量赋值,即name=value,赋值发生在awk处理文本之前;
为了便于理解,这里举几个简单的例子。通过-F参数设置冒号:为分隔符,并打印各个字段:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"1:2:3"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="pun">-</span><span class="pln">F</span><span class="pun">:</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{print $1 " and " $2 " and " $3}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">and</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="kwd">and</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span>
在awk的脚本中访问通过-v选项设置的变量:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ echo </span><span class="pun">|</span><span class="pln"> awk </span><span class="pun">-</span><span class="pln">v a</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print a}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
从上面可以看到,通过-v选项设置的变量在BEGIN
的位置就可以访问了。BEGIN
是一个特殊的pattern,它在awk处理输入之前就会执行,可以认为是一个初始化语句,与此对应的还有END
。
好像还没介绍如何指定处理的文件,是不是最后的argument就是指定的文件?在看我这本书之前,我也是这样认为的,但是实际上arguemnt有两种形式,它们分别是输入文件(file)和变量赋值(assignment)。
awk可以同时指定多个输入文件,如果输入文件的文件名为'-',表示从标准输入读取内容。
变量赋值类似-v选项,它的形式为name=value。awk中的变量名同一般的编程语言无太多区别,但是不能同awk的保留关键字重名,可以查看awk的man手册查询哪些是保留关键字。而变量值只有两种形式:字符串和数值。变量赋值必须位于脚本参数的后面,与文件名参数无先后顺序的要求,但是位于不同位置的赋值它的执行时机是不同的。
我们用实际的例子来解释这个区别,假设有两个文件:a和b,它们的内容分别如下所示:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat a file a </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat b file b</span>
为了说明赋值操作发生的时机,我们在BEGIN,正常处理,END三个地方都打印变量的值。
第一种情况: 变量赋值位于所有文件名参数之前
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print "BEGIN: " var} {print "PROCESS: " var} \ END {print "END: " var }'</span><span class="pln"> </span><span class="kwd">var</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> a </span><span class="kwd">BEGIN</span><span class="pun">:</span><span class="pln"> PROCESS</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">END</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
结果:赋值操作发生在正常处理之前,BEGIN
动作之后。
第二种情况:变量赋值位于所有文件名之后:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print "BEGIN: " var} {print "PROCESS: " var} \ END {print "END: " var }'</span><span class="pln"> a </span><span class="kwd">var</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">BEGIN</span><span class="pun">:</span><span class="pln"> PROCESS</span><span class="pun">:</span><span class="pln"> </span><span class="kwd">END</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
结果:赋值操作发生在正常处理之后,END
动作之前。
第三种情况:变量赋值位于文件名之间:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print "BEGIN: " var} {print "PROCESS: " var} \ END {print "END: " var }'</span><span class="pln"> a </span><span class="kwd">var</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> b </span><span class="kwd">BEGIN</span><span class="pun">:</span><span class="pln"> PROCESS</span><span class="pun">:</span><span class="pln"> PROCESS</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">END</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
结果:赋值操作发生在处理前面的文件之后,并且位于处理后面的文件之前;
总结如下:
- 如果变量赋值在第一个文件参数之前,在
BEGIN
动作之后执行,影响到正常处理和END
动作; - 如果变量赋值在最后一个文件参数之后,在END动作之前执行,仅影响
END
动作; - 如果文件参数不存在,情况同1所述;
- 如果变量赋值位于多个文件参数之间,在变量赋值前面的文件被处理后执行,影响到后续文件的处理和
END
动作;
所以变量赋值一定要考虑清楚用途,否则比较容易出错,不过一般情况下也不会用到变量赋值。
自然地大家会将变量赋值与-v assignment选项进行比较,赋值的形式是一致的,但是-v选项的执行时机比变量赋值要早:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="pun">-</span><span class="pln">v </span><span class="kwd">var</span><span class="pun">=</span><span class="pln">a </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print "BEGIN: " var}'</span><span class="pln"> </span><span class="kwd">BEGIN</span><span class="pun">:</span><span class="pln"> a</span>
可见,-v选项的赋值操作在BEGIN
动作之前就执行了。
变量赋值一定要小心不要与保留关键字重名,否则会报错:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="pun">-</span><span class="pln">v </span><span class="kwd">BEGIN</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print "BEGIN: " BEGIN}'</span><span class="pln"> awk</span><span class="pun">:</span><span class="pln"> fatal</span><span class="pun">:</span><span class="pln"> cannot </span><span class="kwd">use</span><span class="pln"> gawk builtin </span><span class="str" style="color: rgb(221, 17, 68);">`BEGIN' as variable name</span>
记录(Record)与字段(Field)
对于数据库来说,一个数据库表是由多条记录组成的,每一行表示一条记录(Record)。每条记录由多列组成,每一列表示一个字段(Field)。Awk将一个文本文件视为一个文本数据库,因此它也有记录和字段的概念。默认情况下,记录的分隔符是回车,字段的分隔符是空白符,所以文本文件的每一行表示一个记录,而每一行中的内容被空白分隔成多个字段。利用字段和记录,awk就可以非常灵活地处理文件的内容。
可以通过-F选项来修改默认的字段分隔符,例如/etc/passwd的每一行都是由冒号分隔成多个字段的,所以这里就需要将分隔符设置成冒号:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="pun">-</span><span class="pln">F</span><span class="pun">:</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{print $1}'</span><span class="pln"> </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">passwd </span><span class="pun">|</span><span class="pln"> head </span><span class="pun">-</span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> root bin daemon</span>
这里通过$1引用第一人字段,类似地$2表示第二个字段,$3表示第三个字段.... $0则表示整个记录。内置变量NF记录着字段的个数,所以$NF表示最后一个字段:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="pun">-</span><span class="pln">F</span><span class="pun">:</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{print $NF}'</span><span class="pln"> </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">passwd </span><span class="pun">|</span><span class="pln"> head </span><span class="pun">-</span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="pun">/</span><span class="pln">bin</span><span class="pun">/</span><span class="pln">bash </span><span class="pun">/</span><span class="pln">bin</span><span class="pun">/</span><span class="kwd">false</span><span class="pln"> </span><span class="pun">/</span><span class="pln">bin</span><span class="pun">/</span><span class="kwd">false</span>
当然,$(NF-1)表示倒数第二个。
内置变量FS也可以用于更改字段分隔符,它记录着当前的字段分隔符:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="pun">-</span><span class="pln">F</span><span class="pun">:</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{print FS}'</span><span class="pln"> </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">passwd </span><span class="pun">|</span><span class="pln"> head </span><span class="pun">-</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="pun">:</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="pun">-</span><span class="pln">v FS</span><span class="pun">=:</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{print $1}'</span><span class="pln"> </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">passwd </span><span class="pun">|</span><span class="pln"> head </span><span class="pun">-</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> root</span>
记录的分隔符可以通过内置变量RS更改:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="pun">-</span><span class="pln">v RS</span><span class="pun">=:</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{print $0}'</span><span class="pln"> </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">passwd </span><span class="pun">|</span><span class="pln"> head </span><span class="pun">-</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> root</span>
如果将RS设置成空,行为有就一点怪异了,它会将连续不为空行的所有行(一个段落)当作一个记录,而且强制回车为字段分隔符:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat awk_man</span><span class="pun">.</span><span class="pln">txt </span><span class="typ" style="color: rgb(68, 85, 136);">The</span><span class="pln"> awk utility shall execute programs written </span><span class="kwd">in</span><span class="pln"> the awk programming language</span><span class="pun">,</span><span class="pln"> which </span><span class="kwd">is</span><span class="pln"> specialized </span><span class="kwd">for</span><span class="pln"> textual data manipulation</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">An</span><span class="pln"> awk program </span><span class="kwd">is</span><span class="pln"> a sequence of patterns </span><span class="kwd">and</span><span class="pln"> corresponding actions</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">When</span><span class="pln"> input </span><span class="kwd">is</span><span class="pln"> read that matches a pattern</span><span class="pun">,</span><span class="pln"> the action associated </span><span class="kwd">with</span><span class="pln"> that pattern </span><span class="kwd">is</span><span class="pln"> carried </span><span class="kwd">out</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Input</span><span class="pln"> shall be interpreted </span><span class="kwd">as</span><span class="pln"> a sequence of records</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">By</span><span class="pln"> </span><span class="kwd">default</span><span class="pun">,</span><span class="pln"> a record </span><span class="kwd">is</span><span class="pln"> a line</span><span class="pun">,</span><span class="pln"> less its terminating </span><span class="str" style="color: rgb(221, 17, 68);"><newline></span><span class="pun">,</span><span class="pln"> but </span><span class="kwd">this</span><span class="pln"> can be changed </span><span class="kwd">by</span><span class="pln"> </span><span class="kwd">using</span><span class="pln"> the RS built</span><span class="pun">-</span><span class="kwd">in</span><span class="pln"> variable</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Each</span><span class="pln"> record of input shall be matched </span><span class="kwd">in</span><span class="pln"> turn against each pattern </span><span class="kwd">in</span><span class="pln"> the program</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">For</span><span class="pln"> each pattern matched</span><span class="pun">,</span><span class="pln"> the associated action shall be executed</span><span class="pun">.</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {RS="";FS=":"} {print "First line: " $1}'</span><span class="pln"> awk_man</span><span class="pun">.</span><span class="pln">txt </span><span class="typ" style="color: rgb(68, 85, 136);">First</span><span class="pln"> line</span><span class="pun">:</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">The</span><span class="pln"> awk utility shall execute programs written </span><span class="kwd">in</span><span class="pln"> the awk programming language</span><span class="pun">,</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">First</span><span class="pln"> line</span><span class="pun">:</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Input</span><span class="pln"> shall be interpreted </span><span class="kwd">as</span><span class="pln"> a sequence of records</span><span class="pun">.</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">By</span><span class="pln"> </span><span class="kwd">default</span><span class="pun">,</span><span class="pln"> a record </span><span class="kwd">is</span><span class="pln"> a line</span><span class="pun">,</span>
这里,我们将变量赋值放到BEGIN
动作中执行,因为BEGIN
动作是在文件处理之前执行的,专门用于放初始化的语句。FS的赋值在这里是无效的,awk依然使用回车符来分隔字段。
脚本(Script)组成
命令行中的program部分,可以称为awk代码,也可以称为awk脚本。一段awk脚本是由多个'pattern { action }
'序列组成的。action是一个或者多个语句,它在输入行匹配pattern的时候被执行。如果pattern为空,表明这个action会在每一行处理时都会被执行。下面的例子简单地打印文件的每一行,这里不带任何参数的print语句打印的是整个记录,类似'print $0
':
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="pun">-</span><span class="pln">e </span><span class="str" style="color: rgb(221, 17, 68);">'line1\nline2'</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print}'</span><span class="pln"> line1 line2</span>
除了
,还可以在脚本中定义自定义的函数,函数定义格式如下所示:pattern { action }
<span class="kwd">function</span><span class="pln"> name</span><span class="pun">(</span><span class="pln">parameter list</span><span class="pun">)</span><span class="pln"> </span><span class="pun">{</span><span class="pln"> statements </span><span class="pun">}</span>
函数的参数列表用逗号分隔,参数默认是局部变量,无法在函数之外访问,而在函数中定义的变量为全局变量,可以在函数之外访问,如:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo line1 </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">' function t(a) { b=a; print a; } { print b; t("kodango.me"); print b; }'</span><span class="pln"> kodango</span><span class="pun">.</span><span class="pln">me kodango</span><span class="pun">.</span><span class="pln">me</span>
Awk脚本中的语句使用空行或者分号分隔,使用分号可以放在同一行,不过有时候会影响可读性,尤其是分支或循环结构中,很容易出错。
如果Awk中的一个语句太长,要分成多行,可以在行为使用反斜杠'\':
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat test</span><span class="pun">.</span><span class="pln">awk </span><span class="kwd">function</span><span class="pln"> t</span><span class="pun">(</span><span class="pln">a</span><span class="pun">)</span><span class="pln"> </span><span class="pun">{</span><span class="pln"> b</span><span class="pun">=</span><span class="pln">a </span><span class="kwd">print</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">"This is a very long line, so use backslash to escape the newline \ then we will print the variable a: a="</span><span class="pln"> a </span><span class="pun">}</span><span class="pln"> </span><span class="pun">{</span><span class="pln"> </span><span class="kwd">print</span><span class="pln"> b</span><span class="pun">;</span><span class="pln"> t</span><span class="pun">(</span><span class="str" style="color: rgb(221, 17, 68);">"kodango.me"</span><span class="pun">);</span><span class="pln"> </span><span class="kwd">print</span><span class="pln"> b</span><span class="pun">;}</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="pun">-</span><span class="pln">f test</span><span class="pun">.</span><span class="pln">awk </span><span class="typ" style="color: rgb(68, 85, 136);">This</span><span class="pln"> </span><span class="kwd">is</span><span class="pln"> a very </span><span class="kwd">long</span><span class="pln"> line</span><span class="pun">,</span><span class="pln"> so </span><span class="kwd">use</span><span class="pln"> backslash to escape the newline </span><span class="kwd">then</span><span class="pln"> we will </span><span class="kwd">print</span><span class="pln"> the variable a</span><span class="pun">:</span><span class="pln"> a</span><span class="pun">=</span><span class="pln">kodango</span><span class="pun">.</span><span class="pln">me kodango</span><span class="pun">.</span><span class="pln">me</span>
这里我们将脚本写到文件中,并通过-f参数来指定。但是,在一些特殊符号之后,是可以直接换行的,例如", { && ||"。
模式(Pattern)
模式是awk中比较重要的一部分,它有以下几种情况:
/regular expression/
: 扩展的正则表达式(Extended Regular Expression), 关于ERE可以参考这篇文章;relational expression
: 关系表达式,例如大于、小于、等于,关系表达式结果为true表示匹配;BEGIN
: 特殊的模式,在第一个记录处理之前被执行,常用于初始化语句的执行;END
: 特殊的模式,在最后一个记录处理之前被执行,常用于输出汇总信息;pattern, pattern
:模式对,匹配两者之间的所有记录,类似sed的地址对;
例如查找匹配数字3的行:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ seq </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">20</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'/3/ {print}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">13</span>
相反地,可以在在正则表达式之前加上'!'表示不匹配:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ seq </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">5</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'!/3/ {print}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">4</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">5</span>
除了BEGIN
和END
这两个特殊的模式外,其余的模式都可以使用'&&'或者'||'运算符组合,前者表示逻辑与,后者表示逻辑或:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ seq </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">50</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'/3/ && /1/ {print}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">13</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">31</span>
前面的正则都是整行匹配,有时候仅仅需要匹配某个字符,这样我们可以用表达式$n ~ /ere/
:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'$1 ~ /ko/ {print}'</span><span class="pln"> </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">passwd kodango</span><span class="pun">:</span><span class="pln">x</span><span class="pun">:</span><span class="lit" style="color: rgb(68, 85, 136);">1000</span><span class="pun">:</span><span class="lit" style="color: rgb(68, 85, 136);">1000</span><span class="pun">::</span><span class="str" style="color: rgb(221, 17, 68);">/home/</span><span class="pln">kodango</span><span class="pun">:</span><span class="str" style="color: rgb(221, 17, 68);">/bin/</span><span class="pln">bash</span>
有时候我们只想显示特定和行,例如显示第一行:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ seq </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">5</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'NR==1 {print}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
正则表达式(Regular Expression)
和sed篇一样,这里我不会详细介绍正则表达式。因为正则表达式的内容介绍起来太麻烦,还是推荐同学阅读现有的文章(如Linux/Unix工具与正则表达式的POSIX规范),里面对各个流派的正则表达式归纳地很清楚了。
表达式(Expressions)
表达式可以由常量、变量、运算符和函数组成,常数和变量的值可以为字符串和数值。
Awk中的变量有三种类型:用户定义的变量,内置变量和字段变量。其中,内置变量名都是大写的。
变量并不非一定要被声明或者被初始化,一个变量默认的值是空字符串,只是在某些上下文上会隐式的自动转换成数字0(例如数学运算),记住awk中的变量是无类型的,不存在字符串变量还是数字变量的区别,只是有时候为了解说方便,才会这么说。(感谢网友@紫云妃的提醒)
字段变量可以用$n来引用,n的取值范围为[0,NF]。n可以为一个变量,例如$NF代码最后一个字段,而$(NF-1)表示倒数第二个字段。
数组
数组是一种特殊的变量,awk中的数组都是关联数组,它的下标都是字符串值(man手册中的原话是:All arrays in AWK are associative, i.e. indexed by string values),即使你使用的下标是一个数字,awk也会将下标隐式转换成字符串。所以容易给人一个误解,数组的下标可以是数字或者字符串。
数组的赋值很简单,下面将value赋值给数组下标为index的元素:
<span class="pln">array</span><span class="pun">[</span><span class="pln">index</span><span class="pun">]=</span><span class="pln">value</span>
可以用for..in..语法遍历数组元素,其中item是数组元素对应的下标:
<span class="kwd">for</span><span class="pln"> </span><span class="pun">(</span><span class="pln">item </span><span class="kwd">in</span><span class="pln"> array</span><span class="pun">)</span>
当然也可以在if分支判断中使用in操作符:
<span class="kwd">if</span><span class="pln"> </span><span class="pun">(</span><span class="pln">item </span><span class="kwd">in</span><span class="pln"> array</span><span class="pun">)</span>
一个完整的例子如下所示:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"1 2 3"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{ for (i=0;i<NF;i++) a[i]=i; } END { print 3 in a for (i in a) printf "%s: %s\n", i, a[i]; }'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span>
内置变量
Awk在内部维护了许多内置变量,或者称为系统变量,例如之前提到的FS
、RS
等等。常见的内置变量如下表所示
变量名 | 描述 |
---|---|
ARGC | 命令行参数的各个,即ARGV数组的长度 |
ARGV | 存放命令行参数 |
CONVFMT | 定义awk内部数值转换成字符串的格式,默认值为"%.6g" |
OFMT | 定义输出时数值转换成字符串的格式,默认值为"%.6g" |
ENVIRON | 存放系统环境变量的关联数组 |
FILENAME | 当前被处理的文件名 |
NR | 记录的总个数 |
FNR | 当前文件中的记录的总个数 |
FS | 字段分隔符,默认为空白 |
NF | 每个记录中字段的个数 |
RS | 记录的分隔符,默认为回车 |
OFS | 输出时字段的分隔符,默认为空白 |
ORS | 输出时记录的分隔符,默认为回车 |
RLENGTH | 被match函数匹配的子串长度 |
RSTART | 被match函数匹配的子串位于目标字符串的起始下标 |
下面主要介绍几个比较难理解的内置变量:
1. ARGV
与ARGC
ARGV
与ARGC
的意思比较好理解,就像C语言main(int argc, char **argv)
。ARGV
数组的下标从0开始到ARGC
-1,它存放的是命令行参数,并且排除命令行选项(例如-v/-f)以及program部分。因此事实上ARGV
只是存储argument的部分,即文件名(file)以及命令行变量赋值两部分的内容。
通过下面的例子可以大概了解ARGC与ARGV的用法:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { > for (i = 0; i < ARGC; i++) > print ARGV[i] > }'</span><span class="pln"> inventory</span><span class="pun">-</span><span class="pln">shipped BBS</span><span class="pun">-</span><span class="pln">list awk inventory</span><span class="pun">-</span><span class="pln">shipped BBS</span><span class="pun">-</span><span class="pln">list</span>
ARGV
的用法不仅限于此,它是可以修改的,可以更改数组元素的值,可以增加数组元素或者删除数组元素。
a. 更改ARGV
元素的值
假设我们有a, b两个文件,它们各有一行内容:file a和file b。现在利用ARGV,我们可以做到偷梁换柱:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{ARGV[1]="b"} {print}'</span><span class="pln"> a file b</span>
这里要注意ARGV[1]="b"
的引号不能缺少,否则ARGV[1]=b
会将变量b的值赋值给ARGV[1]
。
当awk处理完一个文件之后,它会从ARGV
的下一个元素获取参数,如果是一个文件则继续处理,如果是一个变量赋值则执行赋值操作:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{ARGV[1]="var=1"} {print var}'</span><span class="pln"> a b </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
为什么这里只打印一次变量值呢?可以回头再看看上一篇中介绍变量赋值的内容。
而当下一个元素为空时,则跳过不处理,这样可以避开处理某个文件:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{ARGV[1]=""} {print}'</span><span class="pln"> a b file b</span>
上面的例子中a这个文件就被跳过了。
而当下一个元素的值为"-"时,表明从标准输入读取内容:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{ARGV[1]="-"} {print}'</span><span class="pln"> a b a a </span><span class="com" style="color: rgb(153, 153, 136);"># --> 这里按下CTRL+D停止输入</span><span class="pln"> file b</span>
b. 删除ARGV
元素
删除ARGV
元素和将元素的值赋值为空的效果是一样的,它们都会跳转对某个参数的处理:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{delete ARGV[1]} {print}'</span><span class="pln"> a b file b</span>
删除数组元素可以用delete
语句。
c. 增加ARGV
元素
我第一次看到ARGV
变量的时候就在想,能不能利用ARGV
变量避免提供命令行参数,就像这样:
<span class="pln">awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{ARGV[1]="a";} {print}'</span>
但是事实上这样不行,awk会依然从标准输入中获取内容。下面的方法倒是可以,首先增加ARGC
的值,再增加ARGV
元素,我到现在也没搞懂这两者的区别:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN{ARGC+=1;ARGV[1]="a"} {print}'</span><span class="pln"> file a</span>
2. CONVFMT
与OFMT
Awk中允许数值到字符串相互转换,其中内置变量CONVFMT
定义了awk内部数值到字符串转换的格式,它的默认值为"%.6g":
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { printf "CONVFMT=%s, num=%f, str=%s\n", CONVFMT, 12.11, 12.11 }'</span><span class="pln"> CONVFMT</span><span class="pun">=%.</span><span class="lit" style="color: rgb(68, 85, 136);">6g</span><span class="pun">,</span><span class="pln"> num</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">12.110000</span><span class="pun">,</span><span class="pln"> str</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">12.11</span>
通过更改CONVFMT
,我们可以定义自己的转换格式:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { CONVFMT="%d"; printf "CONVFMT=%s, num=%f, str=%s\n", CONVFMT, 12.11, 12.11 }'</span><span class="pln"> CONVFMT</span><span class="pun">=%</span><span class="pln">d</span><span class="pun">,</span><span class="pln"> num</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">12.110000</span><span class="pun">,</span><span class="pln"> str</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">12</span>
与此对应地还有一个内置变量OFMT,它与CONVFMT
的作用是类似的,只不过是影响输出的时候数字转换成字符串的格式:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { OFMT="%d";print 12.11 }'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">12</span>
3. ENVIRON
ENVIRON
是一个存放系统环境变量的关联数组,它的下标是环境变量名称,值是相应环境变量的值。例如:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { print ENVIRON["USER"] }'</span><span class="pln"> kodango</span>
利用环境变量也可以将值传递给awk:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ U</span><span class="pun">=</span><span class="pln">hello awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { print ENVIRON["U"] }'</span><span class="pln"> hello</span>
可以利用for..in循环遍历ENVIRON
数组:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { for (env in ENVIRON) printf "%s=%s\n", env, ENVIRON[env]; }'</span>
4. RLENGTH
与RSTART
RLENGTH
与RSTART
都是与match
函数相关的,前者表示匹配的子串长度,后者表示匹配的子串位于目标字符串的起始下标。例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {match("hello,world", /llo/); print RSTART,RLENGTH}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span>
关于match
函数,我们会在以后介绍。
运算符
表达式中必然少不了运算符,awk支持的运算符可以参见man手册中的“Expressions in awk”一小节内容:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ man awk </span><span class="pun">|</span><span class="pln"> grep </span><span class="str" style="color: rgb(221, 17, 68);">"^ *Table: Expressions in"</span><span class="pln"> </span><span class="pun">-</span><span class="pln">A </span><span class="lit" style="color: rgb(68, 85, 136);">42</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> sed </span><span class="str" style="color: rgb(221, 17, 68);">'s/^ *//'</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Table</span><span class="pun">:</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Expressions</span><span class="pln"> </span><span class="kwd">in</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Decreasing</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Precedence</span><span class="pln"> </span><span class="kwd">in</span><span class="pln"> awk </span><span class="typ" style="color: rgb(68, 85, 136);">Syntax</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Name</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Type</span><span class="pln"> of </span><span class="typ" style="color: rgb(68, 85, 136);">Result</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Associativity</span><span class="pln"> </span><span class="pun">(</span><span class="pln"> expr </span><span class="pun">)</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Grouping</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Type</span><span class="pln"> of expr N</span><span class="pun">/</span><span class="pln">A $expr </span><span class="typ" style="color: rgb(68, 85, 136);">Field</span><span class="pln"> reference </span><span class="typ" style="color: rgb(68, 85, 136);">String</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A </span><span class="pun">++</span><span class="pln"> lvalue </span><span class="typ" style="color: rgb(68, 85, 136);">Pre</span><span class="pun">-</span><span class="pln">increment </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A </span><span class="pun">--</span><span class="pln"> lvalue </span><span class="typ" style="color: rgb(68, 85, 136);">Pre</span><span class="pun">-</span><span class="pln">decrement </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A lvalue </span><span class="pun">++</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Post</span><span class="pun">-</span><span class="pln">increment </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A lvalue </span><span class="pun">--</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Post</span><span class="pun">-</span><span class="pln">decrement </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A expr </span><span class="pun">^</span><span class="pln"> expr </span><span class="typ" style="color: rgb(68, 85, 136);">Exponentiation</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Right</span><span class="pln"> </span><span class="pun">!</span><span class="pln"> expr </span><span class="typ" style="color: rgb(68, 85, 136);">Logical</span><span class="pln"> </span><span class="kwd">not</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A </span><span class="pun">+</span><span class="pln"> expr </span><span class="typ" style="color: rgb(68, 85, 136);">Unary</span><span class="pln"> plus </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A </span><span class="pun">-</span><span class="pln"> expr </span><span class="typ" style="color: rgb(68, 85, 136);">Unary</span><span class="pln"> minus </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> N</span><span class="pun">/</span><span class="pln">A expr </span><span class="pun">*</span><span class="pln"> expr </span><span class="typ" style="color: rgb(68, 85, 136);">Multiplication</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Numeric</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Left</span><span class="pln"> </span><span class="pun">...以下省略...</span>
语句(Statement)
到目前为止,用得比较多的语句就是print
,其它的还有printf、delete、break、continue、exit、next
等等。这些语句与函数不同的是,它们不会使用带括号的参数,并且没有返回值。不过也有意外,比如printf
就可以像函数一样的调用:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{printf("%s\n", "abc")}'</span><span class="pln"> abc</span>
break
和continue
语句,大家应该比较了解,分别用于跳出循环和跳到下一个循环。
delete
用于删除数组中的某个元素,这个我们在上面介绍ARGV
的时候也使用过。
exit
的用法顾名思义,就是退出awk的处理,然后会执行END
部分的内容:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo $</span><span class="str" style="color: rgb(221, 17, 68);">'line1\nline2'</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print;exit} END {print "exit.."}'</span><span class="pln"> line1 </span><span class="kwd">exit</span><span class="pun">..</span>
next
语句类似sed的n命令,它会读取下一条记录,并重新回到脚本的最开始处执行:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo $</span><span class="str" style="color: rgb(221, 17, 68);">'line1\nline2'</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{ > print "Before next.." > print $0 > next > print "After next.." > }'</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Before</span><span class="pln"> </span><span class="kwd">next</span><span class="pun">..</span><span class="pln"> line1 </span><span class="typ" style="color: rgb(68, 85, 136);">Before</span><span class="pln"> </span><span class="kwd">next</span><span class="pun">..</span><span class="pln"> line2</span>
从上面可以看出next
后面的print语句不会执行。
print与printf语句是使用最多的,它们将内容输出到标准输出。注意在print语句中,输出的变量之间带不带逗号是有区别的:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"1 2"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print $1, $2}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"1 2"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print $1 $2}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">12</span>
print输出时,字段之间的分隔符可以由OFS重新定义:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"1 2"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{OFS=";";print $1,$2}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pun">;</span><span class="lit" style="color: rgb(68, 85, 136);">2</span>
除此之外,print的输出还可以重定向到某个文件中或者某个命令:
<span class="kwd">print</span><span class="pln"> items </span><span class="pun">></span><span class="pln"> output</span><span class="pun">-</span><span class="pln">file </span><span class="kwd">print</span><span class="pln"> items </span><span class="pun">>></span><span class="pln"> output</span><span class="pun">-</span><span class="pln">file </span><span class="kwd">print</span><span class="pln"> items </span><span class="pun">|</span><span class="pln"> command</span>
假设有这一样一个文件,第一列是语句名称,第二列是对应的说明:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat column</span><span class="pun">.</span><span class="pln">txt statement</span><span class="pun">|</span><span class="pln">description </span><span class="kwd">delete</span><span class="pun">|</span><span class="kwd">delete</span><span class="pln"> item </span><span class="kwd">from</span><span class="pln"> an array </span><span class="kwd">exit</span><span class="pun">|</span><span class="kwd">exit</span><span class="pln"> </span><span class="kwd">from</span><span class="pln"> the awk process </span><span class="kwd">next</span><span class="pun">|</span><span class="pln">read </span><span class="kwd">next</span><span class="pln"> input record </span><span class="kwd">and</span><span class="pln"> process</span>
现在我们要将两列的内容分别输出到statement.txt和description.txt两个文件中:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="pun">-</span><span class="pln">F</span><span class="str" style="color: rgb(221, 17, 68);">'|'</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">'{ > print $1 > "statement.txt"; > print $2 > "description.txt" > }'</span><span class="pln"> column</span><span class="pun">.</span><span class="pln">txt </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat statement</span><span class="pun">.</span><span class="pln">txt statement </span><span class="kwd">delete</span><span class="pln"> </span><span class="kwd">exit</span><span class="pln"> </span><span class="kwd">next</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat description</span><span class="pun">.</span><span class="pln">txt description </span><span class="kwd">delete</span><span class="pln"> item </span><span class="kwd">from</span><span class="pln"> an array </span><span class="kwd">exit</span><span class="pln"> </span><span class="kwd">from</span><span class="pln"> the awk process read </span><span class="kwd">next</span><span class="pln"> input record </span><span class="kwd">and</span><span class="pln"> process</span>
下面是一个重定向到命令的例子,假设我们要对下面的文件进行排序:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ cat num</span><span class="pun">.</span><span class="pln">list </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">9</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">5</span>
可以通过将print的内容重定向到"sort -n"命令:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print | "sort -n"}'</span><span class="pln"> num</span><span class="pun">.</span><span class="pln">list </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">5</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">9</span>
printf命令的用法与print类似,也可以重定向到文件或者输出,只不过printf比print多了格式化字符串的功能。printf的语法也大多数语言包括bash的printf命令类似,这里就不多介绍了。
awk的函数分成数学函数、字符串函数、I/O处理函数以及用户自定义的函数,其中用户自定义的函数我们在上一篇中也有简单的介绍,下面我们一一来介绍这几类函数。
数学函数
awk中支持以下数学函数:
atan2(y,x)
:反正切函数;cos(x)
:余弦函数;sin(x)
:正弦函数;exp(x)
:以自然对数e为底指数函数;log(x)
:计算以e 为底的对数值;sqrt(x)
:绝对值函数;int(x)
:将数值转换成整数;rand()
:返回0到1的一个随机数值,不包含1;srand([expr])
:设置随机种子,一般与rand函数配合使用,如果参数为空,默认使用当前时间为种子;
例如,我们使用rand()
函数生成一个随机数值:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print rand(),rand();}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.237788</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.291066</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print rand(),rand();}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.237788</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.291066</span>
但是你会发现,每次awk执行都会生成同样的随机数,但是在一次执行过程中产生的随机数又是不同的。因为每次awk执行都使用了同样的种子,所以我们可以用srand()
函数来设置种子:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {srand();print rand(),rand();}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.171625</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.00692412</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {srand();print rand(),rand();}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.43269</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0.782984</span>
这样每次生成的随机数就不一样了。
利用rand()
函数我们也可以生成1到n的整数:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">' > function randint(n) { return int(n*rand()); } > BEGIN { srand(); print randint(10); > }'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span>
字符串函数
awk中包含大多数常见的字符串操作函数。
1. sub(ere, repl[, in])
描述:简单地说,就是将in中匹配ere的部分替换成repl,返回值是替换的次数。如果in参数省略,默认使用$0。替换的动作会直接修改变量的值。
下面是一个简单的替换的例子:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"hello, world"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print sub(/ello/, "i"); print}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> hi</span><span class="pun">,</span><span class="pln"> world</span>
在repl参数中&是一个元字符,它表示匹配的内容,例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {var="kodango"; sub(/kodango/, "hello, &", var); print var}'</span><span class="pln"> hello</span><span class="pun">,</span><span class="pln"> kodango</span>
2. gsub(ere, repl[, in])
描述:同sub()
函数功能类似,只不过是gsub()
是全局替换,即替换所有匹配的内容。
3. index(s, t)
描述:返回字符串t在s中出现的位置,注意这里位置是从1开始计算的,如果没有找到则返回0。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print index("kodango", "o")}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print index("kodango", "w")}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0</span>
4. length[([s])]
描述:返回字符串的长度,如果参数s没有指定,则默认使用$0作为参数。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print length('</span><span class="pln">kodango</span><span class="str" style="color: rgb(221, 17, 68);">');}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">0</span><span class="pln"> </span><span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ echo </span><span class="str" style="color: rgb(221, 17, 68);">"first line"</span><span class="pln"> </span><span class="pun">|</span><span class="pln"> awk </span><span class="str" style="color: rgb(221, 17, 68);">'{print length();}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">10</span>
5. match(s, ere)
描述: 返回字符串s匹配ere的起始位置,如果不匹配则返回0。该函数会定义RSTART
和RLENGTH
两个内置变量。RSTART
与返回值相同,RLENGTH
记录匹配子串的长度,如果不匹配则为-1。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { print match("kodango", /dango/); printf "Matched at: %d, Matched substr length: %d\n", RSTART, RLENGTH; }'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Matched</span><span class="pln"> at</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pun">,</span><span class="pln"> </span><span class="typ" style="color: rgb(68, 85, 136);">Matched</span><span class="pln"> substr length</span><span class="pun">:</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">5</span>
6. split(s, a[, fs])
描述:将字符串按照分隔符fs,分隔成多个部分,并存到数组a中。注意,存放的位置是从第1个数组元素开始的。如果fs为空,则默认使用FS分隔。函数返回值分隔的个数。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { > split("1;2;3;4;5", arr, ";") > for (i in arr) > printf "arr[%d]=%d\n", i, arr[i]; > }'</span><span class="pln"> arr</span><span class="pun">[</span><span class="lit" style="color: rgb(68, 85, 136);">4</span><span class="pun">]=</span><span class="lit" style="color: rgb(68, 85, 136);">4</span><span class="pln"> arr</span><span class="pun">[</span><span class="lit" style="color: rgb(68, 85, 136);">5</span><span class="pun">]=</span><span class="lit" style="color: rgb(68, 85, 136);">5</span><span class="pln"> arr</span><span class="pun">[</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pun">]=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> arr</span><span class="pun">[</span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pun">]=</span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> arr</span><span class="pun">[</span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pun">]=</span><span class="lit" style="color: rgb(68, 85, 136);">3</span>
这里有一个奇怪的地方是for..in..输出的数组不是按顺序输出的,如果要按顺序输出可以用常规的for循环:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { > split("1;2;3;4;5", arr, ";") > for (i=0;^C [kodango@devops ~]$ awk '</span><span class="kwd">BEGIN</span><span class="pln"> </span><span class="pun">{</span><span class="pln"> </span><span class="pun">></span><span class="pln"> n</span><span class="pun">=</span><span class="pln">split</span><span class="pun">(</span><span class="str" style="color: rgb(221, 17, 68);">"1;2;3;4;5"</span><span class="pun">,</span><span class="pln"> arr</span><span class="pun">,</span><span class="pln"> </span><span class="str" style="color: rgb(221, 17, 68);">";"</span><span class="pun">)</span><span class="pln"> </span><span class="pun">></span><span class="pln"> </span><span class="kwd">for</span><span class="pln"> </span><span class="pun">(</span><span class="pln">i</span><span class="pun">=</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pun">;</span><span class="pln"> i</span><span class="pun"><=</span><span class="pln">n</span><span class="pun">;</span><span class="pln"> i</span><span class="pun">++)</span><span class="pln"> </span><span class="pun">></span><span class="pln"> printf </span><span class="str" style="color: rgb(221, 17, 68);">"arr[%d]=%d\n"</span><span class="pun">,</span><span class="pln"> i</span><span class="pun">,</span><span class="pln"> arr</span><span class="pun">[</span><span class="pln">i</span><span class="pun">];</span><span class="pln"> </span><span class="pun">></span><span class="pln"> </span><span class="pun">}</span><span class="str" style="color: rgb(221, 17, 68);">' arr[1]=1 arr[2]=2 arr[3]=3 arr[4]=4 arr[5]=5</span>
7. sprintf(fmt, expr, expr, ...)
描述:类似printf,只不过不会将格式化后的内容输出到标准输出,而是当作返回值返回。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { > var=sprintf("%s=%s", "name", "value") > print var > }'</span><span class="pln"> name</span><span class="pun">=</span><span class="pln">value</span>
8. substr(s, m[, n])
描述:返回从位置m开始的,长度为n的子串,其中位置从1开始计算,如果未指定n或者n值大于剩余的字符个数,则子串一直到字符串末尾为止。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { print substr("kodango", 2, 3); }'</span><span class="pln"> oda </span><span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { print substr("kodango", 2); }'</span><span class="pln"> odango</span>
9. tolower(s)
描述:将字符串转换成小写字符。
例如:
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print tolower("KODANGO");}'</span><span class="pln"> kodango</span>
10. toupper(s)
描述:将字符串转换成大写字符。
例如
<span class="pun">[</span><span class="pln">kodango@devops </span><span class="pun">~]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {print tolower("kodango");}'</span><span class="pln"> KODANGO</span>
I/O处理函数
1. getline
getline
的用法相对比较复杂,它有几种不同的形式。不过它的主要作用就是从输入中每次获取一行输入。
a. expression | getline [var]
这种形式将前面管道前命令输出的结果作为getline
的输入,每次读取一行。如果后面跟有var,则将读取的内容保存到var变量中,否则会重新设置$0和NF
。
例如,我们将上面的statement.txt文件的内容显示作为getline
的输入:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { while("cat statement.txt" | getline var) print var}'</span><span class="pln"> statement </span><span class="kwd">delete</span><span class="pln"> </span><span class="kwd">exit</span><span class="pln"> </span><span class="kwd">next</span>
上面的例子中命令要用双引号,"cat statement.txt
",这一点同print/printf
是一样的。
如果不加var,则直接写到$0中,注意NF
值也会被更新:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { while("cat statement.txt" | getline) print $0,NF}'</span><span class="pln"> statement </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">delete</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">exit</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="kwd">next</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span>
b. getline [var]
第二种形式是直接使用getline
,它会从处理的文件中读取输入。同样地,如果var没有,则会设置$0,并且这时候会更新
NF
, NR
和FNR
:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'{ > while (getline) > print NF, NR, FNR, $0; > }'</span><span class="pln"> statement</span><span class="pun">.</span><span class="pln">txt </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pln"> </span><span class="kwd">delete</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3</span><span class="pln"> </span><span class="kwd">exit</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">4</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">4</span><span class="pln"> </span><span class="kwd">next</span>
c. getline [var] < expression
第三种形式从expression中重定向输入,与第一种方法类似,这里就不加赘述了。
2. close
close
函数可以用于关闭已经打开的文件或者管道,例如getline
函数的第一种形式用到管道,我们可以用close
函数把这个管道关闭,close
函数的参数与管道的命令一致:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN { while("cat statement.txt" | getline) { print $0; close("cat statement.txt"); }}'</span><span class="pln"> statement statement statement statement statement</span>
但是每次读了一行后,关闭管道,然后重新打开又重新读取第一行就死循环了。所以要慎用,一般情况下也很少会用到close
函数。
3. system
这个函数很简单,就是用于执行外部命令,例如:
<span class="pun">[</span><span class="pln">kodango@devops awk_temp</span><span class="pun">]</span><span class="pln">$ awk </span><span class="str" style="color: rgb(221, 17, 68);">'BEGIN {system("uname -r");}'</span><span class="pln"> </span><span class="lit" style="color: rgb(68, 85, 136);">3.6</span><span class="pun">.</span><span class="lit" style="color: rgb(68, 85, 136);">2</span><span class="pun">-</span><span class="lit" style="color: rgb(68, 85, 136);">1</span><span class="pun">-</span><span class="pln">ARCH</span>
结束语
快速了解Awk系列的几篇文章相对比较粗糙,我是参考Awk的man手册以及《Sed & wk》附录B总结而成的,但是应该可以让大家对awk有一个大致的了解,欢迎大家一起交流。
附Peteris Krumins在他博客上发布的一份cheet sheet:awk cheat sheet (.pdf)