linux 文件读取与处理

作业1:正则表达式应用 从因特网上搜索相关Web网页，处理网页html数据，从中提取出当前时间点北京各监测站的PM2.5浓度，输出格式如下。要求:写出各个处理步骤，并给出解释。
2018-03-15 13:00:00,海淀区万柳,73 
2018-03-15 13:00:00,昌平镇,67 
2018-03-15 13:00:00,奥体中心,66 
2018-03-15 14:00:00,海淀区万柳,73 
2018-03-15 14:00:00,昌平镇,73 
2018-03-15 14:00:00,奥体中心,75

两周以后上课时交。纸版。

Step 0 linux远程登录

YingLvs-MacBook-Pro:~ Ying$ ssh -p 22 ying@10.213.24.102
ying@10.213.24.102's password: 
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.2.0-42-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

97 packages can be updated.
86 updates are security updates.


WARNING: Security updates for your current Hardware Enablement Stack
ended on 2016-08-04:
 * http://wiki.ubuntu.com/1404_HWE_EOL

There is a graphics stack installed on this system. An upgrade to a
configuration supported for the full lifetime of the LTS will become
available on 2016-07-21 and can be installed by running 'update-manager'
in the Dash.
    
Last login: Wed Mar 28 12:41:46 2018 from 10.213.24.110

Step 1 保存网页

ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ curl http://www.86pm25.com/city/beijing.html -o 2018032918pm25_html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13223  100 13223    0     0   3009      0  0:00:04  0:00:04 --:--:--  3052

Step 2 查看所需网页内容

ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ cat 2018032918pm25_html | grep '2018' 
<div class="remark">更新：2018年03月29日 18时</div>
ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ cat 2018032916pm25_html|grep '<tr>'
<tr><td>奥体中心</td><td>99</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>45μg/m³</td><td>148μg/m³</td></tr>
<tr><td>昌平镇</td><td>106</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>41μg/m³</td><td>162μg/m³</td></tr>
<tr><td>定陵</td><td>125</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>41μg/m³</td><td>200μg/m³</td></tr>
<tr><td>东四</td><td>99</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>45μg/m³</td><td>148μg/m³</td></tr>
<tr><td>古城</td><td>103</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>38μg/m³</td><td>156μg/m³</td></tr>
<tr><td>官园</td><td>93</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>37μg/m³</td><td>135μg/m³</td></tr>
<tr><td>海淀区万柳</td><td>97</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>42μg/m³</td><td>143μg/m³</td></tr>
<tr><td>怀柔镇</td><td>115</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>65μg/m³</td><td>179μg/m³</td></tr>
<tr><td>农展馆</td><td>109</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>44μg/m³</td><td>168μg/m³</td></tr>
<tr><td>顺义新城</td><td>119</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>47μg/m³</td><td>188μg/m³</td></tr>
<tr><td>万寿西宫</td><td>87</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>38μg/m³</td><td>124μg/m³</td></tr></table>

cat 连接并列出文件内容 grep在文件中查找字符串

Step 3 使用awk输出所需内容

ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ cat 2018032916pm25_html| awk '/<td[.]*/{printf("%s",$0)};/<div class="remark">[^>]*<\/div>/{printf("%s",$2)}'
class="remark">更新：2018年03月29日<tr><td>奥体中心</td><td>99</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>45μg/m³</td><td>148μg/m³</td></tr><tr><td>昌平镇</td><td>106</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>41μg/m³</td><td>162μg/m³</td></tr><tr><td>定陵</td><td>125</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>41μg/m³</td><td>200μg/m³</td></tr><tr><td>东四</td><td>99</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>45μg/m³</td><td>148μg/m³</td></tr><tr><td>古城</td><td>103</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>38μg/m³</td><td>156μg/m³</td></tr><tr><td>官园</td><td>93</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>37μg/m³</td><td>135μg/m³</td></tr><tr><td>海淀区万柳</td><td>97</td><td><img src="../images/wurandengji/liang.gif"  /> </td><td>42μg/m³</td><td>143μg/m³</td></tr><tr><td>怀柔镇</td><td>115</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>65μg/m³</td><td>179μg/m³</td></tr><tr><td>农展馆</td><td>109</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>44μg/m³</td><td>168μg/m³</td></tr><tr><td>顺义新城</td><td>119</td><td><img src="../images/wurandengji/qing.gif"  /> </td><td>47μg/m³</td><td>188μg/m³</td></tr><tr><td>万寿西宫</td><td>87</ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ if"  /> </td><td>38μg/m³</td><td>124μg/m³</td></tr></table>

awk文本处理语言  

/<td[.]*/{printf("%s",$0)}

将数据所在行输出

/<div class="remark">[^>]*<\/div>/{printf("%s",$2)}

将日期所在行输出

[^>]*

表示所有不含">"符号的集合

Step 4 规格化处理日期 删除无用字符

ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ cat 2018032918pm25_html | sed -e 's/[0-9][0-9]*/ \0 /g' | awk '/<td[.]*/{printf("%s",$0)};/<div class="remark">[^>]*<\/div>/{printf("%s-%s-%s %s:00:00",$3,$5,$7,$9)}'|sed -e 's/<[^>]*>/ /g'  -e 's/μg\/m³//g' 
2018-03-29 18:00:00  奥体中心   109        35    168     昌平镇   94        27    137     定陵   111        30    172     东四   133        33    215     古城   114        33    177     官园   99        36    148     海淀区万柳   89        28    127     农展馆   137        50    224     顺义新城   95        34    139     万寿西宫   125        41    ying@ying-Lenovo-XiaoXin-CHAO7000:hom

 sed -e 's/[0-9][0-9]*/ \0 /g'

用于匹配数字串，将其与前后字符用空格隔开，

/<div class="remark">[^>]*<\/div>/{printf("%s-%s-%s %s:00:00",$3,$5,$7,$9)}'

选择处理后的日期行的第3，5，7，9列按格式输出

|sed -e 's/<[^>]*>/ /g'  -e 's/μg\/m³//g' 

删除所有标签和单位  

μg\/m³

匹配μg/m³因为/是匹配操作符，需要使用\转义

step 5 最终输出

ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ cat 2018032918pm25_html | sed -e 's/[0-9][0-9]*/ \0 /g' | awk '/<td[.]*/{printf("%s",$0)};/<div class="remark">[^>]*<\/div>/{printf("%s-%s-%s %s:00:00",$3,$5,$7,$9)}'|sed -e 's/<[^>]*>/ /g'  -e 's/μg\/m³//g' | awk /2018/{'for  (i=1;i<=10;i++) printf("%s %s,%s,%s\n",$1,$2,$(i*4-1),$(i*4+1))'}
2018-03-29 18:00:00,奥体中心,35
2018-03-29 18:00:00,昌平镇,27
2018-03-29 18:00:00,定陵,30
2018-03-29 18:00:00,东四,33
2018-03-29 18:00:00,古城,33
2018-03-29 18:00:00,官园,36
2018-03-29 18:00:00,海淀区万柳,28
2018-03-29 18:00:00,农展馆,50
2018-03-29 18:00:00,顺义新城,34
2018-03-29 18:00:00,万寿西宫,41

awk /2018/{'for  (i=1;i<=10;i++) printf("%s %s,%s,%s\n",$1,$2,$(i*4-1),$(i*4+1))'}

awk 匹配含有2018的行，循环输出，每行输出已处理好的日期，不同的地点和对应的pm2.5数值。

step 6 附 两天 pm10的对比  

ying@ying-Lenovo-XiaoXin-CHAO7000:homework01$ cat 2018032915pm25_html 2018032812pm25_html | sed -e 's/[0-9][0-9]*/ \0 /g' | awk '/<td[.]*/{printf("%s",$0)};/<div class="remark">[^>]*<\/div>/{printf("%s-%s-%s %s:00:00",$3,$5,$7,$9)}'|sed -e 's/<[^>]*>/ /g'  -e 's/μg\/m³//g' | awk /2018/{'for (j= 0;j<2;j++) for  (i=1;i<=11;i++) printf("%s %s,%s,%s\n",$(j*46+1),$(j*46+2),$(j*46+i*4-1),$(j*46+i*4+2))'}
2018-03-29 15:00:00,奥体中心,143
2018-03-29 15:00:00,昌平镇,183
2018-03-29 15:00:00,定陵,195
2018-03-29 15:00:00,古城,189
2018-03-29 15:00:00,官园,141
2018-03-29 15:00:00,海淀区万柳,143
2018-03-29 15:00:00,怀柔镇,165
2018-03-29 15:00:00,农展馆,137
2018-03-29 15:00:00,顺义新城,186
2018-03-29 15:00:00,天坛,158
2018-03-29 15:00:00,万寿西宫,140
2018-03-28 12:00:00,奥体中心,1241
2018-03-28 12:00:00,昌平镇,—
2018-03-28 12:00:00,定陵,1485
2018-03-28 12:00:00,东四,1333
2018-03-28 12:00:00,古城,1316
2018-03-28 12:00:00,官园,1298
2018-03-28 12:00:00,海淀区万柳,1028
2018-03-28 12:00:00,怀柔镇,919
2018-03-28 12:00:00,农展馆,1347
2018-03-28 12:00:00,顺义新城,1161
2018-03-28 12:00:00,天坛,1213

for (j= 0;j<2;j++) for  (i=1;i<=11;i++) printf("%s %s,%s,%s\n",$(j*46+1),$(j*46+2),$(j*46+i*4-1),$(j*46+i*4+2))'

嵌套的循环语句，在每个参数都加上了文件序号与文件列数的乘积，以得到正确的字符串序号。

附录：远程传输 scp

YingLvs-MacBook-Pro:linux Ying$ scp 2018032909pm25_html ying@10.213.24.105:/home/ying/Develop/linux/homework01
ying@10.213.24.105's password: 
2018032909pm25_html                           100%   13KB 549.5KB/s   00:00