<div class="details" id="article_details">
<div class="article_title">
<span class="ico ico_type_Original"></span>
<h1>
<span class="link_title"><a href="/ultrani/article/details/6750434">
日志分析查看——grep,sed,sort,awk运用
</a></span>
</h1>
</div>
<div class="article_manage clearfix">
<div class="article_l">
<span class="link_categories">
标签:
<a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e6%ad%a3%e5%88%99%e8%a1%a8%e8%be%be%e5%bc%8f">正则表达式</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e8%84%9a%e6%9c%ac">脚本</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e5%b7%a5%e5%85%b7">工具</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e4%bb%bb%e5%8a%a1">任务</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/linux">linux</a>
</span>
</div>
<div class="article_r">
<span class="link_postdate">2011-09-06 10:54</span>
<span title="阅读次数" class="link_view">9539人阅读</span>
<span title="评论次数" class="link_comments"> <a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_pinglun'])" href="#comments">评论</a>(1)</span>
<span data-mod="popu_171" class="link_collect tracking-ad"> <a title="收藏" οnclick="javascript:collectArticle('日志分析查看——grep,sed,sort,awk运用','6750434');return false;" href="javascript:void(0);" target="_blank">收藏</a></span>
<span class="link_report"> <a title="举报" οnclick="javascript:report(6750434,2);return false;" href="#report">举报</a></span>
</div>
</div>
<div id="embody" style="display:none" class="embody">
<span class="embody_t">本文章已收录于:</span>
<div value="{"err":0,"msg":"ok","data":[]}" id="lib" class="embody_c"></div>
</div>
<style type="text/css">
.embody{
padding:10px 10px 10px;
margin:0 -20px;
border-bottom:solid 1px #ededed;
}
.embody_b{
margin:0 ;
padding:10px 0;
}
.embody .embody_t,.embody .embody_c{
display: inline-block;
margin-right:10px;
}
.embody_t{
font-size: 12px;
color:#999;
}
.embody_c{
font-size: 12px;
}
.embody_c img,.embody_c em{
display: inline-block;
vertical-align: middle;
}
.embody_c img{
width:30px;
height:30px;
}
.embody_c em{
margin: 0 20px 0 10px;
color:#333;
font-style: normal;
}
</style>
<script type="text/javascript">
$(function () {
try
{
var lib = eval("("+$("#lib").attr("value")+")");
var html = "";
if (lib.err == 0) {
$.each(lib.data, function (i) {
var obj = lib.data[i];
//html += '<img src="' + obj.logo + '"/>' + obj.name + "&nbsp;&nbsp;";
html += ' <a href="' + obj.url + '" target="_blank">';
html += ' <img src="' + obj.logo + '">';
html += ' <em><b>' + obj.name + '</b></em>';
html += ' </a>';
});
if (html != "") {
setTimeout(function () {
$("#lib").html(html);
$("#embody").show();
}, 100);
}
}
} catch (err)
{ }
});
</script>
<div class="category clearfix">
<div class="category_l">
<img src="http://static.blog.csdn.net/images/category_icon.jpg">
<span>分类:</span>
</div>
<div class="category_r">
<label οnclick="GetCategoryArticles('833106','UltraNi','top','6750434');">
<span οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_fenlei']);">linux技巧<em>(11)</em></span>
<img style="display: inline;" src="http://static.blog.csdn.net/images/arrow_triangle _down.jpg" class="arrow-down">
<img style="display: none;" src="http://static.blog.csdn.net/images/arrow_triangle_up.jpg" class="arrow-up">
<div class="subItem" style="display: none;">
<div class="subItem_t"><a target="_blank" href="http://blog.csdn.net/UltraNi/article/category/833106">作者同类文章</a><i class="J_close">X</i></div>
<ul id="top_833106" class="subItem_l"><li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_10098481undefined5161631064593122" href="http://blog.csdn.net/UltraNi/article/details/10098481" title="linux配置本地tomcat应用80端口转发">linux配置本地tomcat应用80端口转发</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_9227371undefined5685636346680363" href="http://blog.csdn.net/UltraNi/article/details/9227371" title="Linux下最好用的pdf查看器:Okular">Linux下最好用的pdf查看器:Okular</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_9099083undefined2019710026048276" href="http://blog.csdn.net/UltraNi/article/details/9099083" title="批量scp脚本——从多台机器拷贝相同文件">批量scp脚本——从多台机器拷贝相同文件</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_8946638undefined024159835687114106" href="http://blog.csdn.net/UltraNi/article/details/8946638" title="win7 上 linux mint14 (64bit) 双系统安装过程">win7 上 linux mint14 (64bit) 双系统安装过程</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_8020201undefined3946088204180094" href="http://blog.csdn.net/UltraNi/article/details/8020201" title="远程终端链接解决vi中文乱码问题">远程终端链接解决vi中文乱码问题</a></li> <li style="padding-left: 300px;"><a target="_blank" href="http://blog.csdn.net/UltraNi/article/category/833106">更多</a></li></ul>
</div>
</label>
</div>
</div>
<script src="http://static.blog.csdn.net/scripts/category.js" type="text/javascript"></script>
<div class="bog_copyright">
<p class="copyright_p">版权声明:本文为博主原创文章,未经博主允许不得转载。</p>
</div>
<div class="article_content" id="article_content">
<p>概述</p>
<p> 我们日常应用中都离不开日志。可以说日志是我们在排查问题的一个重要依据。但是日志并不是写了就好了,当你想查看日志的时候,你会发现线上日志堆积的长度已经超越了你一行行浏览的耐性的极限了。于是,很有必要通过一些手段来高效地辅助你来快速的从日志中找到你要找的问题。本文通过一个从项目中衍生出来的例子从查找日志,筛选日志和统计日志3个方面层层递进来简述日志文件查看中一些有用的手段。(注:在linux环境下)</p>
<p><br>
</p>
<p>目录</p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t0">0.查找关键日志grep</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t1">1.查找关键日志grep</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t2">2.精简日志内容 sed</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t3">3.对记录进行排序sort</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t4">4.统计日志相关记录数 awk</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t5">5.日志规范化</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t6">6.一些容易遇到的问题</a></p>
<p> </p>
<p>例子背景:</p>
<p> 后台跑一个定时任务,对指定时间段的订单数据表中的每一条记录进行以此任务处理。在日志中输出:</p>
<p> 1.订单id</p>
<p> 2.订单处理状态</p>
<p> 3.日志类别</p>
<p>准备工具:sort, tail, less, uniqu,grep,sed,awk</p>
<p>示例日志:demo.log</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 1395px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=1&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_1" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_1"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>2011-08-23 19:57:00,610 [] INFO bo.CommodityCerOrderBO - =====>属性订正任务执行开始|每页读取100条数据 </span></span></li><li class=""><span>2011-08-23 19:57:05,012 [] INFO bo.CommodityCerOrderBO - 当前正在处理页数:1 </span></li><li class="alt"><span>2011-08-23 19:57:30,688 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids不含0跳过 </span></li><li class=""><span>2011-08-23 19:57:30,709 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104 </span></li><li class="alt"><span>2011-08-23 19:57:31,721 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105 </span></li><li class=""><span>2011-08-23 19:57:32,727 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107 </span></li><li class="alt"><span>2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids成功保存为0|100104|0|100105|100107 </span></li><li class=""><span>2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids不含0跳过 </span></li><li class="alt"><span>2011-08-23 19:57:32,805 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104 </span></li><li class=""><span>2011-08-23 19:57:33,828 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107 </span></li><li class="alt"><span>2011-08-23 19:57:33,838 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46 </span></li><li class=""><span>2011-08-23 19:57:34,850 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106 </span></li><li class="alt"><span>2011-08-23 19:57:35,860 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105 </span></li><li class=""><span>2011-08-23 19:57:36,871 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li><li class="alt"><span>2011-08-23 19:57:36,884 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li><li class=""><span>2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids成功保存为6|100104|0|0|100107|46|100106|100105|3|3 </span></li><li class="alt"><span>2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:attr_ids不含0跳过 </span></li><li class=""><span>2011-08-23 19:57:36,928 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li><li class="alt"><span>2011-08-23 19:57:36,942 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104 </span></li><li class=""><span>2011-08-23 19:57:36,955 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105 </span></li><li class="alt"><span>2011-08-23 19:57:36,969 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107 </span></li><li class=""><span>2011-08-23 19:57:36,980 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46 </span></li><li class="alt"><span>2011-08-23 19:57:36,992 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106 </span></li><li class=""><span>2011-08-23 19:57:37,011 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li></ol></div><pre class="plain" name="code" style="display: none;">2011-08-23 19:57:00,610 [] INFO bo.CommodityCerOrderBO - =====>属性订正任务执行开始|每页读取100条数据
2011-08-23 19:57:05,012 [] INFO bo.CommodityCerOrderBO - 当前正在处理页数:1
2011-08-23 19:57:30,688 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids不含0跳过
2011-08-23 19:57:30,709 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104
2011-08-23 19:57:31,721 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105
2011-08-23 19:57:32,727 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107
2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids成功保存为0|100104|0|100105|100107
2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids不含0跳过
2011-08-23 19:57:32,805 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104
2011-08-23 19:57:33,828 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107
2011-08-23 19:57:33,838 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46
2011-08-23 19:57:34,850 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106
2011-08-23 19:57:35,860 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105
2011-08-23 19:57:36,871 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3
2011-08-23 19:57:36,884 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3
2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids成功保存为6|100104|0|0|100107|46|100106|100105|3|3
2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:attr_ids不含0跳过
2011-08-23 19:57:36,928 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3
2011-08-23 19:57:36,942 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104
2011-08-23 19:57:36,955 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105
2011-08-23 19:57:36,969 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107
2011-08-23 19:57:36,980 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46
2011-08-23 19:57:36,992 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106
2011-08-23 19:57:37,011 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3</pre>
<p><br>
</p>
<p id="content_t0"><span style="font-size:18px"><strong>0.一些最基础的日志查看命令</strong></span></p>
<p>最简单的日志查看命令就是浏览日志文件了,一般会从有限浏览文件末尾的</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 2416px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=2&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_2" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_2"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>tail -400f demo.log #监控最后400行日志文件的变化 等价与 tail -n 400 -f (-f参数是实时) </span></span></li><li class=""><span>less demo.log #查看日志文件,支持上下滚屏,查找功能 </span></li><li class="alt"><span>uniq -c demo.log #标记该行重复的数量,不重复值为1 </span></li></ol></div><pre class="plain" name="code" style="display: none;">tail -400f demo.log #监控最后400行日志文件的变化 等价与 tail -n 400 -f (-f参数是实时)
less demo.log #查看日志文件,支持上下滚屏,查找功能
uniq -c demo.log #标记该行重复的数量,不重复值为1</pre>
<p></p>
<p>以上命令具体使用详见本机man手册</p>
<p><br>
</p>
<p id="content_t1"><span style="font-size:18px"><strong>1.查找关键日志记录 grep</strong></span></p>
<p>浏览了日志文件后你会发现,日志文件成千上万行,怎么能找到我要找的内容呢。这时候,就可已用grep来进行日志的关键行提取了。</p>
<p><strong>grep 简单使用</strong></p>
<p>规则:<em>grep [选项]...模式 [文件]... </em> (模式是正则表达式)<br>
</p>
<p><br>
</p>
<p><span style="font-size:16px">例子1:</span></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 2871px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=3&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_3" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_3"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep 'INFO' demo.log #在文件demo.log中查找所有包行INFO的行 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep 'INFO' demo.log #在文件demo.log中查找所有包行INFO的行</pre>
<p>输出:<br>
2011-08-23 19:57:00,610 [] INFO bo.CommodityCerOrderBO - =====>属性订正任务执行开始|每页读取100条数据<br>
2011-08-23 19:57:05,012 [] INFO bo.CommodityCerOrderBO - 当前正在处理页数:1<br>
2011-08-23 19:57:30,688 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids不含0跳过<br>
...(略)</p>
<p><br>
</p>
<p><span style="font-size:16px">例子2:</span></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 3194px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=4&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_4" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_4"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep -o 'order-fix.curr_id:<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-1-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-1" style="width: 4.604em; display: inline-block;"><span style="display: inline-block; position: relative; width: 4.056em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.546em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-2"><span class="mo" id="MathJax-Span-3" style="font-family: MathJax_Main;">[</span><span class="mn" id="MathJax-Span-4" style="font-family: MathJax_Main;">0</span><span class="mo" id="MathJax-Span-5" style="font-family: MathJax_Main; padding-left: 0.222em;">−</span><span class="mn" id="MathJax-Span-6" style="font-family: MathJax_Main; padding-left: 0.222em;">9</span><span class="mo" id="MathJax-Span-7" style="font-family: MathJax_Main;">]</span><span class="mtext" id="MathJax-Span-8" style="font-family: MathJax_Main; color: red;">\+</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-1">[0-9]\+</script>' demo.log #-o选项只提取order-fix.curr_id:xxx的内容(而不是一整行),并输出到屏幕上 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep -o 'order-fix.curr_id:\([0-9]\+\)' demo.log #-o选项只提取order-fix.curr_id:xxx的内容(而不是一整行),并输出到屏幕上</pre>输出:<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10226<br>
...(略)
<p></p>
<p><span style="font-size:16px">例子3:</span></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 3547px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=5&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_5" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_5"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep -c 'ERROR' demo.log #输出文件demo.log中查找所有包行ERROR的行的数量 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep -c 'ERROR' demo.log #输出文件demo.log中查找所有包行ERROR的行的数量</pre>
<p>输出:17</p>
<p><span style="font-size:16px">例子4:</span><br>
</p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 3700px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=6&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_6" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_6"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep -v 'ERROR' demo.log #查找不含"ERROR"的行 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep -v 'ERROR' demo.log #查找不含"ERROR"的行</pre>输出:(功能和grep 'INFO' demo.log 命令一样,输出略)<br>
<p><br>
</p>
<p><strong><u>grep 用法小结(转自网络图片):请点击直接查看<a target="_blank" href="http://pic002.cnblogs.com/images/2010/165814/2010110910285741.png">大图</a></u></strong></p>
<p><strong>详细用法请man之</strong></p>
<br>
<p id="content_t2"><span style="font-size:18px"><strong>2.精简日志内容 sed</strong></span></p>
<p> 从n多行的日志文件中提取到一定数量的行后,可能你还会觉得有些功能不够,比如你每行并不需要有哪个类抛出的描述,比如你不需要日志时间,或者要把时间格式换个形式展示等等,这时候你就可以通过sed的替换命令来进行对日志文件提取具体内容了。<br>
如果把grep比作过滤器,那sed就是个修改器了。 </p>
<p><strong>sed简单用法</strong>:</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 4171px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=7&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_7" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_7"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed [-n][-e] '命令' 文件 #-n选项是默认不输出信息,除非使用了p命令或者是s命令的p标志符;-e是表明空格后面接的是一个命令 </span></span></li><li class=""><span>sed [-n] -f 脚本 文件 #这个用法是把命令写在脚本里 </span></li></ol></div><pre class="plain" name="code" style="display: none;"> sed [-n][-e] '命令' 文件 #-n选项是默认不输出信息,除非使用了p命令或者是s命令的p标志符;-e是表明空格后面接的是一个命令
sed [-n] -f 脚本 文件 #这个用法是把命令写在脚本里</pre>
<p></p>
<p>»'命令'的格式: [地址1[,地址2]][!] 指令 [参数]<br>
» 地址的格式:用行号标识(1 表明匹配第一行),或者用正则表达式匹配('^INFO'表明该地址匹配以INFO打头的行) <br>
» 指令的例子:p打印指令,s替换指令,d删除指令等等(以下表格摘自abs的sed小册子):</p>
<p>
</p><table border="1" style="font-family:'AR PL UKai CN'; font-size:16px" class="CALSTABLE">
<thead>
<tr>
<th>操作符</th>
<th>名字</th>
<th>效果</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="OPTION">[地址范围]/p</code></td>
<td>打印</td>
<td>打印[指定的地址范围]</td>
</tr>
<tr>
<td><code class="OPTION">[地址范围]/d</code></td>
<td>删除</td>
<td>删除[指定的地址范围]</td>
</tr>
<tr>
<td><code class="OPTION">s/pattern1/pattern2/</code></td>
<td>替换</td>
<td>将指定行中, 将第一个匹配到的pattern1, 替换为pattern2.</td>
</tr>
<tr>
<td><code class="OPTION">[地址范围]/s/pattern1/pattern2/</code></td>
<td>替换</td>
<td>在<tt class="REPLACEABLE"><em>地址范围</em></tt>指定的每一行中, 将第一个匹配到的pattern1, 替换为pattern2.</td>
</tr>
<tr>
<td><code class="OPTION">[地址范围]/y/pattern1/pattern2/</code></td>
<td>transform</td>
<td>在<tt class="REPLACEABLE"><em>地址范围</em></tt>指定的每一行中, 将pattern1中的每个匹配到pattern2的字符都使用pattern2的相应字符作替换. (等价于<span style="font-weight:bold" class="COMMAND">tr</span>命令)</td>
</tr>
<tr>
<td><code class="OPTION">g</code></td>
<td>全局</td>
<td>在每个匹配的输入行中, 将<em>每个</em>模式匹配都作相应的操作. (译者注: 不只局限于第一个匹配)</td>
</tr>
</tbody>
</table>
<em>小结:sed就是遍历对于输入文件的每一行,如果该行匹配地址1,地址2的范围之内,那么就对这一行执行命令。</em><br>
<p></p>
<p><strong><span style="font-size:16px">例1:</span></strong>(摘自abs的sed小册子)</p>
<p>
</p><table border="1" style="font-family:'AR PL UKai CN'; font-size:16px" class="CALSTABLE">
<tbody>
<tr>
<td><code class="OPTION">8d</code></td>
<td>删除输入的第8行.</td>
</tr>
<tr>
<td><code class="OPTION">/^$/d</code></td>
<td>删除所有空行.</td>
</tr>
<tr>
<td><code class="OPTION">1,/^$/d</code></td>
<td>从输入的开头一直删除到第1个空行(第一个空行也删除掉).</td>
</tr>
<tr>
<td><code class="OPTION">/Jones/p</code></td>
<td>只打印那些包含<span class="QUOTE">"Jones"</span>的行(使用<span class="TOKEN">-n</span>选项).</td>
</tr>
<tr>
<td><code class="OPTION">s/Windows/Linux/</code></td>
<td>在每个输入行中, 将第一个出现的<span class="QUOTE">"Windows"</span>实例替换为<span class="QUOTE">"Linux"</span>.</td>
</tr>
<tr>
<td><code class="OPTION">s/BSOD/stability/g</code></td>
<td>在每个输入行中, 将所有<span class="QUOTE">"BSOD"</span>都替换为<span class="QUOTE">"stability"</span>.</td>
</tr>
<tr>
<td><code class="OPTION">s/ *$//</code></td>
<td>删除掉每行结尾的所有空格.</td>
</tr>
<tr>
<td><code class="OPTION">s/00*/0/g</code></td>
<td>将所有连续出现的0都压缩成单个的0.</td>
</tr>
<tr>
<td><code class="OPTION">/GUI/d</code></td>
<td>删除掉所有包含<span class="QUOTE">"GUI"</span>的行.</td>
</tr>
<tr>
<td><code class="OPTION">s/GUI//g</code></td>
<td>将所有<span class="QUOTE">"GUI"</span>都删除掉, 并保持剩余部分的完整性.</td>
</tr>
</tbody>
</table>
<br>
<p></p>
<p>看完基本用法,让我们结合demo.log来具体应用下:</p>
<p><span style="font-size:16px"><strong>例2</strong>:输出demo.log中的某个日期中的ERROR的行</span><br>
</p>
<p>来具体应用下:<br>
</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 5296px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=8&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_8" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_8"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed -n '/^2011-08-23.*ERROR/p' demolog.log </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">sed -n '/^2011-08-23.*ERROR/p' demolog.log</pre>输出:
<p></p>
<p>2011-08-23 19:57:30,709 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:31,721 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:32,727 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107<br>
<br>
</p>
<p><span style="font-size:16px"><strong>例3</strong>:提取demo.log中的日期,日志级别,订单id和状态。</span></p>
<p><span style="font-size:16px"></span></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 5646px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=9&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_9" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_9"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed -f demo.sed2 demo.log </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">sed -f demo.sed2 demo.log</pre><div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 5715px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=10&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_10" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_10"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#n #这一行用法和命令中的-n一样意思,就是默认不输出 </span></span></li><li class=""><span>#demo.sed2 </span></li><li class="alt"><span> </span></li><li class=""><span>#下面的一行是替换指令,就是把19位长的日期和INFO/ERROR,id,和后面的一截提取出来,然后用@分割符把这4个字段重新按顺序组合 </span></li><li class="alt"><span>s/^<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-2-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-9" style="width: 6.521em; display: inline-block;"><span style="display: inline-block; position: relative; width: 5.752em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.546em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-10"><span class="mo" id="MathJax-Span-11" style="font-family: MathJax_Main;">[</span><span class="mo" id="MathJax-Span-12" style="font-family: MathJax_Main;">−</span><span class="mspace" id="MathJax-Span-13" style="height: 0em; vertical-align: 0em; width: 0.222em; display: inline-block; overflow: hidden;"></span><span class="mn" id="MathJax-Span-14" style="font-family: MathJax_Main;">0</span><span class="mo" id="MathJax-Span-15" style="font-family: MathJax_Main; padding-left: 0.222em;">−</span><span class="mn" id="MathJax-Span-16" style="font-family: MathJax_Main; padding-left: 0.222em;">9</span><span class="mo" id="MathJax-Span-17" style="font-family: MathJax_Main;">]</span><span class="mo" id="MathJax-Span-18" style="font-family: MathJax_Main;">{</span><span class="mn" id="MathJax-Span-19" style="font-family: MathJax_Main;">19</span><span class="mo" id="MathJax-Span-20" style="font-family: MathJax_Main;">}</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-2">[-\: 0-9]\{19\}</script>.*<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-3-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-21" style="width: 8.188em; display: inline-block;"><span style="display: inline-block; position: relative; width: 7.227em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.638em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-22"><span class="mi" id="MathJax-Span-23" style="font-family: MathJax_Math; font-style: italic;">I<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.064em;"></span></span><span class="mi" id="MathJax-Span-24" style="font-family: MathJax_Math; font-style: italic;">N<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.085em;"></span></span><span class="mi" id="MathJax-Span-25" style="font-family: MathJax_Math; font-style: italic;">F<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.106em;"></span></span><span class="mi" id="MathJax-Span-26" style="font-family: MathJax_Math; font-style: italic;">O</span><span class="mo" id="MathJax-Span-27" style="font-family: MathJax_Main;">∥</span><span class="mi" id="MathJax-Span-28" style="font-family: MathJax_Math; font-style: italic;">E<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.026em;"></span></span><span class="mi" id="MathJax-Span-29" style="font-family: MathJax_Math; font-style: italic;">R</span><span class="mi" id="MathJax-Span-30" style="font-family: MathJax_Math; font-style: italic;">R</span><span class="mi" id="MathJax-Span-31" style="font-family: MathJax_Math; font-style: italic;">O</span><span class="mi" id="MathJax-Span-32" style="font-family: MathJax_Math; font-style: italic;">R</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-3">INFO\|ERROR</script> .*order-fix.curr_id:<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-4-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-33" style="width: 4.604em; display: inline-block;"><span style="display: inline-block; position: relative; width: 4.056em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.546em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-34"><span class="mo" id="MathJax-Span-35" style="font-family: MathJax_Main;">[</span><span class="mn" id="MathJax-Span-36" style="font-family: MathJax_Main;">0</span><span class="mo" id="MathJax-Span-37" style="font-family: MathJax_Main; padding-left: 0.222em;">−</span><span class="mn" id="MathJax-Span-38" style="font-family: MathJax_Main; padding-left: 0.222em;">9</span><span class="mo" id="MathJax-Span-39" style="font-family: MathJax_Main;">]</span><span class="mtext" id="MathJax-Span-40" style="font-family: MathJax_Main; color: red;">\+</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-4">[0-9]\+</script>,<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-5-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-41" style="width: 1.688em; display: inline-block;"><span style="display: inline-block; position: relative; width: 1.475em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.153em, -0.586em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-42"><span class="mo" id="MathJax-Span-43" style="font-family: MathJax_Main;">.</span><span class="mo" id="MathJax-Span-44" style="font-family: MathJax_Main; padding-left: 0.167em;">∗</span><span class="texatom" id="MathJax-Span-45"><span class="mrow" id="MathJax-Span-46"><span class="mo" id="MathJax-Span-47" style="font-family: MathJax_Main;">$</span></span></span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.077em; vertical-align: -0.147em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-5">.*$</script>/\1@\3@\2@\4/p </span></li></ol></div><pre class="plain" name="code" style="display: none;">#n #这一行用法和命令中的-n一样意思,就是默认不输出
#demo.sed2
#下面的一行是替换指令,就是把19位长的日期和INFO/ERROR,id,和后面的一截提取出来,然后用@分割符把这4个字段重新按顺序组合
s/^\([-\: 0-9]\{19\}\).*\(INFO\|ERROR\) .*order-fix.curr_id:\([0-9]\+\),\(.*$\)/\1@\3@\2@\4/p </pre>
<p>输出:</p>
2011-08-23 19:57:30@10117@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:30@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:31@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:32@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:32@10117@INFO@status:attr_ids成功保存为0|100104|0|100105|100107<span style="font-size:16px"><br>
</span>...略
<p><span style="font-size:16px"><br>
</span></p>
<p><strong>sed详细用法可以参考《sed 与 awk》(第二版), 或者man之</strong></p>
<p><strong>或者点击下面这个参考链接<a target="_blank" href="http://www.reddragonfly.org/abscn/x17814.html">http://www.reddragonfly.org/abscn/x17814.html</a></strong></p>
<p><span style="font-size:16px"><br>
</span></p>
<p id="content_t3"><strong><span style="font-size:18px">3.对记录进行排序 sort</span></strong></p>
<p> 经过了日志文件的精炼后,我们可能不想对日志进行时间排序,这时候我们就可以用sort进行排序。</p>
<p>基本使用</p>
<p><span style="font-size:18px"><strong><span style="font-family:'AR PL UKai CN'; font-size:16px"></span></strong></span></p>
<pre> sort [options] [file...]</pre>
<span style="font-size:18px"><strong></strong></span>
<p></p>
<p>对于demo.log,经过了上面的sed提取后,我希望先用id进行排序,然后再用日志级别倒序进行排序,最后才是日期排序</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 6552px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=11&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_11" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_11"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#排序功能 -t表示用@作为分割符,-k表示用分割出来的第几个域排序(不要漏掉后面的,2/,3/,1,详细意思看下面的参考链接,这里不做详述) </span></span></li><li class=""><span>sed -f test.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 #n为按数字排序,r为倒序 </span></li></ol></div><pre class="plain" name="code" style="display: none;">#排序功能 -t表示用@作为分割符,-k表示用分割出来的第几个域排序(不要漏掉后面的,2/,3/,1,详细意思看下面的参考链接,这里不做详述)
sed -f test.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 #n为按数字排序,r为倒序</pre>输出:
<p></p>
<p><span style="font-size:10px">2011-08-23 19:57:30@10117@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:32@10117@INFO@status:attr_ids成功保存为0|100104|0|100105|100107<br>
2011-08-23 19:57:30@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:31@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:32@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:36@10222@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46<br>
2011-08-23 19:57:37@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3<br>
2011-08-23 19:57:32@10226@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:36@10226@INFO@status:attr_ids成功保存为6|100104|0|0|100107|46|100106|100105|3|3<br>
2011-08-23 19:57:32@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:33@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:33@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46<br>
2011-08-23 19:57:34@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106<br>
2011-08-23 19:57:35@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:36@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3<br>
2011-08-23 19:57:36@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3</span><br>
<br>
</p>
<p><strong>详尽手册<a href="http://ss64.com/bash/sort.html">http://ss64.com/bash/sort.html</a></strong></p>
<p><br>
</p>
<p id="content_t4"><span style="font-size:18px"><strong>4.统计日志相关记录数 awk</strong></span></p>
<p>现在日志已经比较清晰了,但是如果我想对不同日志进行统计怎么办,比如我要统计所有ERROR的日志记录书,或者要统计每个订单有多少个ERROR?这就需要我们的awk帮忙了。</p>
<p><strong>awk简单使用:</strong></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 7557px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=12&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_12" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_12"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>awk [-v 变量名=变量值] [-Fre] [--] '模式 { 语句 }' 变量名=变量值 文件名 </span></span></li><li class=""><span>awk [-v 变量名=变量值] [-Fre] -f 脚本文件 [--] 变量名=变量值 文件名 </span></li></ol></div><pre class="plain" name="code" style="display: none;">awk [-v 变量名=变量值] [-Fre] [--] '模式 { 语句 }' 变量名=变量值 文件名
awk [-v 变量名=变量值] [-Fre] -f 脚本文件 [--] 变量名=变量值 文件名</pre>
<p></p>
<p>和sed一样,awk也支持2中方式调用,一种是把awk脚本直接在命令行写入,第二种是把awk写在文件中在命令行中调用。</p>
<p>awk处理方式也与sed类似,对文件中的每一个输入行进行处理,每个处理首先判断是否是模式中匹配的行,是的话就具体执行相应的语句。</p>
不同的是,awk侧重与对每一行的列进行处理,并且,awk脚本和c语言类似也拥有变量,条件判断,循环等复杂语句,所以这里只能简单介绍一下基本应用,详细的请查看后面给出的相关链接。
<p>而且,awk在处理所有行前和处理完行后各有BEGIN和END语句做预处理和后置处理。</p>
<p><strong><span style="font-size:16px">例子1:打印日志中的第2,3列</span></strong></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 7926px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=13&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_13" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_13"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>awk 'BEGIN{FS="@"} {print $2,$3}' demo.log_after_sort #BEGIN中预处理的是,把@号作为行的列分割符,把分割后的行的第2,3列输出 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">awk 'BEGIN{FS="@"} {print $2,$3}' demo.log_after_sort #BEGIN中预处理的是,把@号作为行的列分割符,把分割后的行的第2,3列输出</pre>输出:(对于从sort得出的结果作为输入)<br>
10117 INFO<br>
10117 INFO<br>
10117 ERROR<br>
10117 ERROR<br>
10117 ERROR<br>
10222 INFO<br>
...略<br>
<p></p>
<p><span style="font-size:16px"><strong>例子2. 统计日志中INFO,ERROR出现的总数,以及总记录数</strong></span></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 8279px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=14&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_14" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_14"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#下面的例子是作为命令行输入的,利用单引号作为换行标记,这样就不用另外把脚本写进文件调用了 </span></span></li><li class=""><span>awk ' </span></li><li class="alt"><span>BEGIN { </span></li><li class=""><span> FS="@" </span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>{ </span></li><li class=""><span> if ($3 == "INFO") {info_count++} </span></li><li class="alt"><span> if ($3 == "ERROR") {error_count++} </span></li><li class=""><span> </span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>END { </span></li><li class=""><span> print "order total count:"NR #NR是awk内置变量,是遍历的当前行号,到了END区域自然行号就等于总数了 </span></li><li class="alt"><span> printf("INFO count:%d ERROR count:%d\n",info_count,error_count) </span></li><li class=""><span>} ' demo.log_after_sort </span></li></ol></div><pre class="plain" name="code" style="display: none;">#下面的例子是作为命令行输入的,利用单引号作为换行标记,这样就不用另外把脚本写进文件调用了
awk '
BEGIN {
FS="@"
}
{
if ($3 == "INFO") {info_count++}
if ($3 == "ERROR") {error_count++}
}
END {
print "order total count:"NR #NR是awk内置变量,是遍历的当前行号,到了END区域自然行号就等于总数了
printf("INFO count:%d ERROR count:%d\n",info_count,error_count)
} ' demo.log_after_sort
</pre>
<p>输出:</p>
<p>order total count:22<br>
INFO count:5 ERROR count:17<br>
<br>
</p>
<p><span style="font-size:16px"><strong>例子3. 对指定时间范围内的日志进行统计,包括输出INFO,ERROR总数,记录总数,每个订单记录分类统计</strong></span><br>
</p>
<p>下面的例子综合了前面sed和sort</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 8878px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=15&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_15" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_15"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed -f demo.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 | awk -f demo.awk </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">sed -f demo.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 | awk -f demo.awk</pre>
<p></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 8947px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=16&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_16" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_16"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#demo.awk </span></span></li><li class=""><span>BEGIN { </span></li><li class="alt"><span> FS="@" </span></li><li class=""><span> stime="2011-08-23 19:57:31" </span></li><li class="alt"><span> etime="2011-08-23 19:57:37" </span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>$1 > stime && $1 < etime { </span></li><li class="alt"><span> if ($3 == "INFO") {info_count++} </span></li><li class=""><span> if ($3 == "ERROR") {error_count++} </span></li><li class="alt"><span> </span></li><li class=""><span> ++total </span></li><li class="alt"><span> </span></li><li class=""><span> status[$2]=status[$2]"\t"$1"\t"$3"\t"$4"\n" </span></li><li class="alt"><span> </span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>END { </span></li><li class="alt"><span> for(i in status){ </span></li><li class=""><span> printf("id:%s:\n%s\n",i,status[i]) </span></li><li class="alt"><span> } </span></li><li class=""><span> </span></li><li class="alt"><span> print "order total count:"total </span></li><li class=""><span> printf("INFO count:%d ERROR count:%d\n",info_count,error_count) </span></li><li class="alt"><span>} <span style="font-size:18px;"><strong> </span></li><li class=""><span></strong></span> </span></li></ol></div><pre class="plain" name="code" style="display: none;">#demo.awk
BEGIN {
FS="@"
stime="2011-08-23 19:57:31"
etime="2011-08-23 19:57:37"
}
$1 > stime && $1 < etime {
if ($3 == "INFO") {info_count++}
if ($3 == "ERROR") {error_count++}
++total
status[$2]=status[$2]"\t"$1"\t"$3"\t"$4"\n"
}
END {
for(i in status){
printf("id:%s:\n%s\n",i,status[i])
}
print "order total count:"total
printf("INFO count:%d ERROR count:%d\n",info_count,error_count)
} <span style="font-size:18px;"><strong>
</strong></span></pre>输出:
<p></p>
<p>id:10117:<br>
</p>
<blockquote style="margin:0pt 0pt 0pt 40px; border:medium none; padding:0px">
<p>2011-08-23 19:57:32<span style="white-space:pre"> </span>INFO status:attr_ids成功保存为0|100104|0|100105|100107<br>
2011-08-23 19:57:32<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107</p>
</blockquote>
<p>id:10226:</p>
<blockquote style="margin:0pt 0pt 0pt 40px; border:medium none; padding:0px">
<p>2011-08-23 19:57:32<span style="white-space:pre"> </span>INFO status:attr_ids不含0跳过<br>
2011-08-23 19:57:32<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:33<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:33<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46<br>
2011-08-23 19:57:34<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106<br>
2011-08-23 19:57:35<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105</p>
</blockquote>
<p><br>
</p>
<p>#这个例子只是举例说明awk的统计用法,实际运用中可能会统计超时的次数,页面访问次数等。</p>
<p><strong><br>
</strong></p>
<p><strong>awk相关资料:</strong></p>
<p>《sed 与 awk》(第二版)</p>
<p><a href="http://ww4.sinaimg.cn/large/69693b30gw1dy53b7yjqaj.jpg"> awk脑图</a></p>
<p><br>
</p>
<p>补充:</p>
<p>其他实践时例子:</p>
<p>1. 在本地分支把代码修改从一个分支复制到另一个分支(例子的<span style="font-family:monospace; white-space:pre">b1022st.txt是一个记录了文件新增或修改的变化的文件路径名</span>)</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 10408px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=17&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_17" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_17"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>awk 'BEGIN{FS="b1022-scm/"} {system("cp -rf b1022-scm/"$2" b1022-lpscm/"$2);}' /home/nizen/b1022st.txt </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">awk 'BEGIN{FS="b1022-scm/"} {system("cp -rf b1022-scm/"$2" b1022-lpscm/"$2);}' /home/nizen/b1022st.txt</pre>通过awk和其system命令结合,这样就把文件从b1022-scm复制到b1022-lpscm下<br>
<p><br>
</p>
<p>2.内置函数 <a href="http://www.cnblogs.com/chengmo/archive/2010/10/08/1845913.html">http://www.cnblogs.com/chengmo/archive/2010/10/08/1845913.html</a><br>
3.内建变量 <a href="http://www.linuxsong.org/2010/09/awk-built-in-variable/">http://www.linuxsong.org/2010/09/awk-built-in-variable/</a><br>
4.shell变量传递 <a href="http://www.51testing.com/?uid-225738-action-viewspace-itemid-246383">http://www.51testing.com/?uid-225738-action-viewspace-itemid-246383</a><br>
5.awk统计某个开始-结束范围内的关键字累加总数:</p>
<div class="dp-highlighter bg_java"><div class="bar"><div class="tools"><b>[java]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 550px; top: 10697px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=18&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_18" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_18"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1" class="dp-j"><li class="alt"><span><span>BEGIN { </span></span></li><li class=""><span> running=<span class="number">0</span><span> </span></span></li><li class="alt"><span> count=<span class="number">0</span><span> </span></span></li><li class=""><span> startRow=<span class="string">"begin =====>"</span><span> id #id,nextId是通过-v 参数从外部传入 </span></span></li><li class="alt"><span> endRow=<span class="string">"begin =====>"</span><span> nextId </span></span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>$<span class="number">0</span><span> ~ startRow{ # ~是匹配运算符,判断$</span><span class="number">0</span><span>是否满足startRow正则表达式 </span></span></li><li class="alt"><span> running = <span class="number">1</span><span> </span></span></li><li class=""><span># printf(<span class="string">"start\n"</span><span>) </span></span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>$<span class="number">0</span><span> ~ endRow { </span></span></li><li class=""><span> running = <span class="number">0</span><span> </span></span></li><li class="alt"><span># printf(<span class="string">"end\n"</span><span>) </span></span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>{ </span></li><li class="alt"><span> <span class="keyword">if</span><span>(running==</span><span class="number">1</span><span>) { # 仅在startRow 和 endRow 范围内统计 </span></span></li><li class=""><span> <span class="keyword">if</span><span>($</span><span class="number">0</span><span> ~ </span><span class="string">"it show"</span><span>) { </span></span></li><li class="alt"><span># printf($<span class="number">0</span><span> </span><span class="string">"\n"</span><span>) </span></span></li><li class=""><span> str=$<span class="number">0</span><span> </span></span></li><li class="alt"><span> sub(/^.*show times:/, <span class="string">""</span><span>, str) </span></span></li><li class=""><span> sub(/ .*$/, <span class="string">""</span><span>, str) </span></span></li><li class="alt"><span> printf(str <span class="string">"\n"</span><span>) </span></span></li><li class=""><span> count = count + str </span></li><li class="alt"><span> } </span></li><li class=""><span> } </span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>END { </span></li><li class=""><span> printf(<span class="string">"showTimeCount:"</span><span>+count) </span></span></li><li class="alt"><span>} </span></li></ol></div><pre class="java" name="code" style="display: none;">BEGIN {
running=0
count=0
startRow="begin =====>" id #id,nextId是通过-v 参数从外部传入
endRow="begin =====>" nextId
}
$0 ~ startRow{ # ~是匹配运算符,判断$0是否满足startRow正则表达式
running = 1
# printf("start\n")
}
$0 ~ endRow {
running = 0
# printf("end\n")
}
{
if(running==1) { # 仅在startRow 和 endRow 范围内统计
if($0 ~ "it show") {
# printf($0 "\n")
str=$0
sub(/^.*show times:/, "", str)
sub(/ .*$/, "", str)
printf(str "\n")
count = count + str
}
}
}
END {
printf("showTimeCount:"+count)
}</pre>6. printf "10ms occur:%.2lf%%\n",t10/total*100 #输出百分比数据,精确到百分位后2位<br>
<p></p>
<p><br>
</p>
<p id="content_t5"><strong><span style="font-size:18px">5.日志规范化</span></strong></p>
<p>从前面可以看出,日志文件为了要让后续工具能够对里面的内容进行提取和处理,就必须要让日志文件规范的输出。</p>
<p>个人想到有几个点可以规范:</p>
<p>1.记录日志时候可以写入一些特殊的文本语句,一遍与工具的检索和处理。</p>
<p>2.记录日志最好不要用中文,因为在不同语言环境下对日志的处理可能因为编码不同导致没法处理日志。</p>
<p>后面再贴下淘宝中找到的一些打印日志的建议:</p>
<p><span style="font-family:sans-serif; line-height:18px"></span></p>
<ul style="line-height:24px; font-size:14px; margin:0.3em 0px 0px 1.5em; padding:0px">
<li style="margin-bottom:0.1em">正常情况下应该返回true, 却返回false的, 反正就是你在对返回值进行检查的时候, 如果不正常, log一下</li><li style="margin-bottom:0.1em">出现异常的地方, 以前认为hsf.log会帮我们记下所有的异常, 但是这个也不一定可靠, 所以还得我们自己记一下</li><li style="margin-bottom:0.1em">日志必须包含上下文信息</li><li style="margin-bottom:0.1em">如果出于统计的需要, 可打可不打</li><li style="margin-bottom:0.1em">在完成代码之后, 查看一下整个代码结构, 在一些关键的点, 加上日志, 正常的info, 少数情况出现的warning, 异常情况的error或者warning</li><li style="margin-bottom:0.1em">打印的日志内容要容易查询, 以前我比较倾向于打中文日志, 虽然易读, 但是中文在linux下的搜索统计稍微有些麻烦,所以如果能加上英文标识(比如说用于唯一标识的前缀), 能识别不同日志, 这个对定位也是非常有好处的.</li></ul>
<br>
<p></p>
<p id="content_t6"><span style="font-size:18px"><strong>6.一些容易遇到的问题</strong></span></p>
<p><strong>a.处理中文出现乱码</strong></p>
<p>这个主要是因为你的linux locale的配置,与编辑文件的语言环境,还有你登录ssh客户端的编码规则有关,所以最好还是不用中文记录日志。</p>
<p><strong>b.正则表达式不同工具的区别</strong></p>
<p>这个主要是因为不同工具的正则表达式定义的元字符不同,网上有总结的,可点击<a target="_blank" href="http://blog.csdn.net/ultrani/article/details/6716727">正则迷雾</a>参考</p>
<p></p>
<p style="margin:4px 0px; padding:2px 0px"><strong><span style="font-size:18px">OO后记:</span></strong></p>
<p style="margin:4px 0px; padding:2px 0px">目前只是简单介绍了grep,sed,sort,awk的几个简单应用,实际上的日志监控回根据不同的情景进行不同的处理。比如需要对调用的耗时进行统计(平均时间或者超时记录),对访问量进行统计,但是基本原理都和本文例子出发点一致。本文一方面是为了记录下学习过程中积累的东西,另一方面为了抛砖引玉引起大家对日志记录的关注。</p>
<br>
</div>
<!-- Baidu Button BEGIN -->
<div data-mod="popu_172" style="float: right;" class="bdsharebuttonbox tracking-ad bdshare-button-style0-16" data-bd-bind="1466854806196">
<a style="background-position:0 0 !important; background-image: url(http://bdimg.share.baidu.com/static/api/img/share/icons_0_16.png?v=d754dcc0.png) !important" data-cmd="more" class="bds_more" href="#" target="_blank"></a>
<a style="background-position:0 -52px !important" title="分享到QQ空间" data-cmd="qzone" class="bds_qzone" href="#" target="_blank"></a>
<a style="background-position:0 -104px !important" title="分享到新浪微博" data-cmd="tsina" class="bds_tsina" href="#" target="_blank"></a>
<a style="background-position:0 -260px !important" title="分享到腾讯微博" data-cmd="tqq" class="bds_tqq" href="#" target="_blank"></a>
<a style="background-position:0 -208px !important" title="分享到人人网" data-cmd="renren" class="bds_renren" href="#" target="_blank"></a>
<a style="background-position:0 -1612px !important" title="分享到微信" data-cmd="weixin" class="bds_weixin" href="#" target="_blank"></a>
</div>
<script>window._bd_share_config = { "common": { "bdSnsKey": {}, "bdText": "", "bdMini": "1", "bdMiniList": false, "bdPic": "", "bdStyle": "0", "bdSize": "16" }, "share": {} }; with (document) 0[(getElementsByTagName('head')[0] || body).appendChild(createElement('script')).src = 'http://bdimg.share.baidu.com/static/api/js/share.js?v=89860593.js?cdnversion=' + ~(-new Date() / 36e5)];</script>
<!-- Baidu Button END -->
<link href="http://static.blog.csdn.net/css/blog_detail.css" rel="stylesheet">
<!--172.16.140.13-->
<!-- Baidu Button BEGIN -->
<script data="type=tools&uid=1536434" id="bdshare_js" type="text/javascript" src="http://bdimg.share.baidu.com/static/js/bds_s_v2.js?cdnversion=407460"></script>
<script type="text/javascript">
document.getElementById("bdshell_js").src = "http://bdimg.share.baidu.com/static/js/shell_v2.js?cdnversion=" + Math.ceil(new Date()/3600000)
</script>
<!-- Baidu Button END -->
<div articleid="6750434" id="digg">
<dl οnclick="btndigga();" class="digg digg_enable" id="btnDigg">
<dt>顶</dt>
<dd>2</dd>
</dl>
<dl οnclick="btnburya();" class="digg digg_enable" id="btnBury">
<dt>踩</dt>
<dd>0</dd>
</dl>
</div>
<div data-mod="popu_222" class="tracking-ad"><a href="javascript:void(0);" target="_blank"> </a> </div>
<div data-mod="popu_223" class="tracking-ad"> <a href="javascript:void(0);" target="_blank"> </a></div>
<script type="text/javascript">
function btndigga() {
$(".tracking-ad[data-mod='popu_222'] a").click();
}
function btnburya() {
$(".tracking-ad[data-mod='popu_223'] a").click();
}
</script>
<ul class="article_next_prev">
<li class="prev_article"><span οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_shangyipian']);location.href='/ultrani/article/details/6716727';">上一篇</span><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_shangyipian'])" href="/ultrani/article/details/6716727">正则迷雾</a></li>
<li class="next_article"><span οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_xiayipian']);location.href='/ultrani/article/details/6861400';">下一篇</span><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_xiayipian'])" href="/ultrani/article/details/6861400">java 网站 分类</a></li>
</ul>
<div style="clear:both; height:10px;"></div>
<div style="" class="similar_article">
<h4>我的同类文章</h4>
<div style="margin:20px 0px 0px 0px" class="similar_c">
<div class="similar_c_t">
<label class="similar_cur">
<span οnclick="GetCategoryArticles('833106','UltraNi','foot','6750434');" style="cursor:pointer">linux技巧<em>(11)</em></span>
</label>
</div>
<div style="max-height:195px;" data-mod="popu_141" class="similar_wrap tracking-ad">
<a style="display:none" href="http://blog.csdn.net" target="_blank">http://blog.csdn.net</a>
<ul class="similar_list fl"><li><em>•</em><a target="_blank" id="foot_aritcle_10098481undefined017966214710877337" href="http://blog.csdn.net/UltraNi/article/details/10098481" title="linux配置本地tomcat应用80端口转发">linux配置本地tomcat应用80端口转发</a><span>2013-08-20</span><label><i>阅读</i><b>4700</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_9099083undefined3779339262718392" href="http://blog.csdn.net/UltraNi/article/details/9099083" title="批量scp脚本——从多台机器拷贝相同文件">批量scp脚本——从多台机器拷贝相同文件</a><span>2013-06-14</span><label><i>阅读</i><b>3743</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_8020201undefined5338487479235362" href="http://blog.csdn.net/UltraNi/article/details/8020201" title="远程终端链接解决vi中文乱码问题">远程终端链接解决vi中文乱码问题</a><span>2012-09-26</span><label><i>阅读</i><b>1111</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6575327undefined9611503053016697" href="http://blog.csdn.net/UltraNi/article/details/6575327" title="sudo 技巧">sudo 技巧</a><span>2011-06-29</span><label><i>阅读</i><b>390</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6548899undefined4366808450972657" href="http://blog.csdn.net/UltraNi/article/details/6548899" title="vim技巧">vim技巧</a><span>2011-06-16</span><label><i>阅读</i><b>459</b></label></li> </ul>
<ul class="similar_list fr"><li><em>•</em><a target="_blank" id="foot_aritcle_9227371undefined9320868202719063" href="http://blog.csdn.net/UltraNi/article/details/9227371" title="Linux下最好用的pdf查看器:Okular">Linux下最好用的pdf查看器:Okular</a><span>2013-07-02</span><label><i>阅读</i><b>8259</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_8946638undefined46725670345209347" href="http://blog.csdn.net/UltraNi/article/details/8946638" title="win7 上 linux mint14 (64bit) 双系统安装过程">win7 上 linux mint14 (64bit) 双系统安装过程</a><span>2013-05-19</span><label><i>阅读</i><b>6347</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6716727undefined6292002285038891" href="http://blog.csdn.net/UltraNi/article/details/6716727" title="正则迷雾">正则迷雾</a><span>2011-08-24</span><label><i>阅读</i><b>1572</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6572841undefined19099063159691076" href="http://blog.csdn.net/UltraNi/article/details/6572841" title="SVN使用指南">SVN使用指南</a><span>2011-06-28</span><label><i>阅读</i><b>377</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6545906undefined5767895978910051" href="http://blog.csdn.net/UltraNi/article/details/6545906" title="linux 技巧摘要">linux 技巧摘要</a><span>2011-06-15</span><label><i>阅读</i><b>622</b></label></li> </ul>
<a class="MoreArticle" href="http://blog.csdn.net/UltraNi/article/category/833106">更多文章</a></div>
</div>
</div>
<script type="text/javascript">
$(function () {
GetCategoryArticles('833106', 'UltraNi','foot','6750434');
});
</script>
</div>
<div class="article_title">
<span class="ico ico_type_Original"></span>
<h1>
<span class="link_title"><a href="/ultrani/article/details/6750434">
日志分析查看——grep,sed,sort,awk运用
</a></span>
</h1>
</div>
<div class="article_manage clearfix">
<div class="article_l">
<span class="link_categories">
标签:
<a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e6%ad%a3%e5%88%99%e8%a1%a8%e8%be%be%e5%bc%8f">正则表达式</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e8%84%9a%e6%9c%ac">脚本</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e5%b7%a5%e5%85%b7">工具</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/%e4%bb%bb%e5%8a%a1">任务</a><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_tag']);" target="_blank" href="http://www.csdn.net/tag/linux">linux</a>
</span>
</div>
<div class="article_r">
<span class="link_postdate">2011-09-06 10:54</span>
<span title="阅读次数" class="link_view">9539人阅读</span>
<span title="评论次数" class="link_comments"> <a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_pinglun'])" href="#comments">评论</a>(1)</span>
<span data-mod="popu_171" class="link_collect tracking-ad"> <a title="收藏" οnclick="javascript:collectArticle('日志分析查看——grep,sed,sort,awk运用','6750434');return false;" href="javascript:void(0);" target="_blank">收藏</a></span>
<span class="link_report"> <a title="举报" οnclick="javascript:report(6750434,2);return false;" href="#report">举报</a></span>
</div>
</div>
<div id="embody" style="display:none" class="embody">
<span class="embody_t">本文章已收录于:</span>
<div value="{"err":0,"msg":"ok","data":[]}" id="lib" class="embody_c"></div>
</div>
<style type="text/css">
.embody{
padding:10px 10px 10px;
margin:0 -20px;
border-bottom:solid 1px #ededed;
}
.embody_b{
margin:0 ;
padding:10px 0;
}
.embody .embody_t,.embody .embody_c{
display: inline-block;
margin-right:10px;
}
.embody_t{
font-size: 12px;
color:#999;
}
.embody_c{
font-size: 12px;
}
.embody_c img,.embody_c em{
display: inline-block;
vertical-align: middle;
}
.embody_c img{
width:30px;
height:30px;
}
.embody_c em{
margin: 0 20px 0 10px;
color:#333;
font-style: normal;
}
</style>
<script type="text/javascript">
$(function () {
try
{
var lib = eval("("+$("#lib").attr("value")+")");
var html = "";
if (lib.err == 0) {
$.each(lib.data, function (i) {
var obj = lib.data[i];
//html += '<img src="' + obj.logo + '"/>' + obj.name + "&nbsp;&nbsp;";
html += ' <a href="' + obj.url + '" target="_blank">';
html += ' <img src="' + obj.logo + '">';
html += ' <em><b>' + obj.name + '</b></em>';
html += ' </a>';
});
if (html != "") {
setTimeout(function () {
$("#lib").html(html);
$("#embody").show();
}, 100);
}
}
} catch (err)
{ }
});
</script>
<div class="category clearfix">
<div class="category_l">
<img src="http://static.blog.csdn.net/images/category_icon.jpg">
<span>分类:</span>
</div>
<div class="category_r">
<label οnclick="GetCategoryArticles('833106','UltraNi','top','6750434');">
<span οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_fenlei']);">linux技巧<em>(11)</em></span>
<img style="display: inline;" src="http://static.blog.csdn.net/images/arrow_triangle _down.jpg" class="arrow-down">
<img style="display: none;" src="http://static.blog.csdn.net/images/arrow_triangle_up.jpg" class="arrow-up">
<div class="subItem" style="display: none;">
<div class="subItem_t"><a target="_blank" href="http://blog.csdn.net/UltraNi/article/category/833106">作者同类文章</a><i class="J_close">X</i></div>
<ul id="top_833106" class="subItem_l"><li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_10098481undefined5161631064593122" href="http://blog.csdn.net/UltraNi/article/details/10098481" title="linux配置本地tomcat应用80端口转发">linux配置本地tomcat应用80端口转发</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_9227371undefined5685636346680363" href="http://blog.csdn.net/UltraNi/article/details/9227371" title="Linux下最好用的pdf查看器:Okular">Linux下最好用的pdf查看器:Okular</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_9099083undefined2019710026048276" href="http://blog.csdn.net/UltraNi/article/details/9099083" title="批量scp脚本——从多台机器拷贝相同文件">批量scp脚本——从多台机器拷贝相同文件</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_8946638undefined024159835687114106" href="http://blog.csdn.net/UltraNi/article/details/8946638" title="win7 上 linux mint14 (64bit) 双系统安装过程">win7 上 linux mint14 (64bit) 双系统安装过程</a></li> <li data-mod="popu_140" class="tracking-ad"><em>•</em><a target="_blank" id="top_aritcle_8020201undefined3946088204180094" href="http://blog.csdn.net/UltraNi/article/details/8020201" title="远程终端链接解决vi中文乱码问题">远程终端链接解决vi中文乱码问题</a></li> <li style="padding-left: 300px;"><a target="_blank" href="http://blog.csdn.net/UltraNi/article/category/833106">更多</a></li></ul>
</div>
</label>
</div>
</div>
<script src="http://static.blog.csdn.net/scripts/category.js" type="text/javascript"></script>
<div class="bog_copyright">
<p class="copyright_p">版权声明:本文为博主原创文章,未经博主允许不得转载。</p>
</div>
<div class="article_content" id="article_content">
<p>概述</p>
<p> 我们日常应用中都离不开日志。可以说日志是我们在排查问题的一个重要依据。但是日志并不是写了就好了,当你想查看日志的时候,你会发现线上日志堆积的长度已经超越了你一行行浏览的耐性的极限了。于是,很有必要通过一些手段来高效地辅助你来快速的从日志中找到你要找的问题。本文通过一个从项目中衍生出来的例子从查找日志,筛选日志和统计日志3个方面层层递进来简述日志文件查看中一些有用的手段。(注:在linux环境下)</p>
<p><br>
</p>
<p>目录</p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t0">0.查找关键日志grep</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t1">1.查找关键日志grep</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t2">2.精简日志内容 sed</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t3">3.对记录进行排序sort</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t4">4.统计日志相关记录数 awk</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t5">5.日志规范化</a></p>
<p><a target="_self" href="http://blog.csdn.net/ultrani/article/details/6750434#content_t6">6.一些容易遇到的问题</a></p>
<p> </p>
<p>例子背景:</p>
<p> 后台跑一个定时任务,对指定时间段的订单数据表中的每一条记录进行以此任务处理。在日志中输出:</p>
<p> 1.订单id</p>
<p> 2.订单处理状态</p>
<p> 3.日志类别</p>
<p>准备工具:sort, tail, less, uniqu,grep,sed,awk</p>
<p>示例日志:demo.log</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 1395px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=1&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_1" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_1"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>2011-08-23 19:57:00,610 [] INFO bo.CommodityCerOrderBO - =====>属性订正任务执行开始|每页读取100条数据 </span></span></li><li class=""><span>2011-08-23 19:57:05,012 [] INFO bo.CommodityCerOrderBO - 当前正在处理页数:1 </span></li><li class="alt"><span>2011-08-23 19:57:30,688 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids不含0跳过 </span></li><li class=""><span>2011-08-23 19:57:30,709 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104 </span></li><li class="alt"><span>2011-08-23 19:57:31,721 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105 </span></li><li class=""><span>2011-08-23 19:57:32,727 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107 </span></li><li class="alt"><span>2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids成功保存为0|100104|0|100105|100107 </span></li><li class=""><span>2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids不含0跳过 </span></li><li class="alt"><span>2011-08-23 19:57:32,805 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104 </span></li><li class=""><span>2011-08-23 19:57:33,828 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107 </span></li><li class="alt"><span>2011-08-23 19:57:33,838 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46 </span></li><li class=""><span>2011-08-23 19:57:34,850 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106 </span></li><li class="alt"><span>2011-08-23 19:57:35,860 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105 </span></li><li class=""><span>2011-08-23 19:57:36,871 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li><li class="alt"><span>2011-08-23 19:57:36,884 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li><li class=""><span>2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids成功保存为6|100104|0|0|100107|46|100106|100105|3|3 </span></li><li class="alt"><span>2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:attr_ids不含0跳过 </span></li><li class=""><span>2011-08-23 19:57:36,928 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li><li class="alt"><span>2011-08-23 19:57:36,942 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104 </span></li><li class=""><span>2011-08-23 19:57:36,955 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105 </span></li><li class="alt"><span>2011-08-23 19:57:36,969 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107 </span></li><li class=""><span>2011-08-23 19:57:36,980 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46 </span></li><li class="alt"><span>2011-08-23 19:57:36,992 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106 </span></li><li class=""><span>2011-08-23 19:57:37,011 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3 </span></li></ol></div><pre class="plain" name="code" style="display: none;">2011-08-23 19:57:00,610 [] INFO bo.CommodityCerOrderBO - =====>属性订正任务执行开始|每页读取100条数据
2011-08-23 19:57:05,012 [] INFO bo.CommodityCerOrderBO - 当前正在处理页数:1
2011-08-23 19:57:30,688 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids不含0跳过
2011-08-23 19:57:30,709 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104
2011-08-23 19:57:31,721 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105
2011-08-23 19:57:32,727 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107
2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids成功保存为0|100104|0|100105|100107
2011-08-23 19:57:32,782 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids不含0跳过
2011-08-23 19:57:32,805 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104
2011-08-23 19:57:33,828 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107
2011-08-23 19:57:33,838 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46
2011-08-23 19:57:34,850 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106
2011-08-23 19:57:35,860 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105
2011-08-23 19:57:36,871 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3
2011-08-23 19:57:36,884 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3
2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10226,status:attr_ids成功保存为6|100104|0|0|100107|46|100106|100105|3|3
2011-08-23 19:57:36,891 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:attr_ids不含0跳过
2011-08-23 19:57:36,928 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3
2011-08-23 19:57:36,942 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104
2011-08-23 19:57:36,955 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105
2011-08-23 19:57:36,969 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107
2011-08-23 19:57:36,980 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46
2011-08-23 19:57:36,992 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106
2011-08-23 19:57:37,011 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10222,status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3</pre>
<p><br>
</p>
<p id="content_t0"><span style="font-size:18px"><strong>0.一些最基础的日志查看命令</strong></span></p>
<p>最简单的日志查看命令就是浏览日志文件了,一般会从有限浏览文件末尾的</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 2416px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=2&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_2" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_2"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>tail -400f demo.log #监控最后400行日志文件的变化 等价与 tail -n 400 -f (-f参数是实时) </span></span></li><li class=""><span>less demo.log #查看日志文件,支持上下滚屏,查找功能 </span></li><li class="alt"><span>uniq -c demo.log #标记该行重复的数量,不重复值为1 </span></li></ol></div><pre class="plain" name="code" style="display: none;">tail -400f demo.log #监控最后400行日志文件的变化 等价与 tail -n 400 -f (-f参数是实时)
less demo.log #查看日志文件,支持上下滚屏,查找功能
uniq -c demo.log #标记该行重复的数量,不重复值为1</pre>
<p></p>
<p>以上命令具体使用详见本机man手册</p>
<p><br>
</p>
<p id="content_t1"><span style="font-size:18px"><strong>1.查找关键日志记录 grep</strong></span></p>
<p>浏览了日志文件后你会发现,日志文件成千上万行,怎么能找到我要找的内容呢。这时候,就可已用grep来进行日志的关键行提取了。</p>
<p><strong>grep 简单使用</strong></p>
<p>规则:<em>grep [选项]...模式 [文件]... </em> (模式是正则表达式)<br>
</p>
<p><br>
</p>
<p><span style="font-size:16px">例子1:</span></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 2871px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=3&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_3" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_3"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep 'INFO' demo.log #在文件demo.log中查找所有包行INFO的行 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep 'INFO' demo.log #在文件demo.log中查找所有包行INFO的行</pre>
<p>输出:<br>
2011-08-23 19:57:00,610 [] INFO bo.CommodityCerOrderBO - =====>属性订正任务执行开始|每页读取100条数据<br>
2011-08-23 19:57:05,012 [] INFO bo.CommodityCerOrderBO - 当前正在处理页数:1<br>
2011-08-23 19:57:30,688 [] INFO bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:attr_ids不含0跳过<br>
...(略)</p>
<p><br>
</p>
<p><span style="font-size:16px">例子2:</span></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 3194px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=4&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_4" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_4"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep -o 'order-fix.curr_id:<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-1-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-1" style="width: 4.604em; display: inline-block;"><span style="display: inline-block; position: relative; width: 4.056em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.546em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-2"><span class="mo" id="MathJax-Span-3" style="font-family: MathJax_Main;">[</span><span class="mn" id="MathJax-Span-4" style="font-family: MathJax_Main;">0</span><span class="mo" id="MathJax-Span-5" style="font-family: MathJax_Main; padding-left: 0.222em;">−</span><span class="mn" id="MathJax-Span-6" style="font-family: MathJax_Main; padding-left: 0.222em;">9</span><span class="mo" id="MathJax-Span-7" style="font-family: MathJax_Main;">]</span><span class="mtext" id="MathJax-Span-8" style="font-family: MathJax_Main; color: red;">\+</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-1">[0-9]\+</script>' demo.log #-o选项只提取order-fix.curr_id:xxx的内容(而不是一整行),并输出到屏幕上 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep -o 'order-fix.curr_id:\([0-9]\+\)' demo.log #-o选项只提取order-fix.curr_id:xxx的内容(而不是一整行),并输出到屏幕上</pre>输出:<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10117<br>
order-fix.curr_id:10226<br>
...(略)
<p></p>
<p><span style="font-size:16px">例子3:</span></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 3547px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=5&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_5" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_5"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep -c 'ERROR' demo.log #输出文件demo.log中查找所有包行ERROR的行的数量 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep -c 'ERROR' demo.log #输出文件demo.log中查找所有包行ERROR的行的数量</pre>
<p>输出:17</p>
<p><span style="font-size:16px">例子4:</span><br>
</p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 3700px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=6&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_6" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_6"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>grep -v 'ERROR' demo.log #查找不含"ERROR"的行 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">grep -v 'ERROR' demo.log #查找不含"ERROR"的行</pre>输出:(功能和grep 'INFO' demo.log 命令一样,输出略)<br>
<p><br>
</p>
<p><strong><u>grep 用法小结(转自网络图片):请点击直接查看<a target="_blank" href="http://pic002.cnblogs.com/images/2010/165814/2010110910285741.png">大图</a></u></strong></p>
<p><strong>详细用法请man之</strong></p>
<br>
<p id="content_t2"><span style="font-size:18px"><strong>2.精简日志内容 sed</strong></span></p>
<p> 从n多行的日志文件中提取到一定数量的行后,可能你还会觉得有些功能不够,比如你每行并不需要有哪个类抛出的描述,比如你不需要日志时间,或者要把时间格式换个形式展示等等,这时候你就可以通过sed的替换命令来进行对日志文件提取具体内容了。<br>
如果把grep比作过滤器,那sed就是个修改器了。 </p>
<p><strong>sed简单用法</strong>:</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 4171px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=7&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_7" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_7"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed [-n][-e] '命令' 文件 #-n选项是默认不输出信息,除非使用了p命令或者是s命令的p标志符;-e是表明空格后面接的是一个命令 </span></span></li><li class=""><span>sed [-n] -f 脚本 文件 #这个用法是把命令写在脚本里 </span></li></ol></div><pre class="plain" name="code" style="display: none;"> sed [-n][-e] '命令' 文件 #-n选项是默认不输出信息,除非使用了p命令或者是s命令的p标志符;-e是表明空格后面接的是一个命令
sed [-n] -f 脚本 文件 #这个用法是把命令写在脚本里</pre>
<p></p>
<p>»'命令'的格式: [地址1[,地址2]][!] 指令 [参数]<br>
» 地址的格式:用行号标识(1 表明匹配第一行),或者用正则表达式匹配('^INFO'表明该地址匹配以INFO打头的行) <br>
» 指令的例子:p打印指令,s替换指令,d删除指令等等(以下表格摘自abs的sed小册子):</p>
<p>
</p><table border="1" style="font-family:'AR PL UKai CN'; font-size:16px" class="CALSTABLE">
<thead>
<tr>
<th>操作符</th>
<th>名字</th>
<th>效果</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="OPTION">[地址范围]/p</code></td>
<td>打印</td>
<td>打印[指定的地址范围]</td>
</tr>
<tr>
<td><code class="OPTION">[地址范围]/d</code></td>
<td>删除</td>
<td>删除[指定的地址范围]</td>
</tr>
<tr>
<td><code class="OPTION">s/pattern1/pattern2/</code></td>
<td>替换</td>
<td>将指定行中, 将第一个匹配到的pattern1, 替换为pattern2.</td>
</tr>
<tr>
<td><code class="OPTION">[地址范围]/s/pattern1/pattern2/</code></td>
<td>替换</td>
<td>在<tt class="REPLACEABLE"><em>地址范围</em></tt>指定的每一行中, 将第一个匹配到的pattern1, 替换为pattern2.</td>
</tr>
<tr>
<td><code class="OPTION">[地址范围]/y/pattern1/pattern2/</code></td>
<td>transform</td>
<td>在<tt class="REPLACEABLE"><em>地址范围</em></tt>指定的每一行中, 将pattern1中的每个匹配到pattern2的字符都使用pattern2的相应字符作替换. (等价于<span style="font-weight:bold" class="COMMAND">tr</span>命令)</td>
</tr>
<tr>
<td><code class="OPTION">g</code></td>
<td>全局</td>
<td>在每个匹配的输入行中, 将<em>每个</em>模式匹配都作相应的操作. (译者注: 不只局限于第一个匹配)</td>
</tr>
</tbody>
</table>
<em>小结:sed就是遍历对于输入文件的每一行,如果该行匹配地址1,地址2的范围之内,那么就对这一行执行命令。</em><br>
<p></p>
<p><strong><span style="font-size:16px">例1:</span></strong>(摘自abs的sed小册子)</p>
<p>
</p><table border="1" style="font-family:'AR PL UKai CN'; font-size:16px" class="CALSTABLE">
<tbody>
<tr>
<td><code class="OPTION">8d</code></td>
<td>删除输入的第8行.</td>
</tr>
<tr>
<td><code class="OPTION">/^$/d</code></td>
<td>删除所有空行.</td>
</tr>
<tr>
<td><code class="OPTION">1,/^$/d</code></td>
<td>从输入的开头一直删除到第1个空行(第一个空行也删除掉).</td>
</tr>
<tr>
<td><code class="OPTION">/Jones/p</code></td>
<td>只打印那些包含<span class="QUOTE">"Jones"</span>的行(使用<span class="TOKEN">-n</span>选项).</td>
</tr>
<tr>
<td><code class="OPTION">s/Windows/Linux/</code></td>
<td>在每个输入行中, 将第一个出现的<span class="QUOTE">"Windows"</span>实例替换为<span class="QUOTE">"Linux"</span>.</td>
</tr>
<tr>
<td><code class="OPTION">s/BSOD/stability/g</code></td>
<td>在每个输入行中, 将所有<span class="QUOTE">"BSOD"</span>都替换为<span class="QUOTE">"stability"</span>.</td>
</tr>
<tr>
<td><code class="OPTION">s/ *$//</code></td>
<td>删除掉每行结尾的所有空格.</td>
</tr>
<tr>
<td><code class="OPTION">s/00*/0/g</code></td>
<td>将所有连续出现的0都压缩成单个的0.</td>
</tr>
<tr>
<td><code class="OPTION">/GUI/d</code></td>
<td>删除掉所有包含<span class="QUOTE">"GUI"</span>的行.</td>
</tr>
<tr>
<td><code class="OPTION">s/GUI//g</code></td>
<td>将所有<span class="QUOTE">"GUI"</span>都删除掉, 并保持剩余部分的完整性.</td>
</tr>
</tbody>
</table>
<br>
<p></p>
<p>看完基本用法,让我们结合demo.log来具体应用下:</p>
<p><span style="font-size:16px"><strong>例2</strong>:输出demo.log中的某个日期中的ERROR的行</span><br>
</p>
<p>来具体应用下:<br>
</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 5296px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=8&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_8" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_8"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed -n '/^2011-08-23.*ERROR/p' demolog.log </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">sed -n '/^2011-08-23.*ERROR/p' demolog.log</pre>输出:
<p></p>
<p>2011-08-23 19:57:30,709 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:31,721 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:32,727 [] ERROR bo.CommodityCerOrderBO - order-fix.curr_id:10117,status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107<br>
<br>
</p>
<p><span style="font-size:16px"><strong>例3</strong>:提取demo.log中的日期,日志级别,订单id和状态。</span></p>
<p><span style="font-size:16px"></span></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 5646px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=9&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_9" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_9"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed -f demo.sed2 demo.log </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">sed -f demo.sed2 demo.log</pre><div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 5715px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=10&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_10" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_10"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#n #这一行用法和命令中的-n一样意思,就是默认不输出 </span></span></li><li class=""><span>#demo.sed2 </span></li><li class="alt"><span> </span></li><li class=""><span>#下面的一行是替换指令,就是把19位长的日期和INFO/ERROR,id,和后面的一截提取出来,然后用@分割符把这4个字段重新按顺序组合 </span></li><li class="alt"><span>s/^<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-2-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-9" style="width: 6.521em; display: inline-block;"><span style="display: inline-block; position: relative; width: 5.752em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.546em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-10"><span class="mo" id="MathJax-Span-11" style="font-family: MathJax_Main;">[</span><span class="mo" id="MathJax-Span-12" style="font-family: MathJax_Main;">−</span><span class="mspace" id="MathJax-Span-13" style="height: 0em; vertical-align: 0em; width: 0.222em; display: inline-block; overflow: hidden;"></span><span class="mn" id="MathJax-Span-14" style="font-family: MathJax_Main;">0</span><span class="mo" id="MathJax-Span-15" style="font-family: MathJax_Main; padding-left: 0.222em;">−</span><span class="mn" id="MathJax-Span-16" style="font-family: MathJax_Main; padding-left: 0.222em;">9</span><span class="mo" id="MathJax-Span-17" style="font-family: MathJax_Main;">]</span><span class="mo" id="MathJax-Span-18" style="font-family: MathJax_Main;">{</span><span class="mn" id="MathJax-Span-19" style="font-family: MathJax_Main;">19</span><span class="mo" id="MathJax-Span-20" style="font-family: MathJax_Main;">}</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-2">[-\: 0-9]\{19\}</script>.*<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-3-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-21" style="width: 8.188em; display: inline-block;"><span style="display: inline-block; position: relative; width: 7.227em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.638em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-22"><span class="mi" id="MathJax-Span-23" style="font-family: MathJax_Math; font-style: italic;">I<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.064em;"></span></span><span class="mi" id="MathJax-Span-24" style="font-family: MathJax_Math; font-style: italic;">N<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.085em;"></span></span><span class="mi" id="MathJax-Span-25" style="font-family: MathJax_Math; font-style: italic;">F<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.106em;"></span></span><span class="mi" id="MathJax-Span-26" style="font-family: MathJax_Math; font-style: italic;">O</span><span class="mo" id="MathJax-Span-27" style="font-family: MathJax_Main;">∥</span><span class="mi" id="MathJax-Span-28" style="font-family: MathJax_Math; font-style: italic;">E<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.026em;"></span></span><span class="mi" id="MathJax-Span-29" style="font-family: MathJax_Math; font-style: italic;">R</span><span class="mi" id="MathJax-Span-30" style="font-family: MathJax_Math; font-style: italic;">R</span><span class="mi" id="MathJax-Span-31" style="font-family: MathJax_Math; font-style: italic;">O</span><span class="mi" id="MathJax-Span-32" style="font-family: MathJax_Math; font-style: italic;">R</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-3">INFO\|ERROR</script> .*order-fix.curr_id:<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-4-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-33" style="width: 4.604em; display: inline-block;"><span style="display: inline-block; position: relative; width: 4.056em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.347em, -0.546em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-34"><span class="mo" id="MathJax-Span-35" style="font-family: MathJax_Main;">[</span><span class="mn" id="MathJax-Span-36" style="font-family: MathJax_Main;">0</span><span class="mo" id="MathJax-Span-37" style="font-family: MathJax_Main; padding-left: 0.222em;">−</span><span class="mn" id="MathJax-Span-38" style="font-family: MathJax_Main; padding-left: 0.222em;">9</span><span class="mo" id="MathJax-Span-39" style="font-family: MathJax_Main;">]</span><span class="mtext" id="MathJax-Span-40" style="font-family: MathJax_Main; color: red;">\+</span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.297em; vertical-align: -0.366em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-4">[0-9]\+</script>,<span class="MathJax_Preview"></span><span class="MathJax" id="MathJax-Element-5-Frame" role="textbox" aria-readonly="true" style=""><nobr><span class="math" id="MathJax-Span-41" style="width: 1.688em; display: inline-block;"><span style="display: inline-block; position: relative; width: 1.475em; height: 0px; font-size: 113%;"><span style="position: absolute; clip: rect(1.905em, 1000em, 3.153em, -0.586em); top: -2.876em; left: 0em;"><span class="mrow" id="MathJax-Span-42"><span class="mo" id="MathJax-Span-43" style="font-family: MathJax_Main;">.</span><span class="mo" id="MathJax-Span-44" style="font-family: MathJax_Main; padding-left: 0.167em;">∗</span><span class="texatom" id="MathJax-Span-45"><span class="mrow" id="MathJax-Span-46"><span class="mo" id="MathJax-Span-47" style="font-family: MathJax_Main;">$</span></span></span></span><span style="display: inline-block; width: 0px; height: 2.876em;"></span></span></span><span style="border-left: 0em solid; display: inline-block; overflow: hidden; width: 0px; height: 1.077em; vertical-align: -0.147em; color: rgb(255, 255, 255);"></span></span></nobr></span><script type="math/tex" id="MathJax-Element-5">.*$</script>/\1@\3@\2@\4/p </span></li></ol></div><pre class="plain" name="code" style="display: none;">#n #这一行用法和命令中的-n一样意思,就是默认不输出
#demo.sed2
#下面的一行是替换指令,就是把19位长的日期和INFO/ERROR,id,和后面的一截提取出来,然后用@分割符把这4个字段重新按顺序组合
s/^\([-\: 0-9]\{19\}\).*\(INFO\|ERROR\) .*order-fix.curr_id:\([0-9]\+\),\(.*$\)/\1@\3@\2@\4/p </pre>
<p>输出:</p>
2011-08-23 19:57:30@10117@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:30@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:31@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:32@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:32@10117@INFO@status:attr_ids成功保存为0|100104|0|100105|100107<span style="font-size:16px"><br>
</span>...略
<p><span style="font-size:16px"><br>
</span></p>
<p><strong>sed详细用法可以参考《sed 与 awk》(第二版), 或者man之</strong></p>
<p><strong>或者点击下面这个参考链接<a target="_blank" href="http://www.reddragonfly.org/abscn/x17814.html">http://www.reddragonfly.org/abscn/x17814.html</a></strong></p>
<p><span style="font-size:16px"><br>
</span></p>
<p id="content_t3"><strong><span style="font-size:18px">3.对记录进行排序 sort</span></strong></p>
<p> 经过了日志文件的精炼后,我们可能不想对日志进行时间排序,这时候我们就可以用sort进行排序。</p>
<p>基本使用</p>
<p><span style="font-size:18px"><strong><span style="font-family:'AR PL UKai CN'; font-size:16px"></span></strong></span></p>
<pre> sort [options] [file...]</pre>
<span style="font-size:18px"><strong></strong></span>
<p></p>
<p>对于demo.log,经过了上面的sed提取后,我希望先用id进行排序,然后再用日志级别倒序进行排序,最后才是日期排序</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 6552px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=11&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_11" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_11"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#排序功能 -t表示用@作为分割符,-k表示用分割出来的第几个域排序(不要漏掉后面的,2/,3/,1,详细意思看下面的参考链接,这里不做详述) </span></span></li><li class=""><span>sed -f test.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 #n为按数字排序,r为倒序 </span></li></ol></div><pre class="plain" name="code" style="display: none;">#排序功能 -t表示用@作为分割符,-k表示用分割出来的第几个域排序(不要漏掉后面的,2/,3/,1,详细意思看下面的参考链接,这里不做详述)
sed -f test.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 #n为按数字排序,r为倒序</pre>输出:
<p></p>
<p><span style="font-size:10px">2011-08-23 19:57:30@10117@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:32@10117@INFO@status:attr_ids成功保存为0|100104|0|100105|100107<br>
2011-08-23 19:57:30@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:31@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:32@10117@ERROR@status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:36@10222@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3<br>
2011-08-23 19:57:36@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46<br>
2011-08-23 19:57:37@10222@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3<br>
2011-08-23 19:57:32@10226@INFO@status:attr_ids不含0跳过<br>
2011-08-23 19:57:36@10226@INFO@status:attr_ids成功保存为6|100104|0|0|100107|46|100106|100105|3|3<br>
2011-08-23 19:57:32@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:33@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:33@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46<br>
2011-08-23 19:57:34@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106<br>
2011-08-23 19:57:35@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105<br>
2011-08-23 19:57:36@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3<br>
2011-08-23 19:57:36@10226@ERROR@status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:3</span><br>
<br>
</p>
<p><strong>详尽手册<a href="http://ss64.com/bash/sort.html">http://ss64.com/bash/sort.html</a></strong></p>
<p><br>
</p>
<p id="content_t4"><span style="font-size:18px"><strong>4.统计日志相关记录数 awk</strong></span></p>
<p>现在日志已经比较清晰了,但是如果我想对不同日志进行统计怎么办,比如我要统计所有ERROR的日志记录书,或者要统计每个订单有多少个ERROR?这就需要我们的awk帮忙了。</p>
<p><strong>awk简单使用:</strong></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 7557px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=12&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_12" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_12"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>awk [-v 变量名=变量值] [-Fre] [--] '模式 { 语句 }' 变量名=变量值 文件名 </span></span></li><li class=""><span>awk [-v 变量名=变量值] [-Fre] -f 脚本文件 [--] 变量名=变量值 文件名 </span></li></ol></div><pre class="plain" name="code" style="display: none;">awk [-v 变量名=变量值] [-Fre] [--] '模式 { 语句 }' 变量名=变量值 文件名
awk [-v 变量名=变量值] [-Fre] -f 脚本文件 [--] 变量名=变量值 文件名</pre>
<p></p>
<p>和sed一样,awk也支持2中方式调用,一种是把awk脚本直接在命令行写入,第二种是把awk写在文件中在命令行中调用。</p>
<p>awk处理方式也与sed类似,对文件中的每一个输入行进行处理,每个处理首先判断是否是模式中匹配的行,是的话就具体执行相应的语句。</p>
不同的是,awk侧重与对每一行的列进行处理,并且,awk脚本和c语言类似也拥有变量,条件判断,循环等复杂语句,所以这里只能简单介绍一下基本应用,详细的请查看后面给出的相关链接。
<p>而且,awk在处理所有行前和处理完行后各有BEGIN和END语句做预处理和后置处理。</p>
<p><strong><span style="font-size:16px">例子1:打印日志中的第2,3列</span></strong></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 7926px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=13&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_13" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_13"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>awk 'BEGIN{FS="@"} {print $2,$3}' demo.log_after_sort #BEGIN中预处理的是,把@号作为行的列分割符,把分割后的行的第2,3列输出 </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">awk 'BEGIN{FS="@"} {print $2,$3}' demo.log_after_sort #BEGIN中预处理的是,把@号作为行的列分割符,把分割后的行的第2,3列输出</pre>输出:(对于从sort得出的结果作为输入)<br>
10117 INFO<br>
10117 INFO<br>
10117 ERROR<br>
10117 ERROR<br>
10117 ERROR<br>
10222 INFO<br>
...略<br>
<p></p>
<p><span style="font-size:16px"><strong>例子2. 统计日志中INFO,ERROR出现的总数,以及总记录数</strong></span></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 8279px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=14&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_14" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_14"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#下面的例子是作为命令行输入的,利用单引号作为换行标记,这样就不用另外把脚本写进文件调用了 </span></span></li><li class=""><span>awk ' </span></li><li class="alt"><span>BEGIN { </span></li><li class=""><span> FS="@" </span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>{ </span></li><li class=""><span> if ($3 == "INFO") {info_count++} </span></li><li class="alt"><span> if ($3 == "ERROR") {error_count++} </span></li><li class=""><span> </span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>END { </span></li><li class=""><span> print "order total count:"NR #NR是awk内置变量,是遍历的当前行号,到了END区域自然行号就等于总数了 </span></li><li class="alt"><span> printf("INFO count:%d ERROR count:%d\n",info_count,error_count) </span></li><li class=""><span>} ' demo.log_after_sort </span></li></ol></div><pre class="plain" name="code" style="display: none;">#下面的例子是作为命令行输入的,利用单引号作为换行标记,这样就不用另外把脚本写进文件调用了
awk '
BEGIN {
FS="@"
}
{
if ($3 == "INFO") {info_count++}
if ($3 == "ERROR") {error_count++}
}
END {
print "order total count:"NR #NR是awk内置变量,是遍历的当前行号,到了END区域自然行号就等于总数了
printf("INFO count:%d ERROR count:%d\n",info_count,error_count)
} ' demo.log_after_sort
</pre>
<p>输出:</p>
<p>order total count:22<br>
INFO count:5 ERROR count:17<br>
<br>
</p>
<p><span style="font-size:16px"><strong>例子3. 对指定时间范围内的日志进行统计,包括输出INFO,ERROR总数,记录总数,每个订单记录分类统计</strong></span><br>
</p>
<p>下面的例子综合了前面sed和sort</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 8878px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=15&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_15" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_15"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>sed -f demo.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 | awk -f demo.awk </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">sed -f demo.sed demolog.log | sort -t@ -k2,2n -k3,3r -k1,1 | awk -f demo.awk</pre>
<p></p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 8947px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=16&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_16" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_16"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>#demo.awk </span></span></li><li class=""><span>BEGIN { </span></li><li class="alt"><span> FS="@" </span></li><li class=""><span> stime="2011-08-23 19:57:31" </span></li><li class="alt"><span> etime="2011-08-23 19:57:37" </span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>$1 > stime && $1 < etime { </span></li><li class="alt"><span> if ($3 == "INFO") {info_count++} </span></li><li class=""><span> if ($3 == "ERROR") {error_count++} </span></li><li class="alt"><span> </span></li><li class=""><span> ++total </span></li><li class="alt"><span> </span></li><li class=""><span> status[$2]=status[$2]"\t"$1"\t"$3"\t"$4"\n" </span></li><li class="alt"><span> </span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>END { </span></li><li class="alt"><span> for(i in status){ </span></li><li class=""><span> printf("id:%s:\n%s\n",i,status[i]) </span></li><li class="alt"><span> } </span></li><li class=""><span> </span></li><li class="alt"><span> print "order total count:"total </span></li><li class=""><span> printf("INFO count:%d ERROR count:%d\n",info_count,error_count) </span></li><li class="alt"><span>} <span style="font-size:18px;"><strong> </span></li><li class=""><span></strong></span> </span></li></ol></div><pre class="plain" name="code" style="display: none;">#demo.awk
BEGIN {
FS="@"
stime="2011-08-23 19:57:31"
etime="2011-08-23 19:57:37"
}
$1 > stime && $1 < etime {
if ($3 == "INFO") {info_count++}
if ($3 == "ERROR") {error_count++}
++total
status[$2]=status[$2]"\t"$1"\t"$3"\t"$4"\n"
}
END {
for(i in status){
printf("id:%s:\n%s\n",i,status[i])
}
print "order total count:"total
printf("INFO count:%d ERROR count:%d\n",info_count,error_count)
} <span style="font-size:18px;"><strong>
</strong></span></pre>输出:
<p></p>
<p>id:10117:<br>
</p>
<blockquote style="margin:0pt 0pt 0pt 40px; border:medium none; padding:0px">
<p>2011-08-23 19:57:32<span style="white-space:pre"> </span>INFO status:attr_ids成功保存为0|100104|0|100105|100107<br>
2011-08-23 19:57:32<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于认证分类参数有误默认取匹配属性名称的第一个属性id:100107</p>
</blockquote>
<p>id:10226:</p>
<blockquote style="margin:0pt 0pt 0pt 40px; border:medium none; padding:0px">
<p>2011-08-23 19:57:32<span style="white-space:pre"> </span>INFO status:attr_ids不含0跳过<br>
2011-08-23 19:57:32<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100104<br>
2011-08-23 19:57:33<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100107<br>
2011-08-23 19:57:33<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:46<br>
2011-08-23 19:57:34<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100106<br>
2011-08-23 19:57:35<span style="white-space:pre"> </span>ERROR status:添加属性id,但由于没有属性在该分类下默认取匹配属性名称的第一个属性id:100105</p>
</blockquote>
<p><br>
</p>
<p>#这个例子只是举例说明awk的统计用法,实际运用中可能会统计超时的次数,页面访问次数等。</p>
<p><strong><br>
</strong></p>
<p><strong>awk相关资料:</strong></p>
<p>《sed 与 awk》(第二版)</p>
<p><a href="http://ww4.sinaimg.cn/large/69693b30gw1dy53b7yjqaj.jpg"> awk脑图</a></p>
<p><br>
</p>
<p>补充:</p>
<p>其他实践时例子:</p>
<p>1. 在本地分支把代码修改从一个分支复制到另一个分支(例子的<span style="font-family:monospace; white-space:pre">b1022st.txt是一个记录了文件新增或修改的变化的文件路径名</span>)</p>
<p></p>
<div class="dp-highlighter bg_plain"><div class="bar"><div class="tools"><b>[plain]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 552px; top: 10408px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=17&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_17" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_17"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1"><li class="alt"><span><span>awk 'BEGIN{FS="b1022-scm/"} {system("cp -rf b1022-scm/"$2" b1022-lpscm/"$2);}' /home/nizen/b1022st.txt </span></span></li></ol></div><pre class="plain" name="code" style="display: none;">awk 'BEGIN{FS="b1022-scm/"} {system("cp -rf b1022-scm/"$2" b1022-lpscm/"$2);}' /home/nizen/b1022st.txt</pre>通过awk和其system命令结合,这样就把文件从b1022-scm复制到b1022-lpscm下<br>
<p><br>
</p>
<p>2.内置函数 <a href="http://www.cnblogs.com/chengmo/archive/2010/10/08/1845913.html">http://www.cnblogs.com/chengmo/archive/2010/10/08/1845913.html</a><br>
3.内建变量 <a href="http://www.linuxsong.org/2010/09/awk-built-in-variable/">http://www.linuxsong.org/2010/09/awk-built-in-variable/</a><br>
4.shell变量传递 <a href="http://www.51testing.com/?uid-225738-action-viewspace-itemid-246383">http://www.51testing.com/?uid-225738-action-viewspace-itemid-246383</a><br>
5.awk统计某个开始-结束范围内的关键字累加总数:</p>
<div class="dp-highlighter bg_java"><div class="bar"><div class="tools"><b>[java]</b> <a οnclick="dp.sh.Toolbar.Command('ViewSource',this);return false;" title="view plain" class="ViewSource" href="#">view plain</a><span data-mod="popu_168"> <a οnclick="dp.sh.Toolbar.Command('CopyToClipboard',this);return false;" title="copy" class="CopyToClipboard" href="#">copy</a><div style="position: absolute; left: 550px; top: 10697px; width: 18px; height: 18px; z-index: 99;"><embed width="18" height="18" align="middle" wmode="transparent" flashvars="id=18&width=18&height=18" pluginspage="http://www.macromedia.com/go/getflashplayer" type="application/x-shockwave-flash" allowfullscreen="false" allowscriptaccess="always" name="ZeroClipboardMovie_18" bgcolor="#ffffff" quality="best" menu="false" loop="false" src="http://static.blog.csdn.net/scripts/ZeroClipboard/ZeroClipboard.swf" id="ZeroClipboardMovie_18"></div></span><span data-mod="popu_169"> <a οnclick="dp.sh.Toolbar.Command('PrintSource',this);return false;" title="print" class="PrintSource" href="#">print</a></span><a οnclick="dp.sh.Toolbar.Command('About',this);return false;" title="?" class="About" href="#">?</a></div></div><ol start="1" class="dp-j"><li class="alt"><span><span>BEGIN { </span></span></li><li class=""><span> running=<span class="number">0</span><span> </span></span></li><li class="alt"><span> count=<span class="number">0</span><span> </span></span></li><li class=""><span> startRow=<span class="string">"begin =====>"</span><span> id #id,nextId是通过-v 参数从外部传入 </span></span></li><li class="alt"><span> endRow=<span class="string">"begin =====>"</span><span> nextId </span></span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>$<span class="number">0</span><span> ~ startRow{ # ~是匹配运算符,判断$</span><span class="number">0</span><span>是否满足startRow正则表达式 </span></span></li><li class="alt"><span> running = <span class="number">1</span><span> </span></span></li><li class=""><span># printf(<span class="string">"start\n"</span><span>) </span></span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>$<span class="number">0</span><span> ~ endRow { </span></span></li><li class=""><span> running = <span class="number">0</span><span> </span></span></li><li class="alt"><span># printf(<span class="string">"end\n"</span><span>) </span></span></li><li class=""><span>} </span></li><li class="alt"><span> </span></li><li class=""><span>{ </span></li><li class="alt"><span> <span class="keyword">if</span><span>(running==</span><span class="number">1</span><span>) { # 仅在startRow 和 endRow 范围内统计 </span></span></li><li class=""><span> <span class="keyword">if</span><span>($</span><span class="number">0</span><span> ~ </span><span class="string">"it show"</span><span>) { </span></span></li><li class="alt"><span># printf($<span class="number">0</span><span> </span><span class="string">"\n"</span><span>) </span></span></li><li class=""><span> str=$<span class="number">0</span><span> </span></span></li><li class="alt"><span> sub(/^.*show times:/, <span class="string">""</span><span>, str) </span></span></li><li class=""><span> sub(/ .*$/, <span class="string">""</span><span>, str) </span></span></li><li class="alt"><span> printf(str <span class="string">"\n"</span><span>) </span></span></li><li class=""><span> count = count + str </span></li><li class="alt"><span> } </span></li><li class=""><span> } </span></li><li class="alt"><span>} </span></li><li class=""><span> </span></li><li class="alt"><span>END { </span></li><li class=""><span> printf(<span class="string">"showTimeCount:"</span><span>+count) </span></span></li><li class="alt"><span>} </span></li></ol></div><pre class="java" name="code" style="display: none;">BEGIN {
running=0
count=0
startRow="begin =====>" id #id,nextId是通过-v 参数从外部传入
endRow="begin =====>" nextId
}
$0 ~ startRow{ # ~是匹配运算符,判断$0是否满足startRow正则表达式
running = 1
# printf("start\n")
}
$0 ~ endRow {
running = 0
# printf("end\n")
}
{
if(running==1) { # 仅在startRow 和 endRow 范围内统计
if($0 ~ "it show") {
# printf($0 "\n")
str=$0
sub(/^.*show times:/, "", str)
sub(/ .*$/, "", str)
printf(str "\n")
count = count + str
}
}
}
END {
printf("showTimeCount:"+count)
}</pre>6. printf "10ms occur:%.2lf%%\n",t10/total*100 #输出百分比数据,精确到百分位后2位<br>
<p></p>
<p><br>
</p>
<p id="content_t5"><strong><span style="font-size:18px">5.日志规范化</span></strong></p>
<p>从前面可以看出,日志文件为了要让后续工具能够对里面的内容进行提取和处理,就必须要让日志文件规范的输出。</p>
<p>个人想到有几个点可以规范:</p>
<p>1.记录日志时候可以写入一些特殊的文本语句,一遍与工具的检索和处理。</p>
<p>2.记录日志最好不要用中文,因为在不同语言环境下对日志的处理可能因为编码不同导致没法处理日志。</p>
<p>后面再贴下淘宝中找到的一些打印日志的建议:</p>
<p><span style="font-family:sans-serif; line-height:18px"></span></p>
<ul style="line-height:24px; font-size:14px; margin:0.3em 0px 0px 1.5em; padding:0px">
<li style="margin-bottom:0.1em">正常情况下应该返回true, 却返回false的, 反正就是你在对返回值进行检查的时候, 如果不正常, log一下</li><li style="margin-bottom:0.1em">出现异常的地方, 以前认为hsf.log会帮我们记下所有的异常, 但是这个也不一定可靠, 所以还得我们自己记一下</li><li style="margin-bottom:0.1em">日志必须包含上下文信息</li><li style="margin-bottom:0.1em">如果出于统计的需要, 可打可不打</li><li style="margin-bottom:0.1em">在完成代码之后, 查看一下整个代码结构, 在一些关键的点, 加上日志, 正常的info, 少数情况出现的warning, 异常情况的error或者warning</li><li style="margin-bottom:0.1em">打印的日志内容要容易查询, 以前我比较倾向于打中文日志, 虽然易读, 但是中文在linux下的搜索统计稍微有些麻烦,所以如果能加上英文标识(比如说用于唯一标识的前缀), 能识别不同日志, 这个对定位也是非常有好处的.</li></ul>
<br>
<p></p>
<p id="content_t6"><span style="font-size:18px"><strong>6.一些容易遇到的问题</strong></span></p>
<p><strong>a.处理中文出现乱码</strong></p>
<p>这个主要是因为你的linux locale的配置,与编辑文件的语言环境,还有你登录ssh客户端的编码规则有关,所以最好还是不用中文记录日志。</p>
<p><strong>b.正则表达式不同工具的区别</strong></p>
<p>这个主要是因为不同工具的正则表达式定义的元字符不同,网上有总结的,可点击<a target="_blank" href="http://blog.csdn.net/ultrani/article/details/6716727">正则迷雾</a>参考</p>
<p></p>
<p style="margin:4px 0px; padding:2px 0px"><strong><span style="font-size:18px">OO后记:</span></strong></p>
<p style="margin:4px 0px; padding:2px 0px">目前只是简单介绍了grep,sed,sort,awk的几个简单应用,实际上的日志监控回根据不同的情景进行不同的处理。比如需要对调用的耗时进行统计(平均时间或者超时记录),对访问量进行统计,但是基本原理都和本文例子出发点一致。本文一方面是为了记录下学习过程中积累的东西,另一方面为了抛砖引玉引起大家对日志记录的关注。</p>
<br>
</div>
<!-- Baidu Button BEGIN -->
<div data-mod="popu_172" style="float: right;" class="bdsharebuttonbox tracking-ad bdshare-button-style0-16" data-bd-bind="1466854806196">
<a style="background-position:0 0 !important; background-image: url(http://bdimg.share.baidu.com/static/api/img/share/icons_0_16.png?v=d754dcc0.png) !important" data-cmd="more" class="bds_more" href="#" target="_blank"></a>
<a style="background-position:0 -52px !important" title="分享到QQ空间" data-cmd="qzone" class="bds_qzone" href="#" target="_blank"></a>
<a style="background-position:0 -104px !important" title="分享到新浪微博" data-cmd="tsina" class="bds_tsina" href="#" target="_blank"></a>
<a style="background-position:0 -260px !important" title="分享到腾讯微博" data-cmd="tqq" class="bds_tqq" href="#" target="_blank"></a>
<a style="background-position:0 -208px !important" title="分享到人人网" data-cmd="renren" class="bds_renren" href="#" target="_blank"></a>
<a style="background-position:0 -1612px !important" title="分享到微信" data-cmd="weixin" class="bds_weixin" href="#" target="_blank"></a>
</div>
<script>window._bd_share_config = { "common": { "bdSnsKey": {}, "bdText": "", "bdMini": "1", "bdMiniList": false, "bdPic": "", "bdStyle": "0", "bdSize": "16" }, "share": {} }; with (document) 0[(getElementsByTagName('head')[0] || body).appendChild(createElement('script')).src = 'http://bdimg.share.baidu.com/static/api/js/share.js?v=89860593.js?cdnversion=' + ~(-new Date() / 36e5)];</script>
<!-- Baidu Button END -->
<link href="http://static.blog.csdn.net/css/blog_detail.css" rel="stylesheet">
<!--172.16.140.13-->
<!-- Baidu Button BEGIN -->
<script data="type=tools&uid=1536434" id="bdshare_js" type="text/javascript" src="http://bdimg.share.baidu.com/static/js/bds_s_v2.js?cdnversion=407460"></script>
<script type="text/javascript">
document.getElementById("bdshell_js").src = "http://bdimg.share.baidu.com/static/js/shell_v2.js?cdnversion=" + Math.ceil(new Date()/3600000)
</script>
<!-- Baidu Button END -->
<div articleid="6750434" id="digg">
<dl οnclick="btndigga();" class="digg digg_enable" id="btnDigg">
<dt>顶</dt>
<dd>2</dd>
</dl>
<dl οnclick="btnburya();" class="digg digg_enable" id="btnBury">
<dt>踩</dt>
<dd>0</dd>
</dl>
</div>
<div data-mod="popu_222" class="tracking-ad"><a href="javascript:void(0);" target="_blank"> </a> </div>
<div data-mod="popu_223" class="tracking-ad"> <a href="javascript:void(0);" target="_blank"> </a></div>
<script type="text/javascript">
function btndigga() {
$(".tracking-ad[data-mod='popu_222'] a").click();
}
function btnburya() {
$(".tracking-ad[data-mod='popu_223'] a").click();
}
</script>
<ul class="article_next_prev">
<li class="prev_article"><span οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_shangyipian']);location.href='/ultrani/article/details/6716727';">上一篇</span><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_shangyipian'])" href="/ultrani/article/details/6716727">正则迷雾</a></li>
<li class="next_article"><span οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_xiayipian']);location.href='/ultrani/article/details/6861400';">下一篇</span><a οnclick="_gaq.push(['_trackEvent','function', 'onclick', 'blog_articles_xiayipian'])" href="/ultrani/article/details/6861400">java 网站 分类</a></li>
</ul>
<div style="clear:both; height:10px;"></div>
<div style="" class="similar_article">
<h4>我的同类文章</h4>
<div style="margin:20px 0px 0px 0px" class="similar_c">
<div class="similar_c_t">
<label class="similar_cur">
<span οnclick="GetCategoryArticles('833106','UltraNi','foot','6750434');" style="cursor:pointer">linux技巧<em>(11)</em></span>
</label>
</div>
<div style="max-height:195px;" data-mod="popu_141" class="similar_wrap tracking-ad">
<a style="display:none" href="http://blog.csdn.net" target="_blank">http://blog.csdn.net</a>
<ul class="similar_list fl"><li><em>•</em><a target="_blank" id="foot_aritcle_10098481undefined017966214710877337" href="http://blog.csdn.net/UltraNi/article/details/10098481" title="linux配置本地tomcat应用80端口转发">linux配置本地tomcat应用80端口转发</a><span>2013-08-20</span><label><i>阅读</i><b>4700</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_9099083undefined3779339262718392" href="http://blog.csdn.net/UltraNi/article/details/9099083" title="批量scp脚本——从多台机器拷贝相同文件">批量scp脚本——从多台机器拷贝相同文件</a><span>2013-06-14</span><label><i>阅读</i><b>3743</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_8020201undefined5338487479235362" href="http://blog.csdn.net/UltraNi/article/details/8020201" title="远程终端链接解决vi中文乱码问题">远程终端链接解决vi中文乱码问题</a><span>2012-09-26</span><label><i>阅读</i><b>1111</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6575327undefined9611503053016697" href="http://blog.csdn.net/UltraNi/article/details/6575327" title="sudo 技巧">sudo 技巧</a><span>2011-06-29</span><label><i>阅读</i><b>390</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6548899undefined4366808450972657" href="http://blog.csdn.net/UltraNi/article/details/6548899" title="vim技巧">vim技巧</a><span>2011-06-16</span><label><i>阅读</i><b>459</b></label></li> </ul>
<ul class="similar_list fr"><li><em>•</em><a target="_blank" id="foot_aritcle_9227371undefined9320868202719063" href="http://blog.csdn.net/UltraNi/article/details/9227371" title="Linux下最好用的pdf查看器:Okular">Linux下最好用的pdf查看器:Okular</a><span>2013-07-02</span><label><i>阅读</i><b>8259</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_8946638undefined46725670345209347" href="http://blog.csdn.net/UltraNi/article/details/8946638" title="win7 上 linux mint14 (64bit) 双系统安装过程">win7 上 linux mint14 (64bit) 双系统安装过程</a><span>2013-05-19</span><label><i>阅读</i><b>6347</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6716727undefined6292002285038891" href="http://blog.csdn.net/UltraNi/article/details/6716727" title="正则迷雾">正则迷雾</a><span>2011-08-24</span><label><i>阅读</i><b>1572</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6572841undefined19099063159691076" href="http://blog.csdn.net/UltraNi/article/details/6572841" title="SVN使用指南">SVN使用指南</a><span>2011-06-28</span><label><i>阅读</i><b>377</b></label></li> <li><em>•</em><a target="_blank" id="foot_aritcle_6545906undefined5767895978910051" href="http://blog.csdn.net/UltraNi/article/details/6545906" title="linux 技巧摘要">linux 技巧摘要</a><span>2011-06-15</span><label><i>阅读</i><b>622</b></label></li> </ul>
<a class="MoreArticle" href="http://blog.csdn.net/UltraNi/article/category/833106">更多文章</a></div>
</div>
</div>
<script type="text/javascript">
$(function () {
GetCategoryArticles('833106', 'UltraNi','foot','6750434');
});
</script>
</div>