日志分析笔记-PERL

最新推荐文章于 2019-08-18 21:34:33 发布
最新推荐文章于 2019-08-18 21:34:33 发布
阅读量585 收藏
点赞数
分类专栏： Linux Perl
Linux 同时被 2 个专栏收录
113 篇文章 3 订阅
订阅专栏
Perl
81 篇文章 2 订阅
订阅专栏
#!/usr/bin/perl

use strict;
use Getopt::Std;
use vars qw($opt_d $opt_s $opt_e $opt_k $opt_g %date_format @filename $keyword $regular);

sub Usage
{
    print "Usage: parselog <[<-d Time_format> <-s Start_time> <-e End_time>] [-g gapcharacter] [-k keyword]> [filename]\n";
    print <<eof;
        -d  Specified time format
            %H: hour (00 .. 23)
            %I: Hour (01 .. 12)
            %k: hour (0 .. 23)
            %l: hour (1 .. 12)
            %M: minute (00 .. 59)
            %n: min (0 .. 59)
            %S: s (00 .. 60)
            %s: seconds (0 .. 60)
            %p: Display the local AM or PM
            %a: day of the week (Sun.. Sat)
            %A: day of the week (Sunday.. Saturday)
            %b: the month (Jan.. Dec)
            %B: the month (January.. December)
            %w: the first days of the week (0 .. 6)
            %d: Day (01 .. 31)
            %D: Day (1 .. 31)
            %m: month (01 .. 12)
            %L: month (1 .. 12)
            %y: the last two digits of the year (00.99)
            %Y: Full Year (0000 .. 9999)
            
        -s  Start Time
        -e  END Time
        -g  The interval between the specified keyword strings, does not support the regular
        -k  Analysis matched the specified keywords, keyword included in the single quotation
            marks, a number of keywords separated by inter-default with -=- can also be 
            customized, and to sequence matching, keyword does not support the regular
        filename Specify the log file to be analyzed, which can specify multiple files
                 (separated by spaces), if there is no file to read from the pipe.
        Script default output for the screen, with no parameters or parameter error output help information.
EOF
    exit -1;
}
sub Reg #过滤正则
{
    my $key = shift @_;
    $key =~ s/\./\\\./g;
    $key =~ s/\*/\\\*/g;
    $key =~ s/\^/\\\^/g;
    $key =~ s/\?/\\\?/g;
    $key =~ s/\$/\\\$/g;
    $key =~ s/\|/\\\|/g;
    $key =~ s/\]/\\\]/g; 
    $key =~ s/\[/\\\[/g;    
    $key =~ s/\\d/\\\\d/g;
    $key =~ s/\\D/\\\\D/g;
    $key =~ s/\\w/\\\\w/g;
    $key =~ s/\\W/\\\\W/g;
    $key =~ s/\\s/\\\\s/g;
    $key =~ s/\\S/\\\\S/g;
    return $key;
}
sub Options #选项参数处理器
{
    getopts('d:s:e:g:k:');
    if (!$opt_k && !$opt_d) {
        Usage;
    } elsif ($opt_d) {
        if (!$opt_s || !$opt_e) {
            Usage;
        }
    }
    if (@ARGV > 0) {
        while (my $tmp = shift @ARGV) {
            push (@filename, $tmp);
        } 
    } else {
        push (@filename, 'STDIN');
    }
    my $interval = '';
    $keyword = '';
    $opt_k = '' if (!$opt_k);
    if ($opt_g) {
        $interval = Reg($opt_g);
    } else {
        $interval = '-=-';
    }
    if ($opt_k =~ /$interval/) {
        $keyword = Reg($opt_k);
        $keyword =~ s/$interval/\.\*/g;
    } else {
        $keyword = Reg($opt_k);
    }
}
sub Parse_regular #参数分析
{
    my $format = shift @_;
    my $str;
    $regular = '';
    Usage unless ($format =~ /%/);
    for (my $i = 0; $i < length($format); $i++) {
        my $char = substr($format, $i, 1);
        if ($char ne '%') {
            Reg($char);
            $regular =  $regular . $char;
        } else {
            $i++;
            $char = substr($format, $i, 1);
            $str = $str . $char;
            if (exists $date_format{$char}) {
                $regular =  $regular . $date_format{$char};
            } else {
                Usage;
            }
        }
    }
    return $str;
}
sub Parse_time #提取匹配时间表
{
    my $line = shift @_;
    my $str = shift @_;
    my ($time, $j, $tmp, %dtime); #$j regular value
    $j = 1;
    if ($line =~ m/$regular/i) {
        for (my $i = 0; $i < length($str); $i++) {
            my $char = substr($str, $i, 1);
            $tmp = eval "\$" . $j;
            $dtime{$char} = $tmp;
            $j++;
        }
        ($time = $dtime{'Y'}) if (exists $dtime{'Y'});
        ($time = $dtime{'y'}) if (exists $dtime{'y'});
        
        ($time = $time . $dtime{'m'}) if (exists $dtime{'m'});
        ($time = $time . $dtime{'L'}) if (exists $dtime{'L'});
        
        if (exists $dtime{'b'}) {
            ($time = $time . '1') if (lc($dtime{'b'}) eq 'jan');
            ($time = $time . '2') if (lc($dtime{'b'}) eq 'feb');
            ($time = $time . '3') if (lc($dtime{'b'}) eq 'mar');
            ($time = $time . '4') if (lc($dtime{'b'}) eq 'apr');
            ($time = $time . '5') if (lc($dtime{'b'}) eq 'may');
            ($time = $time . '6') if (lc($dtime{'b'}) eq 'jun');
            ($time = $time . '7') if (lc($dtime{'b'}) eq 'jul');
            ($time = $time . '8') if (lc($dtime{'b'}) eq 'aug');
            ($time = $time . '9') if (lc($dtime{'b'}) eq 'sept');
            ($time = $time . '9') if (lc($dtime{'b'}) eq 'sep');
            ($time = $time . '10') if (lc($dtime{'b'}) eq 'oct');
            ($time = $time . '11') if (lc($dtime{'b'}) eq 'nov');
            ($time = $time . '12') if (lc($dtime{'b'}) eq 'dec');
        }
        
        if (exists $dtime{'B'}) {
            ($time = $time . '1') if (lc($dtime{'B'}) eq 'january');
            ($time = $time . '2') if (lc($dtime{'B'}) eq 'february');
            ($time = $time . '3') if (lc($dtime{'B'}) eq 'march');
            ($time = $time . '4') if (lc($dtime{'B'}) eq 'april');
            ($time = $time . '5') if (lc($dtime{'B'}) eq 'may');
            ($time = $time . '6') if (lc($dtime{'B'}) eq 'june');
            ($time = $time . '7') if (lc($dtime{'B'}) eq 'july');
            ($time = $time . '8') if (lc($dtime{'B'}) eq 'august');
            ($time = $time . '9') if (lc($dtime{'B'}) eq 'september');
            ($time = $time . '10') if (lc($dtime{'B'}) eq 'october');
            ($time = $time . '11') if (lc($dtime{'B'}) eq 'november');
            ($time = $time . '12') if (lc($dtime{'B'}) eq 'december');
        }
        
        ($time = $time . $dtime{'d'}) if (exists $dtime{'d'});
        ($time = $time . $dtime{'D'}) if (exists $dtime{'D'});
        
        if (exists $dtime{'a'}) {
            ($time = $time . '1') if (lc($dtime{'a'}) eq 'mon');
            ($time = $time . '2') if (lc($dtime{'a'}) eq 'tues');
            ($time = $time . '2') if (lc($dtime{'a'}) eq 'tue');
            ($time = $time . '3') if (lc($dtime{'a'}) eq 'wed');
            ($time = $time . '4') if (lc($dtime{'a'}) eq 'thurs');
            ($time = $time . '5') if (lc($dtime{'a'}) eq 'fri');
            ($time = $time . '6') if (lc($dtime{'a'}) eq 'sat');
            ($time = $time . '6') if (lc($dtime{'a'}) eq 'st');
            ($time = $time . '7') if (lc($dtime{'a'}) eq 'sun');
        }
        
        if (exists $dtime{'A'}) {
            ($time = $time . '1') if (lc($dtime{'A'}) eq 'monday');
            ($time = $time . '2') if (lc($dtime{'A'}) eq 'tuesday');
            ($time = $time . '3') if (lc($dtime{'A'}) eq 'wednesday');
            ($time = $time . '3') if (lc($dtime{'A'}) eq 'midweek');
            ($time = $time . '4') if (lc($dtime{'A'}) eq 'thursday');
            ($time = $time . '5') if (lc($dtime{'A'}) eq 'friday');
            ($time = $time . '6') if (lc($dtime{'A'}) eq 'saturday');
            ($time = $time . '7') if (lc($dtime{'A'}) eq 'sunday');
        }
        
        if (exists $dtime{'w'}) {
            if ($dtime{'w'} == 0) {
                $time = $time . '7'; 
            } else {
                $time = $time . $dtime{'w'};
            }
        } 
        
        if (exists $dtime{'p'}) {
            ($time = $time . '1') if ($dtime{'p'} =~ /AM|am/);
            ($time = $time . '2') if ($dtime{'p'} =~ /PM|pm/);
        }
        
        ($time = $time . $dtime{'H'}) if (exists $dtime{'H'});
        ($time = $time . $dtime{'I'}) if (exists $dtime{'I'});
        ($time = $time . $dtime{'k'}) if (exists $dtime{'k'});
        ($time = $time . $dtime{'l'}) if (exists $dtime{'l'});
        
        ($time = $time . $dtime{'M'}) if (exists $dtime{'M'});
        ($time = $time . $dtime{'n'}) if (exists $dtime{'n'});
        
        ($time = $time . $dtime{'S'}) if (exists $dtime{'S'});
        ($time = $time . $dtime{'s'}) if (exists $dtime{'s'});
        $time = $time + 1 - 1;
        return $time;
    } else {
        return -1;
    }   

}
#[main]
%date_format = ('Y' => '(\d{4})', 
                'y' => '(\d{2})',
                'm' => '([0-1]{1}\d{1})',
                'L' => '(\d{1,2})',
                'd' => '(\d{2})',
                'D' => '(\d{1,2})',
                'H' => '([0-2]{1}\d{1})',
                'M' => '(\d{2})',
                'n' => '(\d{2})',
                'S' => '(\d{2})',
                's' => '(\d{1,2})',
                'I' => '([0-1]{1}\d{1})',
                'k' => '(\d{1,2})',
                'l' => '(\d{1,2})',
                'p' => '(am|pm)',
                'w' => '(\d{1})',
                'a' => '(Mon|Tues|tue|Wed|Thurs|Fri|Sat|St|Sun)',
                'A' => '(Monday|Tuesday|Wednesday|midweek|Thursday|Friday|Saturday|Sunday)',
                'b' => '(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sept|sep|Oct|Nov|Dec)',
                'B' => '(January|February|March|April|may|June|July|August|September|october|november|december)'
                );
my ($string, $start_time, $end_time, $line_time);
Options;
if ($opt_d || $opt_k) {
    if ($opt_d) {
        $string = Parse_regular($opt_d);
        $start_time = Parse_time($opt_s, $string);
        $end_time = Parse_time($opt_e, $string);
        Usage if (($start_time == -1) || ($end_time == -1)); 
    }
    my $handle;  #文件描述符
    my $multi_line = 0; #多行标识
    my $key_flag = 0; #关键字匹配标识
    my @multiline;
    for(my $k = 0; $k < @filename; $k++) {
        my $file = shift @filename;
        if ($file ne 'STDIN') {
            unless (-r $file) {
                print "Read $file failed!\n";
                exit -2;
            }
            open (F, "$file") || die "open $file error!";
            $handle = 'F';
        } else {
            $handle = 'STDIN'; #从管道读取
        }
        
        while (my $lines = <$handle>) {
            if ($opt_k && !$opt_d) {
                print "$lines" if ($lines =~ /$keyword/i);
                next;
            }
            
            if ($opt_d && !$opt_k) {
                $line_time = Parse_time($lines, $string);
                next if (($multi_line == 0) && ($line_time < $start_time));
                last if ($line_time > $end_time);
                if ($line_time >= $start_time && $line_time <= $end_time) {
                    print "$lines";
                    $multi_line = 1;
                } else {
                    print "$lines";
                }
            }
            
            if ($opt_d && $opt_k) { #时间+关键字＝多行匹配
                $line_time = Parse_time($lines, $string);
                
                next if (($line_time == -1) && ($multi_line == 0));
                next if (($line_time != -1) && ($line_time < $start_time));
                last if (($line_time > $end_time) && ($key_flag == 0));
                if (($line_time == -1) && ($multi_line == 1)) {
                    if ($lines =~ /$keyword/i) {
                        push (@multiline, $lines);
                        $key_flag = 1;
                    } else {
                        push (@multiline, $lines);
                    }
                    next;
                }
                $multi_line++;
                if ($multi_line > 1) {
                    if ($key_flag == 1) {
                        while (my $buff = shift @multiline) {
                            print $buff;
                        }
                    }    
                    $multi_line = 1;
                    $key_flag = 0;
                }
                @multiline = ();
                push (@multiline, $lines);
                
                if ($lines =~ /$keyword/i) { #时间+关键字＝单行匹配
                    next if (($line_time != -1) && ($line_time < $start_time));
                    last if ($line_time > $end_time);
                    if ($line_time >= $start_time && $line_time <= $end_time) {
                        print "$lines";
                    }
                } 
            }
        }
        close($handle);
        print "\n\n\n[$file]\n\n\n" if (@filename > 1);
    }
} else {
    Usage;
}

高尔夫golf
关注
专栏目录
perl增量分析日志
zhaoyangjian724的专栏
06-01 735
#!/usr/bin/perl #取文件行数 ##循环开始清空文件 use POSIX; use CGI; use DBI; my $dbName = 'oadb'; my $dbUser = 'query'; my $dbUserPass = 'query'; my $dbh = DBI->connect("dbi:Oracle:$dbName", $dbUser, $dbUserPass) o
MySQL学习笔记-基础篇1
xu_jin_shan的博客
02-26 1238
MySQL 学习笔记–基础篇1 1.数据库概述与MySQL安装 2.基本SELECT语句 3.运算符 4.排序与分页 5.多表查询 6.单行函数 7.聚合函数
Perl 日志分析W3A_system
weixin_30668887的博客
11-04 138
SOC的功能目前已经完成了，正常接收并处理日志。 目前建立规则库中，发现如果一个个弄的话会很负责，所以准备写个完善的程序来做正则校验。 提高效率的同时减轻工作量。 具体的已经放入GIT仓库了。 地址：https://github.com/smarttang/Log_regex_test.git 有需求的可以关注下。 转载于:https://www.cnblogs.com/xiaoCon/...
Perl单线程日志分析工具
weixin_30925411的博客
02-26 70
#!/usr/bin/perl $i=0; while(<>){ #读取数据后去掉换行符 chomp; #把获得的一行数据进行切割 @log_grep=split(/"/,$_); #把切割后的数据交给后面的子程序进行处理 &grep_log_main(@log_grep); #输出处理后的结果 print "源访问ip地址:$ip_home 访问日期:...
使用perl分析多个日志
qin123qq的专栏
11-12 806
#!/bin/perl #perl "D:\01 广东移动ngcrm项目\was瘦身\analyzeaccesslog.pl" "C:/Users/c00210578/Documents" use strict; use warnings; die "使用的参数：$0，不符合要求，请指定目录" unless @ARGV==1; my $dir = $ARGV[0]; my %hash;
perl分析apache日志
weixin_34112030的博客
10-10 96
#!/usr/bin/perl use Getopt::Std; use warnings; use strict; sub load_log_file($) { my $log_filename = shift; open(FILE, "<", $log_filename) || die ...
fanfanfao.zip_微处理器开发_PHP-PERL_
08-10
PERL具有强大的字符串处理能力，支持正则表达式，使得它在数据挖掘、日志分析等领域非常高效。此外，PERL还拥有大量的模块，可以扩展其功能，如CGI模块用于Web开发，DBI模块用于数据库操作。 4. **fanfanfao.m**： ...
PERL系统学习教程笔记.zip
10-25
这使得Perl在处理文本数据时非常强大，常用于日志分析、数据提取等任务。 4. **文件处理**：Perl提供了丰富的文件和I/O操作函数，可以方便地读写文件、处理流和管道，非常适合进行系统管理和脚本编写。 5. **模块...
mysqlsla慢查询分析工具使用笔记
09-10
接着，通过yum命令安装必要的Perl库，如perl-ExtUtils-CBuilder、perl-ExtUtils-MakeMaker、perl-DBI和perl-DBD-MySQL。然后，使用CPAN模块安装YAML和Time::HiRes库。最后，下载mysqlsla的源码包，解压并编译安装。 ...
Perl编程入门：学习笔记与实战解析
此外，Perl的"粘合"特性使得它可以轻松地将不同的工具和数据源整合在一起，这种特性在处理系统管理、日志分析等任务时尤其有用。 Perl在各种领域都有广泛的应用，特别是在生物信息学中，由于其强大的文本处理能力，...
一个十分有用的Perl日志处理包使用说明
01-09
一个十分有用的Perl日志处理包使用说明，日志的写入，多重定向写入日志
perl写日志
noticmp的专栏
09-30 1467
#!/usr/bin/perl -w sub append_log { # ###文件和行号信息 open (LOG, ">>log.txt"); # print LOG "[$0."; # print LOG __LINE__; # print LOG "]"; ###内容字段 $i=0; foreach $v
用perl 统计日志信息
hrb133yqq的专栏
08-03 879
用perl分析日志文件 比如：07/31/2008 this is a test aaa;07/31/2008 this is a test aaa;07/30/2008 this is a test bbb;07/30/2008 this is a test bbb; aaa 统计结果：07/31/2008     2bbb 统计结果：07/30/2008  
perl处理埋点日志
笨蛋一小枚
06-21 540
由前端埋点，获得的日志，就perl简单进行处理。大概格式如下。 0.000 - 172.21.10.66 - - [16/Jan/2017:00:01:09 +0800] count-log.csyd.com "GET /count.js?webID=WEBID&webTitle=WEBTITLE&upURL=UPURL&userID=USERID&machineScreen=MACHINES
Perl日志处理小贴士
beeworkshop的博客
08-18 360
Perl日志处理 Perl能够使你称为Doolittle，不过是通过日志与系统交流。 读取日志文件 #!/usr/bin/perl -w 2 3 $logfile="log/secure"; 4 open LOG, "<", "$logfile" or die "Unable to open $logfile: $!\n"; 5 while(<LOG>){ ...
perl切割日志的利器
renzhelife
09-19 115
用了过awk,python,perl都切过日志，还是感觉perl是切日志的高手，执行效率高，功能也足够强大。下面是一个小例子： open(FILE,"/data/ebi/meta/log"); open(CONF,"/data/ebi/conf/urltype.txt"); open(LOG,"&gt;/data/ebi/meta/log_2"); @array1=(); while(&...
Perl提取wtmpx日志最近用户登录信息写到系统日志Syslog里
11-11 3512
背景知识： 《Perl读取wtmpx日志文件》；logger命令；Syslog日志；crontab命令； 参考书籍： 《Perl网络编程》 求助： Sys::Syslog模块处理系统日志编译警告：your vendor has not defined the Sys::Syslog macro _PATH_LOG... 求各位大大解惑，不胜感激。 源码如下： #!/us
perl 实现日志抓错功能
HexBug
04-21 1055
#!/usr/bin/perl $logfile="/var/log/messages"; open(LOG,$logfile) or die "Unable to open logfile:$!/n"; while(){ print if //berror/b/i; } close(LOG); 
用perl脚本来读取apache访问日志
weixin_33989780的博客
04-26 88
前言 这是我很久以前的写的perl脚本！～ 由于现在很久没用perl了，所以很多已经淡忘了，所以把它放在博客上，每天看看，让自己加深点印象。#!/usr/bin/perl#use CGI qw/:standard/;#my $cgi=new CGI;print "Pragma: no-cache\n";print "Cache-control: no-cache\n";...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值