logparser使用_分析您的Web服务器数据并使用LogParser和Log Parser Lizard GUI进行授权

最新推荐文章于 2024-05-11 06:39:44 发布

cunfusq0176

最新推荐文章于 2024-05-11 06:39:44 发布

阅读量732

点赞数

文章标签： java linux 大数据 mysql python

原文链接：https://www.hanselman.com/blog/analyze-your-web-server-data-and-be-empowered-with-logparser-and-log-parser-lizard-gui

版权

LogParser是一款强大的工具，用于深入研究Web服务器日志，如IIS日志。它允许您对文本文件执行SQL查询，非常适合探索大型日志数据。本文介绍了如何使用LogParser和Log Parser Lizard GUI来获取HTTP错误、版本分布等信息，以及如何利用这些工具来改进产品并进行数据分析。

摘要由CSDN通过智能技术生成

logparser使用

I've been using LogParser whenever I need to really dig into Web Server Logs since before 2005. It's an amazing tool. I love it. Jeff Atwood loves it, and you should to. It may not being something you use every day but when you need it, it's there and it's awesome. It's kind of like a really focused sed or awk. A low-level tool with a high-powered focus.

自2005年以来，每当我需要深入研究Web服务器日志时，我一直在使用LogParser 。这是一个了不起的工具。我喜欢它。 Jeff Atwood喜欢它，您应该这样做。它可能不是您每天都在使用的东西，但是当您需要它时，它就在那里而且很棒。这有点像一个非常专注的sed或awk。具有强大功能的低级工具。

Log Parser hasn't changed that I know of since 2005. I've been working with some folks to try to get it to escape the big house, but we'll see how far we get. Until then, it works fabulously and unchanged after all these years. It's great because while my primary use of LogParser is with IIS Log files, it'll query anything you can plug into it like the File System, Event Logs, the Registry or just a CSV file. The diagram from their docs is at right.

自2005年以来， Log Parser一直没有改变。我一直在和一些人合作，试图让它逃离那栋大房子，但是我们将看到我们能走多远。直到那时，这些年来它一直都很棒并且保持不变。很棒是因为尽管LogParser的主要用途是与IIS日志文件一起使用，但它会查询您可以插入其中的任何内容，例如文件系统，事件日志，注册表或CSV文件。他们文档中的图表在右边。

I did a blog post 6 years ago before FeedBurner where I analyzed traffic to my RSS feed from Newsgator. NewsGator was an RSS reader that would include statistics and information in its User-Agent HTTP Header. I was reminded of this post when I was talking to the NuGet team about how they are releasing new versions of NuGet every month or so but it's not clear how many people are upgrading. It'd also be interesting to find out what other ways folks are hitting the NuGet feed and what they are using to do it. I volunteered, so David Ebbo sent me a day's log file to "figure out."

6年前，我在FeedBurner之前写了一篇博客文章，分析了Newsgator到RSS feed的访问量。 NewsGator是一个RSS阅读器，在其User-Agent HTTP标头中包含统计信息和信息。当我与NuGet团队谈论他们如何每月发布新版本的NuGet时，让我想起了这篇文章，但尚不清楚有多少人正在升级。找出人们使用NuGet提要的其他方式以及他们使用它的方式也很有趣。我是自愿的，因此David Ebbo向我发送了一天的日志文件以“确定”。

Log Parser is wonderful because it effectively lets you run SQL queries against text files. Here's a few choice examples from Atwood's post a few years back:

Log Parser很棒，因为它可以有效地使您对文本文件运行SQL查询。以下是几年前Atwood的帖子中的一些选择示例：

十大最慢的项目 (Top 10 Slowest Items)

SELECT TOP 10 cs-uri-stem AS Url, MIN(time-taken) as [Min], 
AVG(time-taken) AS [Avg], max(time-taken) AS [Max], 
count(time-taken) AS Hits 
FROM ex*.log 
WHERE time-taken < 120000 
GROUP BY Url 
ORDER BY [Avg] DESC

每小时HTTP错误 (HTTP Errors Per Hour)

SELECT date, QUANTIZE(time, 3600) AS Hour, 
sc-status AS Status, COUNT(*) AS Errors 
FROM ex*.log 
WHERE (sc-status >= 400) 
GROUP BY date, hour, sc-status 
HAVING (Errors > 25) 
ORDER BY Errors DESC

Given queries like these, I figured that LogParser would be perfect for me to explore the NuGet web service logs. (Of course, I realize that the service itself could be instrumented, but this is more flexible, and I plan to make these queries run on a schedule and show up on http://stats.nuget.org.)

给定这样的查询，我认为LogParser非常适合探索NuGet Web服务日志。 (当然，我意识到可以对服务本身进行检测，但这更加灵活，我计划使这些查询按计划运行，并显示在http://stats.nuget.org上。)

There are a number of ways to access a NuGet packaging server. You can use the Add Package Dialog, the Command Line, the PowerShell Console within Visual Studio, or the NuGet Package Explorer. There's also some testing data and some "no user agent" stuff in there also. I filtered that out by just charting "NuGet" clients.

有多种方法可以访问NuGet包装服务器。您可以使用“添加软件包”对话框，命令行，Visual Studio中的PowerShell控制台或NuGet软件包资源管理器。也有一些测试数据和一些“没有用户代理”的东西。我只是通过绘制“ NuGet”客户端图表来过滤掉它。

I started doing the initial work from the command line, but it was slow going. I was having trouble visualizing what I wanted and what was being returned. Here is one of my first command lines. It was pretty hairy and hard to build this at the command line.

我从命令行开始进行初始工作，但是进展缓慢。我在想像我想要的东西和返回的东西时遇到了麻烦。这是我的第一个命令行之一。在命令行上很难做到这一点，而且非常麻烦。

C:\u_ex110831>LogParser.exe -i:IISW3C "SELECT DISTINCT cs(User-Agent) AS Client, 
   count(1) AS NumberOfHits 
   FROM u_ex110831.log 
   WHERE Client 
   LIKE 'NuGet%' 
   GROUP BY Client 
   ORDER by count(1) DESC"

Client                                                                                      NumberOfHits
------------------------------------------------------------------------------------------- ------------
NuGet+Add+Package+Dialog/1.4.20701.9038+(Microsoft+Windows+NT+6.1.7601+Service+Pack+1)      38840
NuGet+Command+Line/1.5.20830.9001+(Microsoft+Windows+NT+6.1.7601+Service+Pack+1)            15591
NuGet+Add+Package+Dialog/1.4.20701.9038+(Microsoft+Windows+NT+6.1.7600.0)                   13360
NuGet+Command+Line/1.4.20615.182+(Microsoft+Windows+NT+6.1.7600.0)                          8562
NuGet+Add+Package+Dialog/1.4.20607.9007+(Microsoft+Windows+NT+6.1.7601+Service+Pack+1)      5531
NuGet+Package+Manager+Console/1.4.20701.9038+(Microsoft+Windows+NT+6.1.7601+Service+Pack+1) 5497
NuGet+Command+Line/1.4.20615.182+(Microsoft+Windows+NT+6.1.7601+Service+Pack+1)             3699
NuGet+Package+Manager+Console/1.4.20701.9038+(Microsoft+Windows+NT+6.1.7600.0)              3654
NuGet+Add+Package+Dialog/1.4.20701.9038+(Microsoft+Windows+NT+5.1.2600+Service+Pack+3)      3558
NuGet+Command+Line/1.4.20615.182+(Microsoft+Windows+NT+5.2.3790+Service+Pack+2)             2539
Press a key...

There were 40 unique User Agents in this file and they include the client and its version as well as the operating system. I wanted first to chop it up to find out what Types of NuGet clients were being used. I have broken the lines up to make it clearer in this snippet.

该文件中有40个唯一的用户代理，其中包括客户端及其版本以及操作系统。我想先将其切碎，以找出正在使用的NuGet客户类型。我已经将这些代码弄清楚了，以使之在本摘要中更加清楚。

C:\u_ex110831>LogParser.exe 
    -i:IISW3C "SELECT DISTINCT SUBSTR(cs(User-Agent),0, index_of(cs(User-Agent),'/')) AS Client, 
    count(1) AS NumberOfHits FROM u_ex110831.log
    WHERE Client LIKE 'NuGet%'    
    GROUP BY Client    
    ORDER by count(1) DESC"

Client                        NumberOfHits
----------------------------- ------------
NuGet+Add+Package+Dialog      74761
NuGet+Command+Line            32284
NuGet+Package+Manager+Console 12637
NuGet+Package+Explorer        943
NuGet+Visual+Studio+Extension 49

Statistics:
-----------
Elements processed: 208235
Elements output:    5
Execution time:     0.79 seconds

Pretty amazing, though. A sub-second query over almost a quarter million line long log file with useful results and no database. Reminds me of working on Unix 20 years ago.

真是太神奇了。仅需不到一秒的时间，就可以查询将近25万行的日志文件，并且结果有用，而且没有数据库。让我想起了20年前使用Unix的经历。

After some experimenting and installing the Office Web Components 2003 (discontinued) and was outputting a chart with this MONSTER command line:

经过一些实验和安装Office Web Components 2003(已停产)并使用此MONSTER命令行输出图表：

C:\u_ex110831>LogParser.exe -i:IISW3C -o:CHART -chartType:PieExploded 
        -categories:Off -values:On -view:on
         -chartTitle:"NuGet Clients by User Agent" 
    "SELECT DISTINCT SUBSTR(cs(User-Agent),0,index_of(cs(User-Agent),'/')) AS Client, 
        count(1) AS NumberOfHits 
    INTO foo.png 
    FROM u_ex110831.log 
    WHERE Client
    LIKE 'NuGet%' 
    GROUP BY Client 
    ORDER by count(1) DESC"

Which yields me this profoundly 2003-looking chart, but still allows me to cheer a tiny victory inside. I will be able to get this (or a prettier one) to run as on a schedule (AT or Chron job) and serve it to the interwebs. I t'll probably be better to output a CSV or XML file, then process that with the web server and create a proper interactive chart. Regardless, tiny cheer.

这使我得到了这张看上去很深的2003年图表，但仍然让我为内部的胜利而欢呼。我将能够按计划(AT或Chron作业)运行此文件(或更漂亮的文件)并将其提供给互联网。我可能会输出一个CSV或XML文件，然后用Web服务器处理它并创建一个适当的交互式图表可能会更好。无论如何，微微的欢呼。

Still, I'm thinking I'm too old for this crap. Where's my GUI? What's a brother got to do to drag a DataGrid around here? A little Binging with DuckDuckGo (yes, I'm trying DDG this month) and I find - wait for it - LogParser Lizard GUI.

不过，我觉得我对于这个废话来说太老了。我的GUI在哪里？兄弟将DataGrid拖到这里做什么？与DuckDuckGo进行了一点交流(是的，我本月正在尝试DDG )，我发现-等待它-LogParser Lizard GUI 。

LogParser蜥蜴GUI (LogParser Lizard GUI)

What's this? Oh YES. It's intellisense and tooltips, baby!

这是什么？哦，是的。宝贝，这是智慧和工具提示！

I can't say how much faster this tool made me once I had figured out LogParser. It's funny how you have to suffer at the command line before you can really appreciate a good GUI. At this point I called Jon Galloway for some pair-SQLing and we pounded out a few more queries.

一旦弄清LogParser，我无法说出此工具使我快了多少。有趣的是，您必须先在命令行中受苦才能真正欣赏好的GUI。在这一点上，我打电话给Jon Galloway进行了一些成对SQLing，然后我们提出了更多的查询。

NuGet版本 (NuGet by Version)

I filtered out NuGet Package Explorer because it has its own version scheme. However, I'm not sure about this query, as I wanted to get the Major.Minor versions. I noticed that by coincidence the third value z (of x.y.z) always started with .2 so I cheated with the SUB() below because I couldn't figure out how to just filter out the x.y values. Any thoughts are appreciated.

我过滤掉了NuGet Package Explorer，因为它具有自己的版本方案。但是，我不确定此查询，因为我想获取Major.Minor版本。我注意到，巧合的是，(xyz的)第三个值z始终以.2开头，所以我欺骗了下面的SUB()，因为我不知道如何仅过滤掉xy值。任何想法表示赞赏。

SELECT DISTINCT SUBSTR( cs(User-Agent), 
                   ADD(index_of(cs(User-Agent),'/'),1),
                   SUB(index_of(cs(User-Agent),'.2'),STRLEN(cs(User-Agent))))
          AS Client, count(1) AS NumberOfHits
    FROM u_ex110831.log 
    WHERE cs(User-Agent) NOT LIKE '%Explorer%' AND cs(User-Agent) LIKE '%NuGet%'
    GROUP BY Client     
    ORDER by count(1) DESC 
Client  Hits
------ -----
1.4    98097
1.5    18985
1.3     2524
1.6       69

So then I did the whole version:

因此，我完成了整个版本：

SELECT SUBSTR( cs(User-Agent), 
                   ADD(index_of(cs(User-Agent),'/'),1),
                   SUB(index_of(cs(User-Agent),'+('),STRLEN(cs(User-Agent))))
          AS Client, count(1) AS NumberOfHits
    FROM u_ex110831.log 
    WHERE cs(User-Agent) NOT LIKE '%Explorer%' AND cs(User-Agent) LIKE '%NuGet%'
    GROUP BY Client, cs(User-Agent)
    ORDER by count(1) DESC 

Client           Hits
---------------  ------
1.4.20701.9038   38840
1.5.20830.9001   15591
1.4.20701.9038   13360
1.4.20615.182     8562
1.4.20607.9007    5531
1.4.20701.9037    5497
1.4.20615.182     3699
1.4.20701.9038    3654

I was extremely impressed with how quickly (about an hour) was able to get really substantive, interesting and targeted data out of these log files. The next step will be to get all the logs and run the command line tool create month over month line charts. The goal will be to figure out how many folks are successfully upgrading their NuGet installations as well as how they are using it. Are they using the right-click menu or are they using the console?

我对从这些日志文件中获取真正的，有趣的和有针对性的数据的速度(大约一个小时)印象深刻。下一步将获取所有日志并运行命令行工具创建逐月折线图。目的是弄清楚有多少人成功升级了他们的NuGet安装以及他们如何使用它。他们使用的是右键单击菜单还是使用控制台？

If you've got an application that makes HTTP calls to a service that you own, whether your application is a phone or a custom client, while you can certainly instrument your code on the server side to collect stats, there's a LOT of information in your IIS logs. You can use LogParser Lizard GUI to develop your queries and then schedule runs of the command line tool to generate reports that will really help you improve your product. This technique isn't as sophisticated as custom 3rd party analytics package but you can certainly get a surprising amount of information in a short amount of time with LogParser.

如果您有一个应用程序对您所拥有的服务进行HTTP调用(无论您的应用程序是电话还是自定义客户端)，尽管您当然可以在服务器端检测代码以收集统计信息，但其中包含大量信息您的IIS日志。您可以使用LogParser Lizard GUI来开发查询，然后安排命令行工具的运行以生成报告，这些报告确实可以帮助您改善产品。这种技术不像自定义的第三方分析软件包那样复杂，但是您肯定可以在短时间内通过LogParser获得大量信息。