使用Elasticsearch和Kibana挖掘邮箱

最新推荐文章于 2024-08-05 09:32:41 发布

danpu0978

最新推荐文章于 2024-08-05 09:32:41 发布

阅读量643

点赞数

文章标签： python java 大数据 elasticsearch json

在上一篇文章中，我曾提到Logstash，Kibana和Elasticsearch（ELK堆栈）这三者是最受欢迎的开源解决方案之一，不仅用于日志管理，还用于数据分析。在本文中，我将演示如何使用ELK高效地执行大数据分析。作为参考，让我们获取一些巨大的邮箱数据。邮件档案可以说是最有趣的社交网络数据之一。它无处不在，每条信息都可以说明人们之间的交流。作为组织的CXO，您可能需要分析公司邮件的趋势和模式。

作为参考，我将采用著名的Enron语料库，因为它具有大量的邮件，并且没有任何法律或隐私问题的风险。该数据将被标准化为Unix邮箱（mbox）格式。从mbox格式开始，它将再次转换为单个json文件。

获取安然语料库数据

可以以各种格式下载原始格式的完整Enron数据集。我将从数据集的原始原始形式开始，该数据集本质上是一组文件夹，该文件夹按人和文件夹来组织邮箱的集合。下面的代码片段将说明在下载并取消存档后的语料库的基本结构。继续进行一些操作，以使您熟悉它。

C:\> cd enron_mail_20110402\maildir # Go into the mail directory

C:\enron_mail_20110402\maildir>dir # Show folders/files in the current directory</pre>
allen-p         crandell-s     gay-r           horton-s

lokey-t         nemec-g         rogers-b       slinger-r

tycholiz-b     arnold-j       cuilla-m       geaccone-t
<pre>               ...directory listing truncated...</pre>
neal-s         rodrique-r     skilling-j     townsend-j
<pre>C:\enron_mail_20110402\maildir> cd allen-p/ # Go into the allen-p folder

C:\enron_mail_20110402\maildir\allen-p> dir # Show files in the current directory</pre>
_sent_mail         contacts         discussion_threads notes_inbox

sent_items         all_documents     deleted_items     inbox
<pre>sent               straw 
C:\enron_mail_20110402\maildir\allen-p> cd inbox/ # Go into the inbox for allen-p
 C:\enron_mail_20110402\maildir\allen-p\inbox> dir # Show the files in the inbox for allen-p</pre>
<ol>
	<li>11. 13. 15. 17. 19. 20. 22. 24. 26. 28. 3. 31. 33. 35. 37. 39. 40.</li>
	<li>44. 5. 62. 64. 66. 68. 7. 71. 73. 75. 79. 83. 85. 87. 10. 12. 14.</li>
	<li>18. 2. 21. 23. 25. 27. 29. 30. 32. 34. 36. 38. 4. 41. 43. 45. 6.</li>
</ol>
<pre>   63. 65. 67. 69. 70. 72. 74. 78. 8. 84. 86. 9.
C:\enron_mail_20110402\maildir\allen-p\inbox> cat 1. # Show contents of the file named "1."

Message-ID: &lt;16159836.1075855377439.JavaMail.evans@thyme&gt;
 Date: Fri, 7 Dec 2001 10:06:42 -0800 (PST)
 From: heather.dunton@enron.com
 To: k..allen@enron.com
 Subject: RE: West Position
 Mime-Version: 1.0
 Content-Type: text/plain; charset=us-ascii
 Content-Transfer-Encoding: 7bit
 X-From: Dunton, Heather &lt;/O=ENRON/OU=NA/CN=RECIPIENTS/CN=HDUNTON&gt;
 X-To: Allen, Phillip K. &lt;/O=ENRON/OU=NA/CN=RECIPIENTS/CN=Pallen&gt;
 X-cc:
 X-bcc:
 X-Folder: \Phillip_Allen_Jan2002_1\Allen, Phillip K.\Inbox
 X-Origin: Allen-P
 X-FileName: pallen (Non-Privileged).pst

Please let me know if you still need Curve Shift.

Thanks,

现在，下一步是将邮件数据转换为Unix mbox格式。实际上，mbox只是连接的邮件的大型文本文件，可通过基于文本的工具轻松访问。我已经使用python脚本将其转换为mbox格式。此后，此mbox文件将转换为ELK兼容JSON格式。 json文件可以在这里找到。 json文件的片段可以在下面找到：

{"index":{"_index":"enron","_type":"inbox"}}

[{"X-cc": "", "From": "r-3-728402-1640008-2-359-us2-982d4478@xmr3.com", "X-Folder": "\\jskillin\\Inbox", "Content-Transfer-Encoding": "7bit", "X-bcc": "", "X-Origin": "SKILLING-J", "To": ["jeff.skilling@enron.com"], "parts": [{"content": "\n[IMAGE]\n[IMAGE]\nJoin us June 26th for an on-line seminar featuring Steven J. Kafka, Senior Analyst at Forrester Research, as he discusses how technology can create more effective collaboration in today's virtualized enterprise. Also featuring Mike Hager, VP, OppenheimerFunds, offering insights into implementing these technologies through real-world experiences. Brian Anderson, CMO, Access360 will share techniques and provide tips on how to successfully deploy resources across the virtualized enterprise. \nDon't miss this important event. Register now at http://www.access360.com/webinar/ . For a sneak preview, check out our one-minute animation that illustrates the challenges of provisioning access rights across the \"virtualized\" enterprise.\nAbout Access360\nAccess360 provides the software and services needed for deploying policy-based provisioning solutions. Our solutions help companies automate the process of provisioning employees, contractors and business partners with access rights to the applications they need. With Access360, companies can react instantaneously to changing business environments and relationships and operate with confidence, whether in a closed enterprise environment or across a virtual or extended enterprise.\n \nAccess360 \n\nIf you would prefer not to receive further messages from this sender:\n1. Click on the Reply button.\n2. Replace the Subject field with the word REMOVE.\n3. Click the Send button.\nYou will receive one additional e-mail message confirming your removal.\n\n", "contentType": "text/plain"}], "X-FileName": "jskillin.pst", "Mime-Version": "1.0", "X-From": "Access360 <R-3-728402-1640008-2-359-US2-982D4478@xmr3.com>@ENRON", "Date": {"$date": 991326029000}, "X-To": "Skilling, Jeff </o=ENRON/ou=NA/cn=Recipients/cn=JSKILLIN>", "Message-ID": "<14649554.1075840159275.JavaMail.evans@thyme>", "Content-Type": "text/plain; charset=us-ascii", "Subject": "Forrester Research on Best Practices for the \"Virtualized\" Enterprise"}

当您有大量数据要推送到Elasticsearch中时，最好通过指定数据文件来进行批量导入。每封邮件都在与指定索引（enron）和文档（收件箱）的条目相关的一行中。无需指定ID，因为Elasticsearch会自动指定ID。

Elasticsearch中的数据可以大致分为两种类型- 精确值和全文。确切的值正是它们听起来的样子。示例是日期或用户ID，但也可以包括确切的字符串，例如用户名或电子邮件地址。例如，精确值Foo与精确值foo不同。精确值2014与精确值2014-09-15不同。另一方面，全文是指通常以某种人类语言编写的文本数据，例如推文或电子邮件正文。出于此练习的目的，最好将电子邮件地址 （收件人，抄送，密件抄送）视为精确值。因此，我们首先需要指定映射，可以通过以下方式完成。

curl -XPUT “localhost:9200/enron” -d "{
"settings":
{
    "number_of_shards": 5,
    "number_of_replicas": 1
},
"mappings":
{
    "inbox":
    {
        "_all":
        {
            "enabled": false
        },
        "properties":
        {
            "To":
            {
"type": "string",
"index": "not_analyzed" 
            }, 
            "From": 
            { 
                "type": "string", 
                "index": "not_analyzed" 
            }, 
            "CC": 
            { 
                "type": "string", 
                "index": "not_analyzed" 
            }, 
            "BCC": 
            {
                "type": "string",
                "index": "not_analyzed" 
            } 
        } 
    } 
  } 
}"

您可以验证是否确实设置了映射。

curl -XGET "http://localhost:9200/_mapping?pretty"
{
    "enron" : 
    {
        "mappings" : 
        {
            "inbox" : 
            {
                "_all" : 
                {
                    "enabled" : false
                },
                "properties" : 
                {
                    "BCC" : 
                    {
                        "type" : "string",
                        "index" : "not_analyzed"
                    },
                    "CC" : 
                    {
                        "type" : "string",
                        "index" : "not_analyzed"
                    },
                    "From" : 
                    {
                        "type" : "string",
                        "index" : "not_analyzed"
                    },
                    "To" : 
                    {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        }
    }
}

现在，以以下方式使用json文件加载所有邮箱数据：

curl -XPOST "http://localhost:9200/_bulk" --data-binary @enron.json

我们可以检查是否所有数据都已成功上传。

curl "localhost:9200/enron/inbox/_count?pretty"
{
    "count" : 41299,
    "_shards" : 
    {
        "total" : 5,
        "successful" : 5,
        "failed" : 0
    }
}

您可以看到已经上传了41299条对应于不同消息的记录。现在，通过对该数据进行一些分析来开始有趣的部分。 Kibana提供了出色的分析功能和相关图表。让我们尝试查看每周发送多少消息。