Apache Metron Proposal

Apache Metron Proposal

阿帕奇Metron提案

---

 FINAL 

This proposal is now complete and has been submitted for a VOTE.

---

该提案现已完成，并已提交表决。

Abstract

The Metron project is an open source project dedicated to providing an extensible and scalable advanced security analytics tool. It has strong foundations in the Apache Hadoop ecosystem.

Metron是一个致力于提供一个可扩展的、先进的安全分析工具的一个开源项目。它在Apache的Hadoop生态系统打下坚实的基础。

Proposal

Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat-intelligence information to security telemetry within a single platform.

Metron集成了各种开源的大数据技术，以提供安全监控和分析的集中工具。Metron提供了日志聚合、全面的数据包捕获索引、存储、先进的行为分析和数据丰富功能，同时在单一平台中应用最新威胁情报安全遥测。

Metron can be divided into 4 areas:

1. A mechanism to capture, store, and normalize any type of security telemetry at extremely high rates. Because security telemetry is constantly being generated, it requires a method for ingesting the data at high speeds and pushing it to various processing units for advanced computation and analytics.

2. Real time processing and application of enrichments such as threat intelligence, geolocation, and DNS information to telemetry being collected. The immediate application of this information to incoming telemetry provides the context and situational awareness, as well as the “who” and “where” information that is critical for investigation.

3. Efficient information storage based on how the information will be used:

   1. Logs and telemetry are stored such that they can be efficiently mined and analyzed for concise security visibility
   2. The ability to extract and reconstruct full packets helps an analyst answer questions such as who the true attacker was, what data was leaked, and where that data was sent
   3. Long-term storage not only increases visibility over time, but also enables advanced analytics such as machine learning techniques to be used to create models on the information. Incoming data can then be scored against these stored models for advanced anomaly detection.

4. An interface that gives a security investigator a centralized view of data and alerts passed through the system. Metron’s interface presents alert summaries with threat intelligence and enrichment data specific to that alert on one single page. Furthermore, advanced search capabilities and full packet extraction tools are presented to the analyst for investigation without the need to pivot into additional tools.

Metron可以分为四个部分：

1. 有一种高效率的机制来捕捉，存储，规则化的任何类型的安全遥测。由于安全遥测正在不断生成，它需要接收高速数据，并将其推向各种处理单元，为了此后的计算和分析。
2. 实时处理和应用程序改进，例如，威胁情报，地理位置和DNS信息遥测。此信息传入遥测立即应用提供上下文和情景意识，以及“谁”和“哪里”的信息是用于调查的关键。
3. 高效的信息存储基础上如何信息将用于：
   1. 日志和遥测被存储，使得它们能够有效地开采和简明安全能见度分析
   2. 提取和重建全包的能力有助于分析者了解问题，如真正的攻击者是谁，被泄露的数据是什么，数据被发送到哪里
   3. 随着时间的推移，长期存放不仅增加可视性，也使先进的分析，如机器学习技术能够基于数据建模。传入数据可以根据这些模型进行评分，用于更先进的异常分析。
4. 通过系统，给安全人员数据和警报的视图集中接口。Metron的界面呈现威胁情报和其它数据，并具体到一个页面。此外，高级搜索功能和完整的数据包提取工具被呈现给分析人员调查，而不需要枢转到其他工具。

Big data is a natural fit for powerful security analytics. The Metron framework integrates a number of elements from the Hadoop ecosystem to provide a scalable platform for security analytics, incorporating such functionality as full-packet capture, stream processing, batch processing, real-time search, and telemetry aggregation. With Metron, our goal is to tie big data into security analytics and drive towards an extensible centralized platform to effectively enable rapid detection and rapid response for advanced security threats.

大数据很适合强大的安全分析。Metron框架集成了许多Hadoop的生态元素，为安全分析提供了一个可扩展的平台，这种功能包含全面的数据包捕获、流处理、批量处理、实时搜索和遥测聚集。拥有Metron，我们的目标是将大数据和安全分析相结合，形成一个可扩展的集中的平台，有效地实现快速检测和先进的安全威胁快速响应。

Background

OpenSOC was developed by Cisco over the last two years and pushed out to Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the development was mostly closed and has largely stopped. As evidence of the inactivity, users have complained that pull requests are not answered for a while https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ. Finally, no public releases of OpenSOC have been made. From an Apache point of view, the current community is not viable.

OpenSOC是由思科在过去的两年中开发并推出到Github上。然而，开发基本结束，并已基本停止。用户抱怨说，他们的请求很久都得不到应答。最终，没有OpenSOC的官方发布版本。从Apache官方来看，目前的社区是没有生命力的。

However, some of the developers of the project have left Cisco and have found interest from several others that would like to work together to form an active and open community at Apache starting from the current OpenSOC code base. A message to the current support group proposing moving to Apache got a single positive response. https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ

In general Apache accepts only voluntary contributions and avoids hostile forks. In this case, given that the community is demonstrably dead, it seems fair to fork the existing code at Apache to allow a new community to work on it. Once incubation starts, we will send a message pointing to the new home to the OpenSOC support group.

然而，一些项目的开发人员已经离开思科，他们中的几个人，就想根据当前OpenSOC代码，建立和形成Apache的积极和开放的社区。目前这个行动已经得到Apache肯定的答复。

一般来说，阿帕奇只接受自愿捐款，并避免形成敌对的情况。在这种情况下，考虑到社区已经接近死亡，取代现有的代码并让新社区进行这项工作显得十分合理。一旦开始孵化，我们将发送消息到新的OpenSOC支持小组。

Because Cisco is not currently interested in being involved, the project expects to change their name. The project would like to use Metron, although we will perform a podling name search to check for conflicts. Metron, meaning measure, is half of the greek root for the word 'telemetry.' Metron is also a DC Comics character who “... wanders in search of greater knowledge beyond his own”.

由于思科目前可能没有兴趣被卷入，该项目预计将改变他们的名字。该项目想使用的名字是Metron，我们将执行podling名称搜索，以检查是否存在冲突。Metron，意思是“措施”，是希腊“telemetry”的半根，也是DC漫画角色“......飘荡在搜索超越了自己更大的知识。”

Rationale

Metron strives to move the state of the art in security analytics forward. We want to move away from the proprietary nature of legacy security point tools and develop an open platform where people can contribute and share datasets, machine learning models, telemetry parsers, sources of telemetry enrichment, and threat intelligence feeds. Cyber security is too large of a problem for a single corporation to tackle on its own and the current tooling is too fragmented and proprietary for us to be able to rally around a single tool or vendor.

Metron致力于安全性分析的发展。我们要摆脱传统的安全工具的专有性，距离建立一个开放的平台，在那里，人们可以做出贡献，并共享数据集，建立机器学习模型，遥测解析器，遥测丰富的来源，威胁情报源。网络安全是太大的问题，对于单个企业依靠自己和目前的工具很难解决且过于分散，所以，我们希望能够聚集单一的工具或供应商。

In addition to being open and facilitating advancement in security analytics, Metron has several advantages over a conventional Security Information Management System (SIEM).

除了开放性和安全分析领域的进步，Metron已超过传统的安全信息管理系统（SIEM），并具有几个优点。

• Metron uses all open source stack under the hood and runs on commodity hardware. This means Metron is much cheaper to run then the competition. In security cost plays a major factor because the cost of your countermeasure for monitoring and reacting to a threat should not exceed the cost of what is being protected. By driving down the cost of security the economics works for more assets to be monitored, which means more secure data centers.
• Metron, being in the open, allows additional vetting and scrutiny by the open source community for all of its components. This is a better model for a security-oriented tool than doing it closed source. All the problems should be flushed out and fixed in the open. The closed source competition does not have this kind of rigor, is motivated by marketing and sales, and thus, does not inspire confidence when it comes to security.
• Being Hadoop-based, Metron can process unprecedented volumes of streaming data via Apache Storm. When an organization is hit with malware or malicious behavior most commonly this happens as a part of a global malware campaign, signatures for which are known and are available from third party threat intelligence feeds. Having the ability to take in all the feeds and reference them against every telemetry message processed by Metron in real time does not only facilitate detection of such campaigns, it changes the economics for the “bad guys”. If you have to customize your malware for each of your targets these global attacks become a lot more expensive and non viable for them.
• Metron strives to shift conventional SOC workflows away from being rules-driven to a more data-driven approach that incorporates machine learning and a higher degree of automation and autonomous detection. The modern threat landscape is too dynamic to be manageable via static rules alone, which is what conventional SIEMs rely on. Rule bases tend to bloat, and if improperly maintained turn themselves into sources of false positive alerts.

• 采用开源堆栈引擎，基于商用机器，成本低。
• 开源。
• 具有Hadoop的基础。
• 使用机器学习。

The ability to analyze and model large volumes of data at rest and then being able to push up the output of that into a stream processor is essential in disrupting the

对大量数据进行分析和建模，然后将其推送到流处理器的能力是必须的。

Current Status

现状

As stated in the background section, the current community isn’t healthy, which is why we are proposing moving to Apache Incubator. In this section, we will describe the current state of the OpenSOC project.

Meritocracy

The OpenSOC development is controlled by Cisco and pull requests are being ignored. The development list is private and requests to join are rejected because there is no activity on it. The goal of moving to Apache is to form a meritocracy where a variety of individuals, regardless of their current employer, come together and work together. We understand that diversity, open development, and open governance are critical to being a successful Apache project.

Community

The OpenSOC project is not responding to pull requests or making releases. The easiest solution would be to create a variety of forks of the project on github, but that would further fracture the community and prevent it from reaching critical mass. Our prefered solution is to build a single large diverse and open community at Apache.

Core Developers

The core developers of Metron are James Sirota, Charles Porter, and Mark Bittmann. None of them have experience running an open source project, but they are eager to learn.

Alignment

The ASF is a natural host for Metron given that it is already the home of Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data projects. Metron leverages many of Apache open-source products. We are very interested in a place to develop our community and integrations with the other Apache big data projects.

Known Risks

已知的风险

Orphaned Products

The current product developers are all salaried developers at a small number of companies and thus there is a risk of becoming an orphaned product. However, the companies view Metron as very important to their product offering and plan to ramp up their work in the space. The project is unique in the product space and thus has strong potential to become a sustainable community.

Inexperience with Open Source

The vast majority of the developers are inexperienced with open source development and the Apache Way. One of the major hurdles to graduation from the Apache Incubator will be demonstrating that they have learned the Apache Way and are applying it to how the project is managed. Vinod Kumar Vavilapalli is an Apache Member and plans on actively working as a committer in the project. They also have the other mentors to help them learn as they progress.

Homogenous Developers

The developers are employed by four diverse companies (B23, Hortonworks, Mantech, and Rackspace), They are distributed across the United States. We hope to attract additional diversity as an Apache project.

Reliance on Salaried Developers

Metron is currently being developed exclusively by salaried developers, but the goal of coming to Apache is to form a community of users and developers that is much more diverse including non-salaried developers.

Relationships with Other Apache Products

Metron has a strong relationship and dependency with Apache Flume, Hadoop, HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation community could help with a closer collaboration among these projects and as well as others.

We note that although there is a superficial resemblance to Apache Eagle, which does security analysis of Hadoop audit events, the projects are significantly different. In particular, Metron is focused on analyzing network packet traffic and thus has a very different scope and scale of events than Eagle.

An Excessive Fascination with the Apache Brand

While the Apache brand is important, we are much more interested in finding a home for the project that encourages open development and open governance. We want to form the new community using the Apache Way with its strong focus on meritocracy, organizational independence, and open development.

Documentation

文档

The current information on the OpenSOC project is here: http://opensoc.github.io/ A slide deck presenting background material is here: http://www.slideshare.net/JamesSirota/cisco-opensoc

Initial Source

最初来源

The initial code is on github: http://opensoc.github.io/

External Dependencies

外部依赖

Metron has the following external dependencies:

• Apache Flume
• Apache Hadoop
• Apache HBase
• Apache Hive
• Apache Kafka
• Apache Spark
• Apache Storm

• ElasticSearch

• MySQL

The project understands that it will need to support alternatives for MySQL that are licensed under a ALv2 compatible license.

Cryptography

加密

Metron will eventually support encryption on the wire, but this is not one of the initial goals, and we do not expect Metron to be a controlled export item due to the use of encryption. Metron supports but does not require the Kerberos authentication mechanism to access secured Hadoop services.

Required Resources

所需的资源

Mailing List

• metron-private for private PMC discussions
• metron-dev for developers
• metron-commits for all commits
• metron-users for all users

Version Control

Git is the preferred source control system.

Issue Tracking

• JIRA (METRON)

Other Resources

The existing code already has unit tests so we will make use of existing Apache continuous testing infrastructure. The resulting load should not be very large.

Initial Committers

最初的提交者

• Jim Baker < jim.baker at rackspace dot com >

• Mark Bittmann < mark at b23 dot io >

• Sheetal Dolas < sheetal at hortonworks dot com >

• Discovery Gerdes < discovery.gerdes at rackspace dot com >

• P. Taylor Goetz < ptgoetz at apache dot org >

• Andrew Hartnett < andrew.hartnett at rackspace dot com >

• Dave Hirko < dave at b23 dot io >

• Paul Kehrer < paul.kehrer at rackspace dot com >

• Brad Kolarov < brad at b23 dot io >

• Kiran Komaravolu <kkomaravolu at hortonworks dot com >

• Larry McCay < lmccay at appache.org >

• Ryan Merriman < rmerriman at hortonworks dot com >

• Michael Perez < michael.perez at hortonworks dot com>

• Charles Porter < Charles.Porter at mcs dot mantech dot com >

• Phillip Rhodes < motley.crue.fan at gmail dot com >

• Sean Schulte < sean.schulte at rackspace dot com >

• James Sirota < jsirota at hortonworks dot com >

• Casey Stella < cstella at hortonworks dot com >

• Bryan Taylor < bryan.taylor at rackspace dot com >

• Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >

• Vinod Kumar Vavilapalli < vinodkv at apache dot org >

• George Vetticaden < gvetticaden at hortonworks dot com >

• Oskar Zabik < oskar.zabik at rackspace dot com >

Affiliations

The initial committers are employees of:

• Jim Baker - Rackspace
• Mark Bittmann - B23
• Sheetal Dolas - Hortonworks
• Discovery Gerdes - Rackspace
• P. Taylor Goetz - Hortonworks
• Andrew Hartnett - Rackspace
• Dave Hirko - B23
• Paul Kehrer - Rackspace
• Brad Kolarov - B23
• Kiran Komaravolu - Hortonworks

• Larry McCay - Hortonworks

• Ryan Merriman - Hortonworks
• Michael Perez - Hortonworks
• Charles Porter - Mantech
• Phillip Rhodes - Fogbeam Labs
• Sean Schulte - Rackspace
• James Sirota - Hortonworks
• Casey Stella - Hortonworks
• Bryan Taylor - Rackspace
• Ray Urciuoli - Mantech
• Vinod Kumar Vavilapalli - Hortonworks
• George Vetticaden - Hortonworks
• Oskar Zabik - Rackspace

Sponsors

隶属关系

Champion

• Owen O’Malley - Apache IPMC member

Nominated Mentors

• P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member, Hortonworks

• Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, NASA

• Owen O’Malley < omalley at apache dot org > - Apache IPMC member, Hortonworks

• Billie Rinaldi < billie at apache dot org > - Apache IPMC member, Hortonworks

• Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC member, Hortonworks

Sponsoring Entity

We are requesting the Incubator to sponsor this project.

Addendum

赞助商

After the vote on the proposal started, Debo Dutta (dedutta at cisco dot com) in the office of the Cloud CTO at Cisco has commented that his team at Cisco is very interested in joining the Metron community at Apache.

原文来自：https://wiki.apache.org/incubator/MetronProposal