性能测试 基准测试_用一个明确的获胜者进行基准测试？

性能测试 基准测试

Machine learning competitions like Kaggle often have one clear winner. However, when it comes to production deployments many factors play a role: costs, performance, explainability, bias, speed. Here we examine practical tradeoffs in machine learning deployments as applied to cybersecurity.

像Kaggle这样的机器学习竞赛通常有一个明确的获胜者。 但是，在生产部署中，许多因素都起作用：成本，性能，可解释性，偏见，速度。 在这里，我们研究了应用于网络安全的机器学习部署中的实际取舍。

介绍 (Intro)

One of the biggest opportunities in cybersecurity right now is a comprehensive mapping of vulnerabilities (CVEs) to the MITRE ATT&CK® framework. Below, we explain what is MITRE ATT&CK/CVE, the size of the business opportunity and benchmark the current competitors mapping CVEs to MITRE ATT&CK. We will use a simple analogy of house robbery to make the article more accessible.

当前，网络安全中最大的机会之一就是将漏洞(CVE)全面映射到MITER ATT＆CK®框架。 下面，我们解释什么是MITER ATT＆CK / CVE，商业机会的大小，并以基准将CVE映射到MITER ATT＆CK的当前竞争对手作为基准。 我们将使用一个简单的房屋抢劫类比来使文章更易于访问。

Analogy: To use an example, it turns out that most house robberies start by entering not through the house door but the side window. Armed with that information and the awareness that I’ve left 3 open windows in my house helps me to prioritize my defense strategy. First off, I should close the windows.

打个比方 ：举个例子，事实证明，大多数房屋抢劫不是从房屋门进入而是从侧窗进入。 有了这些信息，并且意识到我在家里留下了3个打开的窗户，这有助于我确定防御策略的优先级。 首先，我应该关闭窗户。

source:Mitre, via attack.mitre (CC0)

来源：Mitre，通过 Attack.mitre (CC0)

什么是MITER ATT＆CK？ (What is MITRE ATT&CK?)

“MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community.”— attack.mitre.org

“ MITER ATT＆CK®是基于现实世界观察结果的全球对抗性战术和技术知识库。 ATT＆CK知识库被用作在私营部门，政府以及网络安全产品和服务社区中开发特定威胁模型和方法的基础。” — Attack.mitre.org

Analogy: In other words, MITRE ATT&CK would be the playbook of steps that robbers would take in order to rob a house. (Identify unoccupied houses. Determine the access route to a window. Find the valuables etc.)

打个比方 ：换句话说，MITRE ATT＆CK将是强盗为了抢劫房屋而采取的步骤的剧本。 (确定空置的房屋。确定通往窗户的通道。找到贵重物品等。)

什么是CVE(网络安全漏洞和披露)？ (What are CVEs (Cybersecurity Vulnerabilities and Exposures)?)

“A CVE is a dictionary that provides definitions for publicly disclosed cybersecurity vulnerabilities and exposures. The goal of CVE is to make it easier to share data across separate vulnerability capabilities (tools, databases, and services) with these definitions. CVE Entries are comprised of an identification number, a description, and at least one public reference.” — cve.mitre.org

“ CVE是一本字典，为公开披露的网络安全漏洞和暴露提供了定义。 CVE的目标是使使用这些定义的不同漏洞功能(工具，数据库和服务)之间的共享数据更容易。 CVE条目由标识号，描述和至少一个公共参考组成。” — cve.mitre.org

Analogy: In the case of our house robbery analogy, vulnerabilities are the weaknesses in your house security: unlocked door, broken cameras, your birthday as the safe password

类比 ：就我们的房屋抢劫类比而言，漏洞是您房屋安全的弱点：未锁门，损坏的摄像头，生日作为安全密码

Why would you map CVEs to MITRE ATT&CK?

为什么将 CVE 映射 到MITER ATT＆CK ？

When you have multiple vulnerabilities, how should you allocate your time and resources? By combining knowledge of your situation (CVE) and understanding of most commonly followed criminal practices (MITRE), you can assign a risk factor and act accordingly. However, currently, only 4% of CVEs are manually mapped to MITRE ATT&CKs thus leaving the security professionals unable to prioritize CVE’s based on their role in the attack chain. As various APT (hackers) used different paths in the ATT&CK framework some controls can have a disproportionate impact given current vulnerabilities (CVEs) in your system.

如果存在多个漏洞，应如何分配时间和资源？ 通过结合您的情况知识(CVE)和对最常用的犯罪行为的了解(MITRE)，您可以分配风险因素并采取相应的行动。 但是，当前只有4％的CVE手动映射到MITER ATT＆CK，因此安全专业人员无法根据CVE在攻击链中的角色来确定其优先级。 由于各种APT(黑客)在ATT＆CK框架中使用了不同的路径，因此，鉴于系统中的当前漏洞(CVE)，某些控件可能会产生不成比例的影响。

Analogy: Knowing that your house has a broken camera and open windows, allows you to prioritize closing the windows since most attackers would use them to break in. However, if the relationship between your situation (CVE) and attacker protocol (MITRE) is off by 5%, you might prioritize a safety action less likely to prevent robbery.

打个比方 ：知道您的房屋有破损的相机并打开窗户，可以让您优先关闭窗户，因为大多数攻击者都会使用它们来闯入。但是，如果您的情况(CVE)与攻击者协议(MITRE)之间的关系不存在5％，您可能会优先考虑不太可能防止抢劫的安全措施。

source: Kindly, via kindlyAnswer.me (CC0)

来源：请通过 kindlyAnswer.me (CC0)

您如何实现此映射？ (How can you implement this mapping?)

Here are 6 competitive approaches based on price, speed, and quality.

这里有6种基于价格，速度和质量的竞争方法。

• Most vendors have already started thinking about this and mapped the most common CVEs seen in the wild to the corresponding MITRE ATT&CK manually.
  大多数供应商已经开始考虑这一点，并将常见的常见CVE手动映射到相应的MITER ATT＆CK。
• Some vendors have built complicated heuristics that map all currently existing CVEs.
  一些供应商已经构建了复杂的启发式方法，以映射当前所有现有的CVE。
• Some vendors began using sophisticated machine learning techniques, including deep learning, and NLP.
  一些供应商开始使用复杂的机器学习技术，包括深度学习和NLP。

source: Kindly, via kindlyAnswer.me (CC0)

来源：请通过 kindlyAnswer.me (CC0)

哪一个最好？ (Which one is best?)

Summary: There is no one good vendor. Manual mapping of attacks seen in the wild gives a small list of roughly 4000 CVEs with high accuracy. Heuristics are scalable to all 100k CVEs with strong model interpretability. An advantage of machine learning techniques is that they perform well for newly emerging CVEs not previously seen by heuristics.

摘要 ：没有一个好的供应商。 手动映射野外出现的攻击可以准确地给出大约4000个CVE的一小部分。 启发式技术可扩展到所有10万个CVE，具有强大的模型可解释性。 机器学习技术的一个优势是，它们对于启发式算法以前未见的新兴CVE表现良好。

Details:

详细资料 ：

• The most expensive solution is not always the best. One of the manual mappings charges $$$ for the full service even though it only works for 4k CVEs. The best machine learning solution maps +100k CVEs for the same money $$$.

  最昂贵的解决方案并不总是最好的。 尽管仅适用于4k CVE，但其中一种手动映射会向全套服务收取$$$的费用。 最好的机器学习解决方案以相同的资金$$映射100K CVE。

• The variability amongst the vendors is very significant. The 2 manual mappings vary widely in quality depending on whether qualified security analysts or Amazon Turk performed the work. Natural language processing is almost twice as accurate as the deep learning technique due to the limited data set size.

  供应商之间的差异非常大 。 2个手动映射的质量差异很大，具体取决于合格的安全分析师或Amazon Turk是否执行了工作。 由于有限的数据集大小，自然语言处理几乎是深度学习技术的两倍。

• For some solutions, results improve with time. While heuristics perform well on existing CVEs, newly emerging CVEs are better mapped by the natural language techniques, which could indicate overfitting by the heuristics.

  对于某些解决方案，结果会随着时间而改善。 虽然启发式算法在现有CVE上表现良好，但是自然语言技术可以更好地映射新兴的CVE，这可能表明启发式算法过于适合。

Conclusion: Choosing the best depends on 1) your specific circumstance and 2) evolves over time as new CVE’s are released. Our meta-APIs meet both of these needs. We find your best fit the first time and continue to reassess as new APIs are released (all without your concerted effort).

结论 ：选择最佳方法取决于1)您的具体情况和2)随着新CVE的发布而随着时间的推移而发展。 我们的元API可以满足这两个需求。 我们会在第一时间找到最合适的方法，并在发布新的API时继续进行重新评估(所有这些都需要您的共同努力)。

下一步？ (Next steps?)

If you have your own solution to map CVE/MITRE ATT&CK, please contact us to be included in the continuous benchmarking, or share your own model API with hi@kindlyanswer.me. You will receive 90% of the revenue we make from it and you keep your intellectual property as we don’t require code submission. If you are a vendor that displays CVEs in their product, review these APIs to improve your product and mapping accuracy and contact hi@kindlyanswer.me to unlock more details. (purchase the model code from one of the vendors, buy a data dump, receive your API keys or login credentials for unlimited requests)Appendix

如果您有用于映射CVE / MITRE ATT＆CK的解决方案，请与我们联系以包含在连续基准测试中，或与hi@kindlyanswer.me共享您自己的模型API。 您会从中获得90％的收入，并且由于我们不需要提交代码，因此您保留了自己的知识产权。 如果您是在其产品中显示CVE的供应商，请查看这些API以改善您的产品和映射准确性，并联系hi@kindlyanswer.me以获得更多详细信息。 (从其中一个供应商处购买模型代码，购买数据转储，接收您的API密钥或登录凭证以进行无限制的请求)

开源的 (Open Source)

To promote the transparency of our work, we share a subset of our validation data in our GitHub. You can always contribute to it with a push request. To avoid fraud by the vendors we use a carefully curated holdout data set for the actual benchmarking.

为了提高工作的透明度，我们在GitHub中共享了一部分验证数据。 您始终可以通过推送请求为它做出贡献。 为避免供应商欺诈，我们将精心挑选的保留数据集用于实际基准测试。

翻译自: https://towardsdatascience.com/benchmarking-with-one-clear-winner-279691cbd11d

性能测试 基准测试