如何为您的Docker映像和代码依赖项构建安全扫描基础架构

Co-Authors: Manish Mehta, Comet

合著者：曼尼斯·梅塔( Manish Mehta) ，彗星

Early this year, our team was asked to augment our existing security tools and software dev+test practices for PCI-DSS and SOC-2 compliance. One of the key areas we had to augment was vulnerability scanning for our K8s-based microservices and a couple of monolithic services. As a result, our DevOps team had to pick between a few commercial tools and open source solutions to implement vulnerability scanning for our software. All of these tools do very similar things: they scan dependencies (either project libraries or OS packages) and compare them with vulnerability databases (like NIST’s NVD and others).

今年年初，我们的团队被要求为PCI-DSS和SOC-2合规性扩展我们现有的安全工具和软件开发+测试实践。 我们必须增强的关键领域之一是基于K8s的微服务和几个整体服务的漏洞扫描。 结果，我们的DevOps团队不得不在一些商业工具和开源解决方案之间进行选择，以对我们的软件实施漏洞扫描。 所有这些工具做的事情都非常相似：它们扫描依赖关系(项目库或OS软件包)，并将其与漏洞数据库(例如NIST的NVD和其他)进行比较。

建立还是购买？ (Build vs. Buy?)

There are a few open source tools that can be used to build your own vulnerability scanning solution, or you can use a commercial product that adds additional features over basic scanning that can make your life easier. Examples include:

有一些开源工具可用于构建自己的漏洞扫描解决方案，或者您可以使用商业产品，该产品在基本扫描之上添加了其他功能，从而使您的生活更加轻松。 示例包括：

• Nice dashboards for quick overview of vulnerabilities in your projects
  出色的仪表板，可快速概述项目中的漏洞
• Notifications (email, Slack, etc.)
  通知(电子邮件，Slack等)
• Integration with Git repositories (GitHub, GitLab)
  与Git存储库集成(GitHub，GitLab)
• Integration with other services (Docker registry, Kubernetes)
  与其他服务(Docker注册表，Kubernetes)集成
• Integration with project management (e.g. Jira)
  与项目管理集成(例如，Jira)
• Automatic merge request creation to fix vulnerable dependencies
  自动合并请求创建以修复易受攻击的依赖项
• Support for teams, projects and RBAC
  支持团队，项目和RBAC

Most commercial solutions augment public vulnerability databased with their own vulnerability database to eliminate false positives/negatives and give you more relevant results. That said, all of these commercial tools become really expensive as the number of developers increase. So you need to consider whether it’s “better” for you to buy or to build your own solution using existing open source tooling.

大多数商业解决方案都使用自己的漏洞数据库来增强公共漏洞数据库的功能，以消除误报/负面影响，并为您提供更相关的结果。 也就是说，随着开发人员数量的增加，所有这些商业工具都变得非常昂贵。 因此，您需要考虑购买它还是使用现有开源工具构建自己的解决方案是“更好”的选择。

Initially, we decided that commercial tool would best suit our need given the time constraint we had. However, after spending 3-months deploying and testing an industry leading commercial product, we faced a many challenges — limited support for golang, it would pick incorrect dependencies and not exit gracefully, problems with containers built from scratch vs those build with some base image, etc. As a result, we decided to give up on it and switch to building our own tool using available open source technologies.

最初，考虑到我们的时间限制，我们认为商业工具最适合我们的需求。 但是，在花了3个月的时间部署并测试了行业领先的商业产品之后，我们面临着许多挑战-对golang的支持有限，它将选择不正确的依赖项而无法正常退出，从头开始构建的容器与带有某些基本映像的容器的问题因此，我们决定放弃它，转而使用可用的开源技术来构建我们自己的工具。

In this post, I am going to provide you a solid base to start with if you choose to go open source. In addition, at the end of the post is the link to our open source repo to get you going!

在这篇文章中，如果您选择开源，我将为您提供坚实的基础。 此外，文章结尾处是指向我们的开放源代码仓库的链接，以助您一臂之力！

扫描什么？ (What to scan?)

There are basically two things you want to scan for vulnerabilities:

您基本上要扫描漏洞的两件事：

1. Your project’s code dependencies (yarn.lock, Gopkg.lock, etc.)

   项目的代码依赖项( yarn.lock ， Gopkg.lock等)

2. OS dependencies (Debian packages, Alpine packages, etc.) in a Docker image
   Docker映像中的OS依赖项(Debian软件包，Alpine软件包等)

In the future, it’s possible to add more capabilities like license checking or static code analysis, but that is out of scope for this post.

将来可能会添加更多功能，例如许可证检查或静态代码分析，但这超出了本文的范围。

什么时候扫描？ (When to scan?)

First of all, you want to ensure that no new vulnerability is introduced by your code or Docker image change, so it’s a good idea to scan every merge request before it’s merged.

首先，您要确保代码或Docker映像更改不会引入新的漏洞，因此最好在合并之前扫描每个合并请求。

But you also need to scan periodically, as new vulnerabilities can appear over time. Also, severities might change — a low severity that you initially ignored may become critical over time as a new attack vector or exploit is found. While commercial tools remember the last scan and will notify you if a new vulnerability is found, in our open source solution, we simply repeat the scan on top of the project’s master branch.

但是您还需要定期进行扫描，因为随着时间的推移可能会出现新的漏洞。 而且，严重性可能会发生变化–随着发现新的攻击媒介或攻击，您最初忽略的低严重性可能会随着时间的流逝而变得至关重要。 商业工具会记住上一次扫描，并会在发现新漏洞时通知您，但在我们的开源解决方案中，我们只需在项目的master分支顶部重复扫描即可。

选择工具 (Choosing tools)

There are a lot of open source tools available but our pick was Trivy because it’s simple to use, under active development, and can scan multiple projects and OS types.

有很多开放源代码工具可用，但是我们选择Trivy是因为它易于使用，处于活跃开发状态并且可以扫描多个项目和OS类型。

Trivy is a simple and comprehensive vulnerability scanner for containers and other artifacts. A software vulnerability is a glitch, flaw, or weakness present in the software or in an operating system. Trivy detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). Trivy is easy to use — just install the binary and you’re ready to scan! All you need to do for scanning is to specify a target such as an image name of the container.

Trivy是用于容器和其他工件的简单而全面的漏洞扫描程序。 软件漏洞是软件或操作系统中存在的故障，缺陷或弱点。 Trivy检测OS软件包(Alpine，RHEL，CentOS等)的漏洞和应用程序依赖项(捆绑程序，Composer，npm，yarn等)。 Trivy易于使用-只需安装二进制文件即可进行扫描！ 扫描所需要做的就是指定一个目标，例如容器的图像名称。

It can be easily installed and executed locally against the built Docker image or project’s root. It also supports multiple outputs, including JSON and table output:

可以轻松地在构建的Docker映像或项目的根目录下本地安装和执行它。 它还支持多个输出，包括JSON和表输出：

Unfortunately there are projects which Trivy cannot scan (e.g. Golang), so we had to rely on OWASP Dependency-Check as a lot of our code is in golang.

不幸的是，有一些Trivy无法扫描的项目(例如Golang)，因此我们不得不依靠OWASP Dependency-Check 因为我们很多代码都在golang中。

Dependency-Check is a Software Composition Analysis (SCA) tool that attempts to detect publicly disclosed vulnerabilities contained within a project’s dependencies. It does this by determining if there is a Common Platform Enumeration (CPE) identifier for a given dependency. If found, it will generate a report linking to the associated CVE entries.

依赖项检查是一种软件组成分析(SCA)工具，它试图检测项目依赖项中包含的公开披露的漏洞。 它通过确定给定依赖项是否存在通用平台枚举(CPE)标识符来完成此操作。 如果找到，它将生成一个报告，链接到相关的CVE条目。

It can generate html pages with report, JUnit and JSON output (and some others).

它可以生成带有报告，JUnit和JSON输出(以及其他一些输出)的html页面。

报告和仪表板 (Reports and dashboards)

Now we are getting closer to more interesting stuff 😊

现在我们正在接近更有趣的东西😊

As we are already use Elasticsearch + Kibana + Fluentd in our DevOps and SRE teams, it was a natural fit to use existing infrastructure to analyze JSON output of our security scans.

由于我们的DevOps和SRE团队已经使用Elasticsearch + Kibana + Fluentd ，因此使用现有基础结构来分析安全扫描的JSON输出是很自然的选择。

We only needed to find a way to send data from untrusted infrastructure (CI runners) to our secure Elasticsearch. For this purpose, we decided to have a message queue in the middle. Fluentd has in_sqs plugin to read messages from Amazon SQS and it’s also simple to use so final architecture looks like this:

我们只需要找到一种方法即可将数据从不受信任的基础结构(CI运行器)发送到我们的安全Elasticsearch。 为此，我们决定在中间放置一个消息队列。 Fluentd具有in_sqs插件可以从Amazon SQS读取消息，并且使用起来也很简单，因此最终的架构如下所示：

Security scanning architecture

安全扫描架构

Once data reaches Elasticsearch, it is simple to create a few dashboards with Kibana or use Discover to query vulnerabilities with all details as needed.

数据到达Elasticsearch后，可以轻松地使用Kibana创建一些仪表板，或使用Discover根据需要查询具有所有详细信息的漏洞。

Overview of vulnerabilities

漏洞概述

Searching for critical vulnerabilities

搜索关键漏洞

解决漏洞(Addressing vulnerabilities)

So now we have scans and some nice dashboard, what next? We cannot expect that developers will watch these dashboards on regular basis. Instead we decided to fail CI when an issue of high severity or above is found and has a fix available, that way responsibility is moved to developers and project owners.

所以现在我们有了扫描程序和一些不错的仪表板，接下来该怎么办？ 我们不能期望开发人员会定期观看这些仪表板。 相反，当发现严重性或严重性更高的问题并有可用的修复程序时，我们决定通过CI失败，从而将责任转移给开发人员和项目所有者。

Naturally, we needed to have an option to whitelist some CVEs or to override this default behavior on a per-project basis. So we created a repository with security scanning profiles which are consumed by our scanning tool which wraps Trivy execution and handles its behavior and output.

自然，我们需要有一个选项可以将某些CVE列入白名单或根据每个项目覆盖此默认行为。 因此，我们使用安全扫描配置文件创建了一个存储库，该配置文件由扫描工具使用，该扫描工具包装了Trivy执行并处理其行为和输出。

Now any developer can submit a merge request against this repository and ask for an exception. Example profile for kibana project:

现在，任何开发人员都可以针对该存储库提交合并请求，并请求例外。 kibana项目的示例配置文件：

profile:
  trivy:
  # Fail on HIGH and CRITICAL
  severity: HIGH  # Don’t fail on vulnerabilities that does not have fix
  ignore_unfixed: True  whitelist:
    vulnerabilities:
      - name: CVE-2018–1000620
        severity: CRITICAL
        package: cryptiles
        ref: https://github.com/elastic/kibana/issues/75995
        note: “Reported to upstream, whitelist until end of year”
        until: 2020–12–31

未来的改进 (Future improvements)

This implementation of security scanning is a solid base that can be extended beyond our initial use-cases. For example:

安全扫描的这种实现是坚实的基础，可以扩展到我们最初的用例之外。 例如：

• Add Slack notifications, utilize security profiles to specify per-project channel and options
  添加Slack通知，利用安全配置文件指定每个项目的通道和选项
• Add integration with Gitlab Issues, Jira or any other ticketing system
  添加与Gitlab Issue，Jira或任何其他票务系统的集成
• Create simple service or sidecar to execute scan of images deployed on Kubernetes cluster
  创建简单的服务或辅助工具以执行对Kubernetes集群上部署的映像的扫描

分叉并使用它(Fork it and use it)

You can find Dockerfile and tooling in our public Gitlab repository: https://gitlab.com/volterra.io/security-scanning

您可以在我们的公共Gitlab存储库中找到Dockerfile和工具： https://gitlab.com/volterra.io/security-scanning

翻译自: https://medium.com/volterra-io/how-to-build-security-scanning-infrastructure-for-your-docker-images-and-code-dependencies-62394ef5b3b6