Voidbox – Docker on YARN

为什么80%的码农都做不了架构师？>>>   

Voidbox – Docker on YARN

前些天对照翻译软件翻译的。待完成...

http://tech.hulu.com/blog/2015/08/06/voidbox-docker-on-yarn/

Voidbox – Docker on YARN

August 6th, 2015 by Huahui Yang

下面是翻译：

1. Voidbox Motivation

1.Voidbox动机

YARN is the distributed resource management system in Hadoop 2.0, which is able to schedule cluster resources for diverse high-level applications such as MapReduce, Spark. However, nowadays, all existing framework on top of YARN are designed with assumption of specific system environment. How to support user applications with arbitrary complex environment dependencies is still an open question. Docker gives the answer.

YARN是在Hadoop 2.0上面的分布式资源管理系统，能够在不同高层应用上调度集群资源，如MapReduce, Spark。然而,如今,所有的现有框架上的YARN与假设特定系统环境的设计？？。如何支持用户应用程序具有任意复杂环境的依赖仍然是一个悬而未决的问题。Docker给出了答（da）案（an）（"答^案"是敏感词）。

Docker is a very popular container virtualization technology. It provides a way to run almost any application isolated in a container. Docker is an open platform for developing, shipping, and running applications. Docker automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.

Docker是非常受欢迎的容器虚拟化技术。它提供一种在容器中几乎分离所有应用的方法。Docker是一个开放的平台开发、交付和运行应用程序？？。Docker自动化部署任何应用程序作为一个轻量级的,可移植的、自给自足的容器，将运行在任何地方。

In order to integrate the unique advantages of Docker and YARN, the Hulu engineering team developed Voidbox. Voidbox enables any application encapsulated in docker image running on YARN cluster along with MapReduce and Spark. Voidbox brings the following benefits:

为了整合Docker和YARN的独特优势，Hulu的工程团队开发了Voidbox。Voidbox允许任何应用程序封装在docker镜像在YARN集群上运行MapReduce和Spark。Voidbox带来以下好处:

Ease creating distributed application

轻松创建分布式应用程序

Voidbox handles most common issues in distributed computation system, say it, cluster discovery, elastic resource allocation, task coordination, disaster recovery. With its well-designed interface, it’s easy to implement a distributed application.

Voidbox处理分布式计算系统中最常见的问题,包括集群发现,弹性资源分配、任务协调、灾难恢复。凭借其设计良好的界面,很容易实现一个分布式应用程序。

Simplify deployment

简化部署

Without Voidbox, we need to create and maintain dedicated VM for application with complex environment even though the VM image is huge and not easy to deploy. With Voidbox, we could easily get resource allocated and make app run right the time we need it. Additional maintenance work is eliminated.

没有Voidbox，我们需要创建和维护应用程序专用的虚拟机，即使在复杂环境下VM镜像是巨大的和不易于部署的。使用Voidbox，我们可以很容易地获得资源分配，使应用程序在我们需要它的时候运行正确，额外的维护工作被淘汰。

Improve cluster efficiency

提高集群效率

As we could deploy Spark/MR and all kinds of Voidbox applications from different department together, we could maximize cluster usage.

我们可以一起从不同的部门部署Spark/MR和各种Voidbox应用程序，最大化地使用集群。

Thus, YARN as a big data operating platform has been further consolidated and enhanced.

因此，YARN作为一个大数据操作平台得到了进一步的巩固和增强。

Voidbox supports Docker container-based DAG(Directed Acyclic Graph) tasks in execution. Moreover, Voidbox provides several ways to submit applications considering demands of the production environment and the debugging environment. In addition, Voidbox can cooperate with Jenkins, GitLab and private Docker Registry to set up a set of developing, testing, automatic release process.

Voidbox支持Docker容器基于DAG（有向无环图）执行任务。此外,Voidbox考虑生产环境和调试环境的要求，提供了几种方法来提交应用程序。此外，voidbox能与Jenkins合作，GitLab和私有Docker Registry设置一套开发，测试，自动发布的过程。

2.Voidbox Architecture

2.Voidbox架构

2.1 YARN Architecture Overview

2.1 YARN架构概述

YARN enables multiple applications to share resources dynamically in a cluster. Here is the architecture of applications running in YARN cluster:

YARN使多个应用程序在集群中动态共享资源。这里是应用程序运行在YARN集群的体系结构：

TODO 图

voidbox_figure1

Figure 1. YARN Architecture

As shown in figure 1, a client submits a job to Resource Manager. The Resource Manager performs its scheduling function according to the resource requirements of the application. Application Master is responsible for the application tasks scheduling and execution of an application’s lifecycle.

如图1所示，客户端向Resource Manager（资源管理器）提交工作。Resource Manager根据应用程序的资源需求执行调度功能。Application Master负责应用程序的生命周期的调度和执行。

Functionality of each modules:

各模块的功能：

Resource Manager: Responsible for resource management and scheduling in cluster.

资源管理器：负责资源管理和调度集群。

NodeManager: Running on the compute nodes in cluster, taking care of task execution in the individual machine, collecting informations and keeping heartbeat with Resource Manager.

NodeManager:在集群中的计算节点上运行，在单独的机器上执行任务的执行，收集信息，并与资源管理器保持心跳。

Application Master: Takes care of requesting resources from YARN, then allocates resources to run tasks in Container.

Application Master: 需要从YARN中请求资源，然后在容器中分配资源来运行任务。

Container: Container is an abstract notion which incorporates elements such as memory, cpu, disk, network etc.

Container:容器是一个抽象的概念，包括内存、处理器、磁盘、网络等元素。

HDFS: Distributed file system in YARN cluster.

HDFS: YARN集群的分布式文件系统。

2.2 Voidbox Architecture Design

2.2 Voidbox架构设计

In Voidbox architecture, YARN is responsible for the cluster’s resource management. Docker acts as the task execution engine above of the operating system, cooperating with Docker Registry. Voidbox helps to translate user programming code into Docker container-based DAG tasks, apply for resources according to requirements and deal with DAG in execution.

在Voidbox架构，YARN负责集群资源管理。Docker作为任务执行引擎之上的操作系统，与Docker Registry合作。Voidbox帮助将用户程序代码为Docker容器基于DAG任务申请的资源，根据需要和DAG执行协议。

TODO 图

voidbox_figure2

Figure 2. Voidbox Architecture

As shown in figure 2, each box stands for one machine with several modules running inside. To make the architecture more clearly, we divide them into three parts, and functionality of Voidbox modules and Docker modules:

如图2所示，每一个盒子都代表一台机器，里面有几个模块。为了使结构更加清晰，我们把他们分为三个部分，分别为功能模块、Voidbox模块、Docker模块：

Voidbox Modules:

Voidbox模块：

Voidbox Client: The client program. Through Voidbox Client, users can submit a Voidbox application, stop it, and so on. By the way, Voidbox application contains several Docker jobs and a Docker job contains one or more Docker tasks.

Voidbox Client: 客户端程序。通过Voidbox Client，用户可以提交一Voidbox应用或让它（指的是Voidbox Client还是应用程序？？？）停下来，等。顺便说一下，Voidbox应用包含几个Docker工作和一个Docker的工作包含一个或多个Docker的任务。

Voidbox Master: Actually, it’s an application master in YARN, and takes care of requesting resources from YARN, then allocates resources to Docker tasks.

Voidbox Master:其实，这是一个YARN应用主节点，YARN和需要照顾的请求资源，然后将资源分配给Docker的任务。

Voidbox Driver: Responsible for task scheduling of a single Voidbox application. Voidbox supports Docker container-based DAG task scheduling and between tasks we can insert some other codes. So Voidbox Driver should handle the order scheduling of DAG task dependencies and execute the user’s code.

Voidbox Driver:负责一个Voidbox任务调度中的应用。Voidbox支持基于DAG任务调度、任务间的docker容器可以插入一些其他的代码。所以Voidbox Driver应处理为DAG任务依赖调度和执行用户代码。

Voidbox Proxy: The bridge between YARN and Docker engine, responsible for transiting commands from YARN to Docker engine, such as start or kill Docker container, etc.

Voidbox Proxy: 一个voidbox负责任务调度中的应用。voidbox支持基于DAG任务调度、任务间的泊坞容器可以插入一些其他的代码。所以voidbox司机应处理为DAG任务依赖调度和执行用户代码。

State Server: Maintaining the informations of Docker engine’s health status, providing the list of machines which can run Docker container. So Voidbox Master can apply for resources more efficiently.

State Server: 纱和Docker引擎之间的桥梁，负责过渡命令从纱线到码头工人发动机，如启动或杀死Docker容器等。

Docker Modules:

Docker模块

Docker Registry: Docker image storage, acting as an internal version control tool of Docker image.

Docker Registry: 泊坞窗图像存储，作为码头工人形象的内部版本控制工具。

Docker Engine: Docker container execution engine, obtaining specified Docker image from Docker Registry and launching Docker container.

Docker Engine: Docker容器执行引擎，从注册表获取指定的码头工人形象和Docker推出Docker容器。

Jenkins: Cooperating with GitLab, when application codes update, Jenkins will take care of automated testing, packaging, generating the Docker image and uploading to Docker Registry, to complete the application automatically release process.

Jenkins: 配合GitLab，当应用程序代码更新，Jenkins会照顾自动化测试，打包，生成的Docker镜像和上传到Docker Registry，完成应用程序自动发布的过程。

2.3 Running Mode

2.3 运行模式

Voidbox provides two application running modes: yarn-cluster mode and yarn-client mode.

Voidbox提供两个应用运行模式：yarn-cluster模式（yarn集群模式)和yarn-client模式（yarn客户端模式）。

In yarn-cluster mode, the control component and resource management component are running in the YARN cluster. After we submit the Voidbox application, Voidbox Client can quit at any time without affecting the running time of application. It’s for the production environment.

在yarn-cluster模式，控制组件和资源管理组件是运行在YARN集群。在我们提交Voidbox应用，Voidbox退出at any time没有Voidbox Client？？？？？？？可以影响运行时间的应用。这是生产环境。

In yarn-client mode, the control component is running in Voidbox Client, and other components are in the cluster. Users can see much more detailed logs about the application’s status. When Voidbox Client quits, the application in cluster will exit too. So it’s more convenient for debugging.

在色织客户模式，控制部分是运行在客户端voidbox，and other components are in the簇。用户可以看到更详细的日志关于应用程序的状态。当客户Voidbox对等的，应用集群将退出了。所以这是更方便的为调试。

Here we briefly introduce the implementation architecture of the two modes:

在这里，我们简要介绍了实施？？？实现？？？架构的模式：

yarn-cluster mode

voidbox_figure3

Figure 3. yarn-cluster mode

As shown in figure 3, Voidbox Master and Voidbox Driver are both running in the cluster. Voidbox Driver is responsible for controlling the logic and Voidbox Master takes care of application resource management.

如图3所示，Voidbox Master和Voidbox Driver都运行在集群。Voidbox Driver负责控制和Voidbox Master照顾应用资源管理的逻辑。

yarn-client mode

voidbox_figure4

Figure 4. yarn-client mode

As shown in figure 4, Voidbox Master is running in the cluster, and Voidbox Driver is running in Voidbox Client. Users can submit Voidbox application in IDE for debugging.

as shown in图4，Voidbox Master是运行集群，和Voidbox Driver是运行的Voidbox Client。用户可以提交Voidbox应用在IDE中调试。

2.4 Running Procedure

Here are the procedures of submitting a Voidbox application and its lifecycle:

这是一Voidbox提交应用程序的生命周期：

Users write a Voidbox application by Voidbox SDK and generate a java archive, then submit it to the YARN cluster by Voidbox Client;

After receiving Voidbox application, Resource Manager will allocate resources for Voidbox Master, then launch it.

Voidbox Master starts Voidbox Driver, the latter will decompose Voidbox application into several Docker jobs(a job contains one or more Docker tasks). 

Voidbox Driver calls Voidbox Master interface to launch the Docker tasks in compute nodes.

Voidbox Master requests resources from Resource Manager, and Resource Manager allocates some YARN containers according to the YARN cluster status. 

Voidbox Master launches Voidbox Proxy in YARN container, and the latter is responsible for communication with Docker engine to start the Docker container.

User’s Docker task is running in Docker container, and the log output to a local file. User can see real-time application logs through YARN Web Portal.

After all Docker tasks are done, the logs will be aggregate to HDFS, so user still can get the application logs by history server.

2.5 Docker integrating with YARN in resource management

YARN acts as a uniform resource manager in the cluster, and is responsible for resource management on all machines. Docker as a container engine also has the function of resource management. So how to integrate their resource management function is particularly important.

In YARN, the user task can only run in the YARN container, while Docker container can only be handled by Docker engine. This case would get out of the management of YARN and damage the unified management and scheduling principle of YARN, which could produce resource leaks risk issue. In order to enable YARN to manage and schedule Docker container, we need to build a proxy layer between YARN and Docker engine. This is why Docker Proxy is introduced. Through Voidbox Proxy, YARN can manage the container lifecycle including start, stop, etc.

In order to understand Voidbox Proxy more clearly, we take stopping Voidbox application as an example. When a user needs to kill Voidbox application, YARN will recycle all the resources of the application. At this point, YARN will send a kill signal to the related machines. The corresponding Voidbox Proxy will catch the kill signal, then stop Docker container in Docker engine to do the resource recycling. So with the help of Voidbox Proxy, it can not only stop YARN container, but also stop the Docker container to avoid resources leaks issue(This is the problem existing in open source version, see YARN-1964).

3. Fault Tolerance

3.故障容错

Although Docker has some stable releases, the enterprise production environment has a variety versions of operating system or kernel, so it brings unstable factors. We consider multiple levels in Voidbox fault-tolerant design to ensure Voidbox’s high availability.

虽然Docker有一些稳定的发布，企业生产环境有多种版本的操作系统或内核，所以它带来的不稳定因素。我们认为，在Voidbox容错设计确保Voidbox的高可用性多层次。

Voidbox Master fault tolerance

Voidbox Master容错

If Resource Manager finds Voidbox Master crashes, it will notify NodeManager to recycle all the YARN containers belonging to this Voidbox application, then restart Voidbox Master.

如果Resource Manager（资源管理器）找到Voidbox Master崩溃，它将通知NodeManager回收所有YARN容器属于这个Voidbox应用程序，然后重新启动Voidbox Master。

Voidbox Proxy fault tolerance

Voidbox Proxy容错

If Voidbox Master finds Voidbox Proxy crashes, it will recycle Docker containers on behalf of Voidbox Proxy.

如果Voidbox Master发现Voidbox Proxy崩溃，它将回收Docker容器的Voidbox Proxy。

Docker container fault tolerance

Docker容器容错

Each Voidbox application can configure the maximum retry times on failure, when the Docker container crashes, Voidbox Master will do some work according to the exit code of Docker container.

每个Voidbox应用可以配置对失败的最大重试次数，当Docker容器崩溃，Voidbox Master会根据Docker容器的退出代码做一些工作。

4. Programming model

4. 程序设计模型

4.1 DAG Programming model

4.1 DAG程序设计模型

Voidbox Provides Docker container-based DAG programming model. A sample would look similar to this:

voidbox提供Docker容器基于DAG的编程模型。一个样本看起来类似于这个：

voidbox-figure5

Figure 5. Docker container-based DAG programming model

As shown in figure 5, there are four jobs in this Voidbox application, and each job can configure its requirements of CPU, Memory, Docker image, parallelism and so on. Job3 will start when job1 and job2 both complete. Job1, job2 and job3 make a stage, so user can insert their codes after this stage is done, and finally start running job4.

如图5所示，在voidbox应用四个工作，每个工作可以配置CPU，内存，其要求的码头工人形象，排比等。工作时将开始投产和职业完全。职业和工作岗位，做一个阶段，所以用户可以插入自己的代码在这个阶段完成，并开始运行的工作吧。

4.2 Shell mode to submit one task

外壳模式提交一个任务

In most cases, we would like to run a single Docker container-based task without programming. So Voidbox supports shell mode to describe and submit the Docker container-based task, actually it’s a implementation based on DAG programming mode.The example usage of Voidbox in shell mode:

在大多数情况下，我们要运行一个基于Docker容器无需编程任务单。所以voidbox支持shell模式来描述和提交任务的Docker容器，其实这是一个实现基于DAG的编程模式。在壳模式voidbox示例用法：

docker-submit.sh \

-docker_image centos \

-shell_command “echo Hello Voidbox” \

-container_memory 1000 \

-cpu_shares 2

The shell script above will submit a task to run “echo Hello Voidbox” in a docker image named ‘centos’, and the resource requirement is 1000Mb memory, 2 cpu virtual cores. 

上面的脚本将提交一个任务运行“echo Hello Voidbox”docker镜像命名为“centos，且资源的要求是1000MB内存、2 CPU虚拟处理器。

5. Voidbox in Action

5.Voidbox实战

At present we can run Docker, MapReduce, Spark and other applications in YARN cluster. There has been lots of short tasks using Voidbox within HULU.

目前我们可以运行Docker，MapReduce，纱线簇火花和其他应用程序。在HULU已经有大量的短期任务中使用Voidbox。

Automation testing process

自动化测试过程

Cooperating with Jenkins, GitLab and private Docker registry, when the application codes update, Jenkins will complete automatic test, package program, regenerate Docker image and push it to the private Docker Registry. It’s a process of development, testing and automatically release.

配合Jenkins,GitLab和私有Docker registry,当应用程序代码更新,Jenkins将完成自动测试,打包,生成docker镜像,推送镜像到Docker Registry。这是一个开发、测试和自动发布的过程。

Complex tasks in parallel

并行的复杂任务

Test Framework is used to do some testings to detect the availability of some components. The project is implemented by Ruby/Java and has complex dependencies. So we maintain two layers of Docker image, the first layer is the system software as a base image, and the second layer is the business level. We publish a test framework Docker image and use some timing scheduling software to start Voidbox application regularly. Thanks to Voidbox, we solve the issues such as the complex dependencies and the multitasking parallelism.

测试框架是用来做一些试验来检测一些组件的可用性。该项目由Ruby/Java实现，并具有复杂的依赖关系。所以我们保持Docker镜像的两层,第一层是系统软件为基础的镜像,和第二个层是业务层。我们发布一个测试框架Docker镜像和使用一些定期定时调度软件开始Voidbox应用程序。由于Voidbox,我们解决一些问题,比如复杂的依赖关系和多任务并行性。

Facematch(link:http://tech.hulu.com/blog/2014/05/03/face-match-system-overview/) is a video analysis application. It’s implemented by C and has lots of graphics libraries. That can be optimized by Voidbox: first of all we need to package all face match program into a Docker image, then write Voidbox application to handle the multiple videos. Voidbox solves the complex machine environment and the parallelism control problem.

Facematch（链接：http://tech.hulu.com/blog/2014/05/03/face-match-system-overview/）是视频分析中的应用。它是由C++实现的，并且有很多图形库。可以优化Voidbox：首先我们需要封装所有人脸匹配程序到Docker镜像，然后写Voidbox应用程序处理多个视频。Voidbox解决复杂的机器环境和并行控制问题。

Building complex workflow

构造复杂工作流

Some tasks have a dependent with each other, such as it needs to load user behaviors first, then do the analysis of user behaviors. These two steps have successively dependencies. We use Voidbox container-based programming model to handle this case easily.

一些任务有相互依赖关系，比如需要先加载用户行为，然后进行用户行为分析。这两者先后相依。我们用Voidbox基于容器的编程模型来处理这种情况很容易。

6. Different from DockerContainerExecutor in YARN 2.6.0

6.不同于在YARN 2.6.0中DockerContainerExecutor

DockerContainerExecutor(link:https://issues.apache.org/jira/browse/YARN-1964) is released in YARN 2.6.0 and it’s alpha version. Not mature enough, and it is only an encapsulation layer above the default executor.

DockerContainerExecutor（链接：https://issues.apache.org/jira/browse/yarn-1964）发布于YARN 2.6.0的alpha版本。不够成熟，它只是一个封装层以上的默认执行。

DockerContainerExecutor is difficult to coexist with other ContainerExecutor in one YARN cluster.

DockerContainerExecutor很难与其它的ContainerExecutor共存于一个YARN集群。

Voidbox features

Voidbox特征:

DAG programming model

DAG的编程模型

Configurable container level of fault tolerance

可配置容器级别的容错

A variety of running modes, considering development environment and production environment

多种运行方式，考虑开发环境和生产环境

Share YARN cluster resources with other Hadoop job

与其他的Hadoop作业共享YARN集群资源

Graphical log view tool

图形化的日志视图工具

7. Future work

未来的工作

Support more versions of YARN

支持更多的版本的YARN

Voidbox would like to support more versions in the future besides YARN 2.6.0.

Voidbox想在将来除了YARN2.6.0外支持更多的版本。

Voidbox Master fault tolerance, persistent metadata to reduce the cost in case of retry

Voidbox Master容错,持久的元数据来降低重试的成本

Currently, if a Voidbox Master crashes, YARN will recycle resources belonging to this Voidbox application and restart Voidbox Master to do some tasks from the very beginning. It’s not necessary to impact tasks which are already done or running. We might keep some metadatas in the State Server to reduce the cost in case of Void我们会保持状态的一些元数据服务器来降低成本的Voidbox Master失败。

Voidbox Master as a permanent service

Voidbox Master作为一个永久的服务

Voidbox will support long running Voidbox Master to receive streaming tasks.

Voidbox将支持长期运行Voidbox Master接收流任务。

Support long service

支持长期服务

Voidbox will support long running service if Voidbox Master’s downtime doesn’t influence running task.

Voidbox将支持长时间运行的服务，如果Voidbox Master停机时间不影响运行的任务的话。

转载于:https://my.oschina.net/ylchou/blog/507054