一些DAG流程框架

AI之风盛行，为什么这么说？记得去年参加公司的校园招聘时，在面试的50人中95%的人学校都有人工智能的课程，如python、matlab等，但很少的人从事过具体应用实践，更不必说工程化场景了，由此看出AI的热度。人工智能前景的确很不错，市场也确实很有刚需，但真正要掌握好，真不是照着教科书打个helloworld那么容易。所以说，为了降低门槛，一直想做个工具，实现从数据准备、特征工程、模型训练、到评估的整个过程，也就是常说由多任务组成的pipeline。

在一个流程系统中，任务间往往存在复杂的依赖关系，为保证pipeline的正确执行，就是要解决各任务间依赖的问题，这样DAG结合拓扑排序是解决存在依赖关系的一类问题的利器。DAG ( Directed Acyclic Graph)，有向无环图，是指任意一条边有方向，且不存在环路的图。如果把依赖关系的问题建模成 DAG, 依赖关系成为 Graph 中的 Directed Edge, 然后通过拓扑排序，不断遍历和剔除无依赖的接点，可以达到快速解决依赖的目的。

要打造一个机器学习的平台，重要是要解决如何设计一个很好的DAG流程调度系统。下面一些开源的工作流调度系统可按需选择：

• Airflow

- Python-based platform for running directed acyclic graphs (DAGs) of tasks

• Argo

- Open source container-native workflow engine for getting work done on Kubernetes

• Azkaban

- Batch workflow job scheduler created at LinkedIn to run Hadoop jobs.

• Brigade

- Brigade is a tool for running scriptable, automated tasks in the cloud — as part of your Kubernetes cluster.

• Cadence

- An orchestration engine to execute asynchronous long-running business logic developed by Uber Engineering.

• CloudSlang

- Workflow engine to automate your DevOps use cases.

• Conductor

- Netflix's Conductor is an orchestration engine that runs in the cloud.

• Cromwell

- Workflow engine written in Scala and designed for simplicity and scalability. Executes workflows written in WDL or CWL.

• DigDag

- Digdag is a simple tool that helps you to build, run, schedule, and monitor complex pipelines of tasks.

• Fission Workflows

- A high-perfomant workflow engine for serverless functions on Kubernetes.

• Flor

- A workflow engine written in Ruby.

• Imixs-Workflow

- A powerful human-centric Workflow Engine based on the BPMN 2.0 standard.

• Kiba

- Data processing & ETL framework for Ruby

• Mistral

- Workflow service, in OpenStack foundation.

• Oozie

- Workflow Scheduler for Hadoop.

• Piper

- A distributed Java workflow engine designed to be dead simple.

• Pinball

- scalable workflow manager by Pinterest

• RunDeck

- Job Scheduler and Runbook Automation.

• Titanoboa

- Titanoboa is a platform for creating complex workflows on JVM.

• Wexflow

- A high-performance, extensible, modular and cross-platform workflow engine.

• Workflow Engine - A lightweight .NET and Java workflow engine.
• Workflow Core

- Workflow Core is a light weight workflow engine targeting .NET Standard.

• Copper

- A high performance Java workflow engine.

• Zeebe

- A workflow engine for microservices orchestration that's capable of executing BPMN models, developed by the team at Camunda