四种工作流的比较

转载 2012年03月23日 15:51:52
HamakeComparisonWithOtherWorkflowEngines  

Phase-Support
Updated Jun 6, 2010 by vorl.s...@gmail.com

The table below attempts to compare Hamake and similar workflow engines for Hadoop (Oozie, Azkaban, Cascading) based on some key features. Although all of these systems could be used to solve similar problems, they differ significantly in design, philosophy, target user profile, usage scenarios, etc. So our feature-wise comparison is in no way conclusive. Please use it as a guideline, but read respective systems documentation to understand better which one is more suitable for your problem.

Feature Hamake Oozie Azkaban Cascading
workflow discription language XML XML (xPDL based) text file with key/value pairs Java API
dependencies mechanism data-driven explicit explicit explicit
requires Servlet/JSP container No Yes Yes No
allows to track a workflow progress console/log messages web page web page Java API
ability to schedule a Hadoop job execution at given time no yes yes yes
execution model command line utility daemon daemon API
allows to run Pig Latin scripts yes yes yes yes
event notification no no no yes
requires installation no yes yes no
supported Hadoop version 0.18+ 0.20+ currently unknown 0.18+
retries no at workflow node level yes yes
ability to run arbitrary commands yes yes yes yes
can be run on Amazon EMR yes no currently unknown yes

From FAQ:

What is the difference between Hamake and Cascading?

In short: Cascading is an API, while 'hamake' is an utility. Some differences:

  • hamake does not require any custom programming. It helps to automate running your existing Hadoop tasks and PIG scripts
  • We found hamake especially suitable for incremental processing of datasets
  • You can use 'hamake' to automate tasks written in other languages, for example using Hadoop streaming

How Hamake differs from Oozie and Azkaban?

Oozie and Azkaban are server-side systems that have to be installed and run as a service. Hamake is a lightweight client-side utility that does not require installation and has very simple syntax for workflow definition. Most importantly, Hamake is built based on dataflow programming principles - your Hadoop tasks execution sequence is controlled by the data. 

相关文章推荐

Azkaban2.5.0与Hadoop2.5.1的集成

本文主要介绍Azkaban在Hadoop2.5.1、Hive0.13.1环境下的安装和部署。 组件版本 Centos 6.5 64位 Java 1.7 Hadoop 2...

azkaban学习笔记总结

1. 任务调度概述 一个完整的数据分析系统通常都是由大量任务单元组成: shell脚本程序,java程序,mapreduce程序、hive脚本等各任务单元之间存在时间先后及前后依赖关系 ...

Azkaban成功搭建环境的过程

1、先将下载下来 2、将压缩包上传到hadoop的NameNode上 3、用tar xvf azkaban-0.10.tar.gz解压 4、设置HADOOP_HOME环境变量: exp...

浅谈工作流调度系统

浅谈工作流调度系统 http://ju.outofmemory.cn/entry/221885 每个公司都有自己的一套或者多套调度系统,从简单到复杂,满足各种定制化...
  • hsg77
  • hsg77
  • 2016-11-30 15:45
  • 1220

Oozie和Azkaban的技术选型和对比

源码角度分析Oozie和Azkaban的优劣

Azkaban调研

Azkaban调研 在作业设计器中,实用哪种作业工作流的引擎是最主要的,现在对Azkaban和Oozie进行调研。 一.Azkaban介绍 Azkaban是一个类似于Oozie的任务调度系统,它...

工作流调度系统介绍,常见工作流调度系统对比,azkaban与Oozie对比,Azkaban介绍与特性(来自学习笔记)

1. 工作流调度器azkaban1.1 概述1.1.1为什么需要工作流调度系统一个完整的数据分析系统通常都是由大量任务单元组成:shell脚本程序,java程序,mapreduce程序、hive脚本等...

三大工作流引擎对比

1.从《功夫》说起 时下的新新人类看到我,一定会认为在下是个十足的老古董,这不,《功夫》这样的片子我到今年2月底才看。不过看过《功夫》,我想的一定比一般的人多:周星星浪迹江湖,和他胖子大哥出去敲竹...
  • 1950
  • 1950
  • 2011-12-31 17:30
  • 2471

Java开源工作流比较-2

【dice_com】: 工作流系统最好不要碰,很容易陷进去出不来。 Adobe,IBM,等大公司都有自己的工作流系统,我也都研究过,都作的很好,功能与架构都是类似的,但这些大公司在中国市场都找不到...
  • 1950
  • 1950
  • 2011-12-31 17:37
  • 3358

Azkaban简介和使用

概述 为什么需要工作流调度系统 l 一个完整的数据分析系统通常都是由大量任务单元组成: shell脚本程序,java程序,mapreduce程序、hive脚本等 l 各任务单元之间存在时间先后及前后...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:深度学习:神经网络中的前向传播和反向传播算法推导
举报原因:
原因补充:

(最多只允许输入30个字)