安装好hawq,参考:http://blog.csdn.net/u011563666/article/details/78814520
hawq配置yarn支持分为yarn和yarn HA。本文简单介绍hawq配置yarn。配置yarn HA请参考:
http://hawq.incubator.apache.org/docs/userguide/2.1.0.0-incubating/resourcemgmt/YARNIntegration.html#highlyavailableyarn
1. 安装好hadoop HDFS、YARN系统
2. 修改yarn配置文件yarn-site.xml
添加或修改:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>xxx.xxx.xxx.xxx/resouremanager hostname</value>
</property>
3. 修改capacity-scheduler.xml
给调度器新增一个队列:hawqque
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default,hawqque</value>
</property>
为hawqque配置参数
<property>
<name>yarn.scheduler.capacity.root.hawqque.capacity</name>
<value>50</value>
<description>hawqque queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.hawqque.user-limit-factor</name>
<value>1</value>
<description>
hawqque queue user limit a percentage from 0.0 to 1.0.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.hawqque.maximum-capacity</name>
<value>80</value>
<description>
The maximum capacity of the hawqque queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.hawqque.state</name>
<value>RUNNING</value>
<description>
The state of the hawqque queue. State can be one of RUNNING or STOPPED.
</description>
</property>
配置队列权限:
<property>
<name>yarn.scheduler.capacity.root.hawqque.acl_submit_applications</name>
<value>*</value>
<description>
The ACL of who can submit jobs to the hawqque queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.hawqque.acl_administer_queue</name>
<value>*</value>
<description>
The ACL of who can administer jobs on the hawqque queue.
</description>
</property>
4.配置hawq,启用yarn
修改/usr/local/apache-hawq/etc/hawq-size.xml
vim /usr/local/apache-hawq/etc/hawq-site.xml
修改hawq_global_rm_type参数为yarn,同时配置好yarn参数地址,及hawq的调度对了名称hawqque
<property>
<name>hawq_global_rm_type</name>
<value>yarn</value>
<description>The resource manager type to start for allocating resource.
'none' means hawq resource manager exclusively uses whole
cluster; 'yarn' means hawq resource manager contacts YARN
resource manager to negotiate resource.
</description>
</property>
<property>
<name>hawq_rm_yarn_address</name>
<value>x.x.x.x/resouremanager hostname:8032</value>
<description>The address of YARN resource manager server.</description>
</property>
<property>
<name>hawq_rm_yarn_scheduler_address</name>
<value>x.x.x.x/resourcemanager hostname:8030</value>
<description>The address of YARN scheduler server.</description>
</property>
<property>
<name>hawq_rm_yarn_queue_name</name>
<value>hawqque</value>
<description>The YARN queue name to register hawq resource manager.</description>
</property>
5. 初始化hawq集群
保证hdfs、yarn系统运行正常,执行:
hawq init cluster
6. 查看yarn页面,确认启动成功
http://xxx.xx.xxx.xx:8088/
查看scheduler,查看Applications->running,会有hawq的YARN类型Job。