Hive on Spark：扩充队列

最新推荐文章于 2022-12-09 22:01:50 发布

常小白_J

最新推荐文章于 2022-12-09 22:01:50 发布

阅读量500

点赞数

文章标签： hive spark 大数据数据仓库

本文链接：https://blog.csdn.net/weixin_55842692/article/details/122343926

版权

扩充队列
   HQL的任务是有优先级的，如何保证有优先级的任务能够有充足的资源保障呢？
   扩充一个子队列，只能提交hive任务。
   将$HADOOP_HOME/etc/hadoop/capacity-schdualer.xml替换成以下内容：


		<!--
		Licensed under the Apache License, Version 2.0 (the "License");
		you may not use this file except in compliance with the License.
		You may obtain a copy of the License at
			http://www.apache.org/licenses/LICENSE-2.0
		Unless required by applicable law or agreed to in writing, software
		distributed under the License is distributed on an "AS IS" BASIS,
		WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
		See the License for the specific language governing permissions and
		limitations under the License. See accompanying LICENSE file.
		-->
		<configuration>
		<!-- 容量调度器最多可以容纳多少个job-->
		<property>
			<name>yarn.scheduler.capacity.maximum-applications</name>
			<value>10000</value>
			<description>
			Maximum number of applications that can be pending and running.
			</description>
		</property>
		
		<!-- 当前队列中启动的MRAppMaster进程，所占用的资源可以达到队列总资源的多少
				通过这个参数可以限制队列中提交的Job数量  -->
		<property>
			<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
			<value>0.5</value>
			<description>
			Maximum percent of resources in the cluster wh