CDH 中hbaese, mapreduce内存控制

1, hbase启动失败：OutOfMemoryError

由于内存不够，导入大批量数据时发生内存溢出，所以需要调整内存分配，在浏览器打开7180集群管理界面，进入到hbase

1, 配置–>输入框搜索：java

• 优化值： master堆大小–> 4G
• 优化值： regionserver堆大小–> 15G
  

2，mapreduce运行卡顿

查看/var/log/ hadoop-cmf-yarn-JOBHISTORY-node1.cluster.log.out ,提示gc频繁，所以想到给mapreduce分配的内存不足, 需要加大内存

• nodemanager.resource.memory-mb = scheduler.maximum-allocation-mb =60G
• 优化值： map堆大小–> 10G, map容器内存–>12G
• 优化值： reduce堆大小–> 20G, reduce容器内存–>24G

2019-02-23 08:00:21,593 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[JOBHISTORY], CATEGORY=[LOG_MESSAGE], ROLE=[yarn-JOBHISTORY-a6c222f47368c75014f30b4eb3a45e95], SEVERITY=[IMPORTANT], SERVICE=[yarn], HOST_IDS=[f64f5e51-45af-4278-88e1-b18ccaea3de6], SERVICE_TYPE=[YARN], LOG_LEVEL=[WARN], HOSTS=[node1.cluster], EVENTCODE=[EV_L
OG_EVENT]}, content=Detected pause in JVM or host machine (eg GC): pause of approximately 14944ms
GC pool 'ParNew' had collection(s): count=1 time=14631ms, timestamp=1550926817327}
2019-02-23 10:19:51,816 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1009ms
No GCs detected
2019-02-23 10:35:18,805 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5783ms
GC pool 'ParNew' had collection(s): count=1 time=5852ms

修改mapred-site.xml中的map , reduce的内存参数

• io.sort.mb map缓冲区大小//默认：256m -->512
• io.sort.factor 并行处理spill的个数//默认：64 -->80
  在浏览器打开7180集群管理界面，进入到yarn

1,配置–>输入框搜索：java

会出现对应的java堆内存参数，修改后保存

2, 配置–>输入框搜索：memory