记录 cdh6.2.1 一些版本信息

已于 2024-08-25 12:56:49 修改
阅读量720 收藏 8
点赞数 25
分类专栏： hadoop 文章标签： hadoop cdh
于 2024-08-25 12:43:41 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/hxsln11/article/details/141527525
版权

1.

[root@hadoop01 ~]# hadoop version
Hadoop 3.0.0-cdh6.2.1
Source code repository http://github.com/cloudera/hadoop -r 7ce7dcdb76fd7f700cd7749bbc6390f76ff2abae
Compiled by jenkins on 2019-09-11T08:04Z
Compiled with protoc 2.5.0
From source with checksum b6cc25ff15ff575f159489abbee34c5
This command was run using /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hadoop-common-3.0.0-cdh6.2.1.jar
[root@hadoop01 ~]#

1.1 hadoop的环境变量  hadoop-env.sh

# Prepend/Append plugin parcel classpaths

if [ "$HADOOP_USER_CLASSPATH_FIRST" = 'true' ]; then
  # HADOOP_CLASSPATH={{HADOOP_CLASSPATH_APPEND}}
  :
else
  # HADOOP_CLASSPATH={{HADOOP_CLASSPATH}}
  :
fi
# JAVA_LIBRARY_PATH={{JAVA_LIBRARY_PATH}}

export HADOOP_MAPRED_HOME=$( ([[ ! '/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce' =~ CDH_MR2_HOME ]] && echo /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce ) || echo ${CDH_MR2_HOME:-/usr/lib/hadoop-mapreduce/}  )
export YARN_OPTS="-Xmx825955249 -Djava.net.preferIPv4Stack=true $YARN_OPTS"
export HADOOP_CLIENT_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"

2.

[root@hadoop01 ~]# hive --version
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive 2.1.1-cdh6.2.1
Subversion file:///container.redhat7/build/cdh/hive/2.1.1-cdh6.2.1/rpm/BUILD/hive-2.1.1-cdh6.2.1 -r fc0582dc4d2e2342d1eef8cb702bad68e1e2cdd2
Compiled by jenkins on Wed Sep 11 01:31:00 PDT 2019
From source with checksum 6bf22b68335469e1b010bcfecc8ea570
[root@hadoop01 ~]#

Hive 2.1.1-cdh6.2.1

2.1 hive环境变量  hive-env.sh

# HIVE_AUX_JARS_PATH={{HIVE_AUX_JARS_PATH}}
# JAVA_LIBRARY_PATH={{JAVA_LIBRARY_PATH}}
export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-"$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"}"
HBASE_HIVE_DEFAULT_JAR=$(find /usr/lib/hive/lib -name hive-hbase-handler-*.jar 2> /dev/null | tail -n 1),$(sed "s: :,:g" <<< $(find /usr/lib/hbase -regextype posix-egrep -regex "/usr/lib/hbase/(hbase|hbase-client|hbase-server|hbase-protocol|hbase-common|hbase-hadoop-compat|hbase-hadoop2-compat|(lib/htrace-core.*)).jar" 2> /dev/null))
HIVE_AUX_JARS_PATH=$([[ -n $HIVE_AUX_JARS_PATH ]] && echo $HIVE_AUX_JARS_PATH,)$( ([[ ! '/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.2.1.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-hadoop-compat.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-server.jar' =~ HIVE_HBASE_JAR ]] &&  echo /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.2.1.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-hadoop-compat.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hbase/hbase-server.jar) || echo ${HBASE_HIVE_DEFAULT_JAR:-}),$(find /usr/share/java/mysql-connector-java.jar 2> /dev/null),$(find /usr/share/cmf/lib/postgresql-*jdbc*.jar 2> /dev/null | tail -n 1),$(find /usr/share/java/oracle-connector-java.jar 2> /dev/null)
export HIVE_AUX_JARS_PATH=$(echo $HIVE_AUX_JARS_PATH | sed 's/,,*/,/g' | sed 's/^,//' | sed 's/,$//')
export HIVE_OPTS="${HIVE_OPTS} --hiveconf hive.query.redaction.rules=${HIVE_CONF_DIR}/redaction-rules.json --hiveconf hive.exec.query.redactor.hooks=org.cloudera.hadoop.hive.ql.hooks.QueryRedactor"
export HADOOP_CLIENT_OPTS="-Xmx2147483648 -Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"

3.

find / -name core-site.xml

hdfs-site.xml

mapred-site.xml

yarn-site.xml

find / -name hive-site.xml

3.1 core-site.xml

[root@hadoop01 ~]# find / -name core-site.xml
/run/cloudera-scm-agent/process/223-yarn-RESOURCEMANAGER/core-site.xml
/run/cloudera-scm-agent/process/222-yarn-NODEMANAGER/core-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/hive-conf/core-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/yarn-conf/core-site.xml
/run/cloudera-scm-agent/process/231-hdfs-NAMENODE/core-site.xml
/run/cloudera-scm-agent/process/230-hdfs-DATANODE/core-site.xml
/run/cloudera-scm-agent/process/225-yarn-JOBHISTORY/core-site.xml
/etc/hive/conf.cloudera.hive/core-site.xml
/etc/hbase/conf.cloudera.hbase/core-site.xml
/etc/hadoop/conf.cloudera.yarn/core-site.xml
/etc/hadoop/conf.cloudera.hdfs/core-site.xml
/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/core-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop-kms/conf.dist/core-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/oozie/conf.dist/hadoop-conf/core-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop-httpfs/conf.empty/core-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.pseudo/core-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.impala/core-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.empty/core-site.xml
[root@hadoop01 ~]#

3.1.1 core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop01:8020</value>
  </property>
  <property>
    <name>fs.trash.interval</name>
    <value>1</value>
  </property>
  <property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
  </property>
  <property>
    <name>hadoop.security.authentication</name>
    <value>simple</value>
  </property>
  <property>
    <name>hadoop.security.authorization</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.rpc.protection</name>
    <value>authentication</value>
  </property>
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>DEFAULT</value>
  </property>
  <property>
    <name>hadoop.proxyuser.oozie.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.oozie.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.flume.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.flume.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hive.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hive.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hue.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hue.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.httpfs.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.httpfs.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hdfs.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hdfs.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.yarn.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.yarn.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.security.group.mapping</name>
    <value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
  </property>
  <property>
    <name>hadoop.security.instrumentation.requires.admin</name>
    <value>false</value>
  </property>
  <property>
    <name>net.topology.script.file.name</name>
    <value>/etc/hadoop/conf.cloudera.yarn/topology.py</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>65536</value>
  </property>
  <property>
    <name>hadoop.ssl.enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.ssl.require.client.cert</name>
    <value>false</value>
    <final>true</final>
  </property>
  <property>
    <name>hadoop.ssl.keystores.factory.class</name>
    <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
    <final>true</final>
  </property>
  <property>
    <name>hadoop.ssl.server.conf</name>
    <value>ssl-server.xml</value>
    <final>true</final>
  </property>
  <property>
    <name>hadoop.ssl.client.conf</name>
    <value>ssl-client.xml</value>
    <final>true</final>
  </property>
</configuration>

3.2  hdfs-site.xml

[root@hadoop01 ~]# find / -name hdfs-site.xml
/run/cloudera-scm-agent/process/223-yarn-RESOURCEMANAGER/hdfs-site.xml
/run/cloudera-scm-agent/process/222-yarn-NODEMANAGER/hdfs-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/hive-conf/hdfs-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/yarn-conf/hdfs-site.xml
/run/cloudera-scm-agent/process/231-hdfs-NAMENODE/hdfs-site.xml
/run/cloudera-scm-agent/process/230-hdfs-DATANODE/hdfs-site.xml
/run/cloudera-scm-agent/process/225-yarn-JOBHISTORY/hdfs-site.xml
/etc/hive/conf.cloudera.hive/hdfs-site.xml
/etc/hbase/conf.cloudera.hbase/hdfs-site.xml
/etc/hadoop/conf.cloudera.yarn/hdfs-site.xml
/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml
/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/hdfs-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop-httpfs/conf.empty/hdfs-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.pseudo/hdfs-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.impala/hdfs-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.empty/hdfs-site.xml
[root@hadoop01 ~]#

3.2.1 hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///dfs/nn</value>
  </property>
  <property>
    <name>dfs.namenode.servicerpc-address</name>
    <value>hadoop01:8022</value>
  </property>
  <property>
    <name>dfs.https.address</name>
    <value>hadoop01:9871</value>
  </property>
  <property>
    <name>dfs.https.port</name>
    <value>9871</value>
  </property>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop01:9870</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.blocksize</name>
    <value>134217728</value>
  </property>
  <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>false</value>
  </property>
  <property>
    <name>fs.permissions.umask-mode</name>
    <value>022</value>
  </property>
  <property>
    <name>dfs.client.block.write.locateFollowingBlock.retries</name>
    <value>7</value>
  </property>
  <property>
    <name>dfs.namenode.acls.enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.client.read.shortcircuit</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.domain.socket.path</name>
    <value>/var/run/hdfs-sockets/dn</value>
  </property>
  <property>
    <name>dfs.client.read.shortcircuit.skip.checksum</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.client.domain.socket.data.traffic</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
    <value>true</value>
  </property>
</configuration>

3.3  mapred-site.xml

[root@hadoop01 ~]# find / -name mapred-site.xml
/run/cloudera-scm-agent/process/223-yarn-RESOURCEMANAGER/mapred-site.xml
/run/cloudera-scm-agent/process/222-yarn-NODEMANAGER/mapred-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/hive-conf/mapred-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/yarn-conf/mapred-site.xml
/run/cloudera-scm-agent/process/225-yarn-JOBHISTORY/mapred-site.xml
/etc/hive/conf.cloudera.hive/mapred-site.xml
/etc/hadoop/conf.cloudera.yarn/mapred-site.xml
/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/mapred-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.pseudo/mapred-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.empty/mapred-site.xml
[root@hadoop01 ~]#
[root@hadoop01 ~]#

3.3.1  mapred-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>10000000</value>
  </property>
  <property>
    <name>mapreduce.job.counters.max</name>
    <value>120</value>
  </property>
  <property>
    <name>mapreduce.job.counters.groups.max</name>
    <value>50</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress</name>
    <value>false</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress.type</name>
    <value>BLOCK</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress.codec</name>
    <value>org.apache.hadoop.io.compress.DefaultCodec</value>
  </property>
  <property>
    <name>mapreduce.map.output.compress.codec</name>
    <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
  </property>
  <property>
    <name>zlib.compress.level</name>
    <value>DEFAULT_COMPRESSION</value>
  </property>
  <property>
    <name>mapreduce.task.io.sort.factor</name>
    <value>64</value>
  </property>
  <property>
    <name>mapreduce.map.sort.spill.percent</name>
    <value>0.8</value>
  </property>
  <property>
    <name>mapreduce.reduce.shuffle.parallelcopies</name>
    <value>10</value>
  </property>
  <property>
    <name>mapreduce.task.timeout</name>
    <value>600000</value>
  </property>
  <property>
    <name>mapreduce.client.submit.file.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>mapreduce.job.reduces</name>
    <value>3</value>
  </property>
  <property>
    <name>mapreduce.task.io.sort.mb</name>
    <value>256</value>
  </property>
  <property>
    <name>mapreduce.map.speculative</name>
    <value>false</value>
  </property>
  <property>
    <name>mapreduce.reduce.speculative</name>
    <value>false</value>
  </property>
  <property>
    <name>mapreduce.job.reduce.slowstart.completedmaps</name>
    <value>0.8</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>hadoop01:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop01:19888</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.https.address</name>
    <value>hadoop01:19890</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.admin.address</name>
    <value>hadoop01:10033</value>
  </property>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <value>/user</value>
  </property>
  <property>
    <name>mapreduce.am.max-attempts</name>
    <value>2</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>1024</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
    <value>1</value>
  </property>
  <property>
    <name>mapreduce.job.ubertask.enable</name>
    <value>false</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Djava.net.preferIPv4Stack=true -Xmx5120m -Xmx825955249</value>
  </property>
  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Djava.net.preferIPv4Stack=true -Xmx1800m</value>
  </property>
  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Djava.net.preferIPv4Stack=true -Xmx2800m</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.admin.user.env</name>
    <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>2048</value>
  </property>
  <property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>3072</value>
  </property>
  <property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>1</value>
  </property>
  <property>
    <name>mapreduce.job.heap.memory-mb.ratio</name>
    <value>0.8</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_CLIENT_CONF_DIR,$PWD/mr-framework/*,$MR2_CLASSPATH</value>
  </property>
  <property>
    <name>mapreduce.application.framework.path</name>
    <value>hdfs://hadoop01:8020//user/yarn/mapreduce/mr-framework/3.0.0-cdh6.2.1-mr-framework.tar.gz#mr-framework</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.jhist.format</name>
    <value>binary</value>
  </property>
  <property>
    <name>mapreduce.admin.user.env</name>
    <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value>
  </property>
  <property>
    <name>mapreduce.job.redacted-properties</name>
    <value>fs.s3a.access.key,fs.s3a.secret.key,fs.adl.oauth2.credential,dfs.adls.oauth2.credential,fs.azure.account.oauth2.client.secret</value>
  </property>
  <property>
    <name>mapreduce.job.acl-view-job</name>
    <value> </value>
  </property>
  <property>
    <name>mapreduce.job.acl-modify-job</name>
    <value> </value>
  </property>
  <property>
    <name>mapreduce.cluster.acls.enabled</name>
    <value>false</value>
  </property>
</configuration>

3.4 yarn-site.xml

[root@hadoop01 ~]# find / -name yarn-site.xml
/run/cloudera-scm-agent/process/223-yarn-RESOURCEMANAGER/yarn-site.xml
/run/cloudera-scm-agent/process/222-yarn-NODEMANAGER/yarn-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/hive-conf/yarn-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/yarn-conf/yarn-site.xml
/run/cloudera-scm-agent/process/225-yarn-JOBHISTORY/yarn-site.xml
/etc/hive/conf.cloudera.hive/yarn-site.xml
/etc/hadoop/conf.cloudera.yarn/yarn-site.xml
/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/yarn-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.pseudo/yarn-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hadoop/conf.empty/yarn-site.xml
[root@hadoop01 ~]#

3.4.1 yarn-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
  <property>
    <name>yarn.acl.enable</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.admin.acl</name>
    <value>*</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop01:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop01:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop01:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop01:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop01:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.https.address</name>
    <value>hadoop01:8090</value>
  </property>
  <property>
    <name>yarn.resourcemanager.client.thread-count</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.client.thread-count</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.client.thread-count</name>
    <value>1</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>512</value>
  </property>
  <property>
    <name>yarn.scheduler.increment-allocation-mb</name>
    <value>512</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>5632</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
  </property>
  <property>
    <name>yarn.scheduler.increment-allocation-vcores</name>
    <value>1</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>4</value>
  </property>
  <property>
    <name>yarn.resourcemanager.amliveliness-monitor.interval-ms</name>
    <value>1000</value>
  </property>
  <property>
    <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
    <value>600000</value>
  </property>
  <property>
    <name>yarn.resourcemanager.am.max-attempts</name>
    <value>2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>
    <value>600000</value>
  </property>
  <property>
    <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
    <value>1000</value>
  </property>
  <property>
    <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
    <value>600000</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
  </property>
  <property>
    <name>yarn.resourcemanager.max-completed-applications</name>
    <value>10000</value>
  </property>
  <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
  </property>
  <property>
    <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
    <value>logs</value>
  </property>
</configuration>

3.5  hive-site.xml

[root@hadoop01 ~]# find / -name hive-site.xml
/run/cloudera-scm-agent/process/199-oozie-OOZIE_SERVER/hive-conf/hive-site.xml
/etc/hive/conf.cloudera.hive/hive-site.xml
/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/etc/hive/conf.dist/hive-site.xml
[root@hadoop01 ~]#

3.5.1 hive-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://hadoop02:9083</value>
  </property>
  <property>
    <name>hive.metastore.client.socket.timeout</name>
    <value>300</value>
  </property>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
  </property>
  <property>
    <name>hive.warehouse.subdir.inherit.perms</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.auto.convert.join</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.auto.convert.join.noconditionaltask.size</name>
    <value>20971520</value>
  </property>
  <property>
    <name>hive.optimize.bucketmapjoin.sortedmerge</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.smbjoin.cache.rows</name>
    <value>10000</value>
  </property>
  <property>
    <name>hive.server2.logging.operation.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/var/log/hive/operation_logs</value>
  </property>
  <property>
    <name>mapred.reduce.tasks</name>
    <value>-1</value>
  </property>
  <property>
    <name>hive.exec.reducers.bytes.per.reducer</name>
    <value>67108864</value>
  </property>
  <property>
    <name>hive.exec.copyfile.maxsize</name>
    <value>33554432</value>
  </property>
  <property>
    <name>hive.exec.reducers.max</name>
    <value>1099</value>
  </property>
  <property>
    <name>hive.vectorized.groupby.checkinterval</name>
    <value>4096</value>
  </property>
  <property>
    <name>hive.vectorized.groupby.flush.percent</name>
    <value>0.1</value>
  </property>
  <property>
    <name>hive.compute.query.using.stats</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.vectorized.execution.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.vectorized.execution.reduce.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.vectorized.use.vectorized.input.format</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.vectorized.use.checked.expressions</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.vectorized.use.vector.serde.deserialize</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.vectorized.adaptor.usage.mode</name>
    <value>chosen</value>
  </property>
  <property>
    <name>hive.vectorized.input.format.excludes</name>
    <value>org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat</value>
  </property>
  <property>
    <name>hive.merge.mapfiles</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.merge.mapredfiles</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.cbo.enable</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.fetch.task.conversion</name>
    <value>minimal</value>
  </property>
  <property>
    <name>hive.fetch.task.conversion.threshold</name>
    <value>268435456</value>
  </property>
  <property>
    <name>hive.limit.pushdown.memory.usage</name>
    <value>0.1</value>
  </property>
  <property>
    <name>hive.merge.sparkfiles</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.merge.smallfiles.avgsize</name>
    <value>16777216</value>
  </property>
  <property>
    <name>hive.merge.size.per.task</name>
    <value>268435456</value>
  </property>
  <property>
    <name>hive.optimize.reducededuplication</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.optimize.reducededuplication.min.reducer</name>
    <value>4</value>
  </property>
  <property>
    <name>hive.map.aggr</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.map.aggr.hash.percentmemory</name>
    <value>0.5</value>
  </property>
  <property>
    <name>hive.optimize.sort.dynamic.partition</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.execution.engine</name>
    <value>mr</value>
  </property>
  <property>
    <name>spark.executor.memory</name>
    <value>912680550b</value>
  </property>
  <property>
    <name>spark.driver.memory</name>
    <value>966367641b</value>
  </property>
  <property>
    <name>spark.executor.cores</name>
    <value>4</value>
  </property>
  <property>
    <name>spark.yarn.driver.memoryOverhead</name>
    <value>102m</value>
  </property>
  <property>
    <name>spark.yarn.executor.memoryOverhead</name>
    <value>153m</value>
  </property>
  <property>
    <name>spark.dynamicAllocation.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>spark.dynamicAllocation.initialExecutors</name>
    <value>1</value>
  </property>
  <property>
    <name>spark.dynamicAllocation.minExecutors</name>
    <value>1</value>
  </property>
  <property>
    <name>spark.dynamicAllocation.maxExecutors</name>
    <value>2147483647</value>
  </property>
  <property>
    <name>hive.metastore.execute.setugi</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.support.concurrency</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.zookeeper.quorum</name>
    <value>hadoop02,hadoop01</value>
  </property>
  <property>
    <name>hive.zookeeper.client.port</name>
    <value>2181</value>
  </property>
  <property>
    <name>hive.zookeeper.namespace</name>
    <value>hive_zookeeper_namespace_hive</value>
  </property>
  <property>
    <name>hive.cluster.delegation.token.store.class</name>
    <value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value>
  </property>
  <property>
    <name>hive.server2.enable.doAs</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.server2.use.SSL</name>
    <value>false</value>
  </property>
  <property>
    <name>spark.shuffle.service.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.strict.checks.orderby.no.limit</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.strict.checks.no.partition.filter</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.strict.checks.type.safety</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.strict.checks.cartesian.product</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.strict.checks.bucketing</name>
    <value>true</value>
  </property>
</configuration>

4.

[root@hadoop01 ~]# hbase version
错误: 找不到或无法加载主类 org.apache.hadoop.hbase.util.GetJavaProperty
HBase 2.1.0-cdh6.2.1
Source code repository file:///container.redhat7/build/cdh/hbase/2.1.0-cdh6.2.1/rpm/BUILD/hbase-2.1.0-cdh6.2.1 revision=Unknown
Compiled by jenkins on Wed Sep 11 01:05:56 PDT 2019
From source with checksum 0fdfcb6ec3a5278e944b4941476771dd

4.1

[root@hadoop01 ~]# hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.0-cdh6.2.1, rUnknown, Wed Sep 11 01:05:56 PDT 2019
Took 0.0054 seconds
hbase(main):001:0>
hbase(main):002:0*

create 'testtable', 'cf'
put 'testtable', 'row1', 'cf:col1', 'value1'
scan 'testtable'

5.http://hadoop01:7180/cmf/clusters/1/add-service/index

添加所所需要的组件

6.sqoop

[root@hadoop01 ~]# sqoop version
Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
24/07/21 02:37:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1
Sqoop 1.4.7-cdh6.2.1
git commit id
Compiled by jenkins on Wed Sep 11 01:28:03 PDT 2019
[root@hadoop01 ~]#

#!/bin/bash
 
# 检查Sqoop是否安装
if sqoop version >/dev/null 2>&1; then
  echo "Sqoop is installed."
else
  echo "Sqoop is not installed."
fi


 


 

hxsln11
关注
专栏目录
CDH6.2.1CM6.2.1版本部署操作文档
cyg521的博客
05-21 2036
1简介 1.1Cloudera Data Hub介绍 CDH是最完整、测试最全面和最流行的Apache Hadoop及相关项目发布版本。 CDH包含Hadoop核心元素：可扩展存储和分布式计算引擎，同时提供了Web接口和企业级功能。 CDH基于Apache许可证开源协议，提供了包括批处理、交互式SQL、交互式查询和基于角色的访问控制一站式Hadoop大数据解决方案。 CDH优势： 灵活 — 存储任意类型数据，同时可以利用多种计算框架（批处理、交互式SQL、文本检索、机器学习...
Centos7.7 CDH6.2.1 安装教程
Java_Road_Far的博客
04-04 3233
文章目录此教程以 6.2.0 的版本为例一、下载 此教程以 6.2.0 的版本为例 一、下载 CDH下载地址 下载cdh 选择对应linux的下载，例如我下载的是 centos7.7 下使用的cdh6.2.0 下载cm ...
[CDH]企业数据仓库数据管理平台CDH6.2.1离线安装记录
qq_45661358的博客
10-29 2105
# [CDH]企业数据仓库数据管理平台CDH6.2.1离线安装记录
CDH6.2.1搭建（史上最全的安装教程）
热门推荐
summer089089的博客
07-27 1万+
1.Cloudera Manager 1.1 CM简介 1.1.1 CM简介 Cloudera Manager是一个拥有集群自动化安装、中心化管理、集群监控、报警功能的一个工具，使得安装集群从几天的时间缩短在几个小时内，运维人员从数十人降低到几人以内，极大的提高集群管理的效率。 1.1.2 CM架构 1.2 CM部署准备配置 1.2.1 修改hosts文件 [root@node01 ~]# vim /etc/hosts 127.0.0.1 localhost localhost ::1 loc
CDH6.2.1 Hive与阿里Hbase增强版集成记录
欲望以提升热忱 毅力以磨平高山
10-17 355
说明：采用阿里hbase与自己的hive做集成，非使用阿里HBASE的请忽略。 第一种：建立 Hive 内部表并自动生成Hbase表 目标：建立 Hive 表，关联 HBase 表，插入数据到 Hive 表的同时能够影响 HBase 表。 进入hive -e hive>set hbase.zookeeper.quorum=ld-xxx.hbaseue.rds.aliyuncs.com:30020; hive>set hbase.client.username=xxx; hive&g.
大数据平台 CDH6.2.1 搭建
qq_36306519的博客
04-26 1944
大数据平台 CDH6.2.1 搭建
Centos7.8+CDH6.2.1从零开始安装
百看不如一RUN
06-06 2888
前言，为什么我要开始重新装一遍呢，因为，我攒了个台式机，还没有装CDH，学习需要用，顺道记录一下安装记录，当做以后的笔记，以后查漏补缺。 一、CDH是什么？ CDH是Cloudera的100%开源平台发行版，包括Apache Hadoop，专门为满足企业需求而构建。CDH提供了企业使用所需的一切。通过将Hadoop与十几个其他关键的开源项目集成，Cloudera创建了一个功能先进的系统，帮助您执行端到端的大数据工作流。 1.1 CDH的重点项目有哪些？ 相信大家对这些大数据组件应该都是很熟悉了，那么我.
自制Flink Parcel集成CDH（Flink1.13.2 + CDH6.2.1+Scala2.11）
weixin_44389063的博客
08-10 1564
自制Flink Parcel集成CDH（Flink1.13.2 + CDH6.2.1+Scala2.11） 记录制作flink parcel 环境（虚拟机，系统CentOS7.6） （1）jdk1.8（open） （2）maven3.8.1 （3）parcel制作工具 jdk、maven自行安装 flink 相关下载 注：可不需要提前下载 flink下载地址：https://archive.apache.org/dist/flink/flink-1.13.2/flink-1.13.2-bin-scal
搭建CDH错误记录(踩坑无数)
Mr_Robot的博客
04-18 655
搭建CDH遇到的问题，以及解决方法
livy-0.7.1-incubating-bin-cdh6.2.1.zip
11-20
在本案例中，我们关注的是"Livy-0.7.1-incubating-bin-cdh6.2.1.zip"，这是一个针对CDH（Cloudera Distribution Including Apache Hadoop）6.2.1版本编译的Livy二进制发行版。 **Livy的核心功能：** Livy的主要目标...
Cloudera Manager6.2.1及CDH最新版本安装全程记录
12-22
Cloudera Manager6.2.1
CDH 6.3.1 ranger ranger-2.1.0-admin.tar.gz
11-04
CDH 6.3.1 版本是其中的一个重要迭代，它包含了众多组件的更新和优化，旨在提升数据处理效率、安全性以及整体性能。Ranger是CDH中的核心组件之一，主要负责提供全面的数据管理和安全策略实施。在CDH 6.3.1中，Ranger...
WPF+MVVM案例实战（十九）- 自定义字体图标按钮的封装与实现（EF类）
lingshuangcanxue
11-02 153
各种类型的自定义按钮实现
Hadoop——HDFS
weixin_53891720的博客
11-03 500
HDFS（Hadoop Distributed File System）是Apache Hadoop的核心组件之一，是一个分布式文件系统，专门设计用于在大规模集群上存储和管理海量数据。它的设计目标是提供高吞吐量的数据访问和容错能力，以支持大数据处理任务。下面是HDFS的主要特性和功能：HDFS是一个为处理和存储海量数据而设计的分布式文件系统，具备高容错性、高吞吐量和可扩展性等优点。它通过将数据分割成多个块并在集群中分布存储，确保了数据的高可用性和高性能，成为大数据处理的基础设施之一。在HDFS（Hadoop
Hadoop-MapReduce任务三种运行开发模式
Yz9876的博客
10-31 456
数据在hdfs上，代码在本地，使用本机的电脑的资源运行我们的MR。输入和输出路径指的都是本地路径，运行时耗费的资源也是本地资源。数据在本地，代码也在本地，使用本机的电脑的资源运行我们的MR。这个里面的输入和输出路径指的是hdfs上的路径。数据在hdfs上，代码 跑 在yarn上。
Hadoop-002-部署并配置HDFS集群
smilefyou的博客
10-30 807
第一篇前期环境准备后, 本文主要是HDFS的部署完整配置, 直到搭建完成成功访问 http://hadoop-1:9870
hadoop_mapred-site.xml
小太阳的代码之路
10-31 537
hadoop3.2.3的高可用集群mapred-site.xml配置实例。
WPF使用prism框架发布订阅实现消息提示
最新发布
xll的博客
11-03 178
我们在使用wpf中自带的消息提示弹出框MessageBox.Show()十分笨拙，那么怎么去使用prism框架的发布订阅去实现消息提示呢？
20241102-Windows 10上安装虚拟机VMware10.0.2、Hadoop3.3.6与jdk1.8.0
Chain
11-03 508
Windows 10 专业版上安装虚拟机 VMware10.0.2、Xshell8.0、Hadoop3.3.6 与 jdk1.8.0【命令式】 笔记、笔记、笔记、笔记 1. 写在前面 好片： Hadoop 保姆级超详细安装教程_哔哩哔哩_bilibili ——文章的从虚拟机到 Hadoop 安装【主】 好文： Hadoop3.3.6 完全分布式集群【三个节点】的安装配置-CSDN 博客 ——文章的 Hadoop 安装的 Hadoop 文件配置等 Windows 10 端提前安装好必要的软件：VMware
下载：Redhat7上的CM6.2.1与CDH6.2.1资源包
安装和部署CM6.2.1和CDH6.2.1时，首先需要确保系统满足所有先决条件，例如兼容的操作系统版本、正确的硬件配置和网络设置。然后，按照Cloudera官方文档的步骤进行安装，包括安装JDK、配置CM服务器和代理、上传并激活...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值