国产化Kylin V10上搭建hadoop集群

最新推荐文章于 2025-03-16 15:16:30 发布
最新推荐文章于 2025-03-16 15:16:30 发布
阅读量1k 收藏 7
点赞数 4
文章标签： kylin hadoop 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u011564831/article/details/142732207
版权

文章目录

概要

目前程序员的业务普遍要结合国产化，可能是互联网大厂不景气下ToB的一个业务方向，我所在的公司普遍要求Arm架构Kylin系统的机器上实现国产化
本问介绍国产化Kylin V10上搭建hadoop集群的步骤

最终实现在k8s 集群上
hadoop-hadoop-yarn-rm 1 节点

hadoop-hadoop-yarn-proxyserver 1 节点

hadoop-hadoop-yarn-nm 3节点

hadoop-hadoop-hdfs-nn 1节点

hadoop-hadoop-hdfs-dn 3节点

在这里插入图片描述

整体架构流程

uname -a 查看我使用的机器系统

Linux node42 4.19.90-52.36.v2207.ky10.aarch64 #4 SMP Thu Mar 14 22:54:32 CST 2024 aarch64 aarch64 aarch64 GNU/Linux

参考开源项目里定义的启动bootstrap.sh

#!/usr/bin/env sh
source /etc/profile

wait_for() {
    if [ -n "$1" ] && [ -n "$2" ]; then
        echo "Waiting for $1 to listen on $2..."
        while ! nc -z $1 $2; do
            echo "waiting..."
            echo "Host: $1, Port: $2"
            sleep 1s
        done
    else
        echo "Error: Both host and port must be provided."
    fi
}

start_hdfs_namenode() {
        namenode_dir=`grep -A1 'dfs.namenode.name.dir' ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml |tail -1|sed 's/<value>//'|sed 's/<\/value>//'`
        if [ ! -d ${namenode_dir}/current ];then
           ${HADOOP_HOME}/bin/hdfs namenode -format
        fi

        ${HADOOP_HOME}/bin/hdfs --loglevel INFO --daemon start namenode

        tail -f ${HADOOP_HOME}/logs/*namenode*.log
}

start_hdfs_datanode() {

        wait_for $1 $2

        ${HADOOP_HOME}/bin/hdfs --loglevel INFO --daemon start datanode

        tail -f ${HADOOP_HOME}/logs/*datanode*.log
}

start_yarn_resourcemanager() {

        ${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start resourcemanager

        tail -f ${HADOOP_HOME}/logs/*resourcemanager*.log
}

start_yarn_nodemanager() {

        wait_for $1 $2

        ${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start nodemanager

        tail -f ${HADOOP_HOME}/logs/*nodemanager*.log
}

start_yarn_proxyserver() {

        wait_for $1 $2

        ${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start proxyserver

        tail -f ${HADOOP_HOME}/logs/*proxyserver*.log
}

start_mr_historyserver() {

        wait_for $1 $2

        ${HADOOP_HOME}/bin/mapred --loglevel INFO  --daemon  start historyserver

        tail -f ${HADOOP_HOME}/logs/*historyserver*.log
}

start_hive_metastore() {

        if [ ! -f ${HIVE_HOME}/formated ];then
                schematool -initSchema -dbType mysql --verbose >  ${HIVE_HOME}/formated
        fi

        $HIVE_HOME/bin/hive --service metastore

}

start_hive_hiveserver2() {

        $HIVE_HOME/bin/hive --service hiveserver2
}


case $1 in
        hadoop-hdfs-nn)
                start_hdfs_namenode $2 $3
                ;;
        hadoop-hdfs-dn)
                start_hdfs_datanode $2 $3
                ;;
        hadoop-yarn-rm)
                start_yarn_resourcemanager $2 $3
                ;;
        hadoop-yarn-nm)
                start_yarn_nodemanager $2 $3
                ;;
        hadoop-yarn-proxyserver)
                start_yarn_proxyserver $2 $3
                ;;
        hadoop-mr-historyserver)
                start_mr_historyserver $2 $3
                ;;
        hive-metastore)
                start_hive_metastore $2 $3
                ;;
        hive-hiveserver2)
                start_hive_hiveserver2 $2 $3
                ;;
        *)
                echo "请输入正确的服务启动命令~"
        ;;

技术细节

构建arm镜像需要的基本组件(自己百度下载arm版hadoop，jdk, hive 和spark, flink)
在这里插入图片描述

FROM centos:centos7.9.2009

RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone

USER root

RUN export LANG=zh_CN.UTF-8

# 创建用户和用户组，跟yaml编排里的user: 10000:10000
RUN groupadd --system --gid=10000 hadoop && useradd --system --home-dir /home/hadoop --uid=10000 --gid=hadoop hadoop

# 安装sudo
#RUN mkdir -p /etc/sudoers ; yum -y install sudo ; chmod 640 /etc/sudoers

# 给hadoop添加sudo权限
RUN echo "hadoop ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

RUN yum -y install net-tools telnet wget nc expect which bind-utils

#RUN mkdir /opt/apache/

RUN mkdir -p /opt/apache/hadoop/logs

# 安装 JDK
ADD jdk-8u421-linux-aarch64.tar.gz /opt/apache/
ENV JAVA_HOME /opt/apache/jdk1.8.0_421
ENV PATH $JAVA_HOME/bin:$PATH

# 配置 Hadoop
ENV HADOOP_VERSION 3.3.6
ADD hadoop-${HADOOP_VERSION}.tar.gz /opt/apache/
ENV HADOOP_HOME /opt/apache/hadoop-${HADOOP_VERSION}
RUN ln -s $HADOOP_HOME /opt/apache/hadoop

ENV HADOOP_COMMON_HOME=${HADOOP_HOME} \
    HADOOP_HDFS_HOME=${HADOOP_HOME} \
    HADOOP_MAPRED_HOME=${HADOOP_HOME} \
    HADOOP_YARN_HOME=${HADOOP_HOME} \
    HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop \
    PATH=${PATH}:${HADOOP_HOME}/bin

# 配置Hive
ENV HIVE_VERSION 3.1.3
ADD apache-hive-${HIVE_VERSION}-bin.tar.gz /opt/apache/
ENV HIVE_HOME=/opt/apache/hive
ENV PATH=$HIVE_HOME/bin:$PATH
RUN ln -s /opt/apache/apache-hive-${HIVE_VERSION}-bin ${HIVE_HOME}

# 配置spark
ENV SPARK_VERSION 3.3.2
ADD spark-${SPARK_VERSION}-bin-hadoop3.tgz /opt/apache/
ENV SPARK_HOME=/opt/apache/spark
ENV PATH=$SPARK_HOME/bin:$PATH
RUN ln -s /opt/apache/spark-${SPARK_VERSION}-bin-hadoop3 ${SPARK_HOME}

# 配置 flink
ENV FLINK_VERSION 1.17.0
ADD flink-${FLINK_VERSION}-bin-scala_2.12.tgz /opt/apache/
ENV FLINK_HOME=/opt/apache/flink
ENV PATH=$FLINK_HOME/bin:$PATH
RUN ln -s /opt/apache/flink-${FLINK_VERSION} ${FLINK_HOME}

# 创建namenode、datanode存储目录
RUN mkdir -p /opt/apache/hadoop/data/{hdfs,yarn} /opt/apache/hadoop/data/hdfs/namenode /opt/apache/hadoop/data/hdfs/datanode/data{1..3} /opt/apache/hadoop/data/yarn/{local-dirs,log-dirs,apps}

COPY bootstrap.sh /opt/apache/

COPY config/* ${HADOOP_HOME}/etc/hadoop/

# hive  config
COPY hive-config/* ${HIVE_HOME}/conf/

COPY mysql-connector-java-5.1.49/mysql-connector-java-5.1.49-bin.jar ${HIVE_HOME}/lib/

RUN sudo mkdir -p /home/hadoop/ && sudo chown -R hadoop:hadoop /home/hadoop/

#RUN yum -y install which
ENV ll "ls -l"

RUN chown -R hadoop:hadoop /opt/apache
WORKDIR /opt/apache

构建镜像

docker build -t hadoop_hive:arm .

这里直接给出hadoop的部署的yaml文件

---
# Source: hadoop/templates/hdfs/hdfs-dn-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-hdfs-dn
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-dn
  minAvailable: 1
---
# Source: hadoop/templates/hdfs/hdfs-nn-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-hdfs-nn
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-nn
  minAvailable: 1
---
# Source: hadoop/templates/hive/hive-hiveserver2-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-hive-hiveserver2
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-hiveserver2
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hive-hiveserver2
  minAvailable: 1
---
# Source: hadoop/templates/hive/hive-metastore-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-hive-metastore
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-metastore
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hive-metastore
  minAvailable: 1
---
# Source: hadoop/templates/mr-historyserver/mr-historyserver-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-mr-historyserver
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: mr-historyserver
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: mr-historyserver
  minAvailable: 1
---
# Source: hadoop/templates/yarn-proxyserver/yarn-proxyserver-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-yarn-proxyserver
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-proxyserver
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-proxyserver
  minAvailable: 1
---
# Source: hadoop/templates/yarn/yarn-nm-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-yarn-nm
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-nm
  minAvailable: 1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-hadoop-hdfs-dn-0
  namespace: hadoop
data:
  hdfs-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration><property>
          <name>dfs.webhdfs.enabled</name>
          <value>true</value>
      </property><property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
          <name>dfs.datanode.hostname</name>
          <value>hadoop-hadoop-hdfs-dn-0.hadoop-hadoop-hdfs-dn.hadoop.svc.cluster.local</value>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:9864</value>
      </property>

      <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:9866</value>
      </property>

      <property>
        <name>dfs.replication</name>
          <value>3</value>
      </property>

      <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data1,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data2,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data3</value>
        <description>DataNode directory</description>
      </property>

      <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/namenode</value>
        <description>NameNode directory for namespace and transaction logs storage.</description>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>dfs.namenode.rpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

    </configuration>
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-hadoop-hdfs-dn-1
  namespace: hadoop
data:
  hdfs-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration><property>
          <name>dfs.webhdfs.enabled</name>
          <value>true</value>
      </property><property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
          <name>dfs.datanode.hostname</name>
          <value>hadoop-hadoop-hdfs-dn-1.hadoop-hadoop-hdfs-dn.hadoop.svc.cluster.local</value>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:9864</value>
      </property>

      <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:9866</value>
      </property>

      <property>
        <name>dfs.replication</name>
          <value>3</value>
      </property>

      <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data1,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data2,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data3</value>
        <description>DataNode directory</description>
      </property>

      <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/namenode</value>
        <description>NameNode directory for namespace and transaction logs storage.</description>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>dfs.namenode.rpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

    </configuration>
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-hadoop-hdfs-dn-2
  namespace: hadoop
data:
  hdfs-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration><property>
          <name>dfs.webhdfs.enabled</name>
          <value>true</value>
      </property><property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
          <name>dfs.datanode.hostname</name>
          <value>hadoop-hadoop-hdfs-dn-2.hadoop-hadoop-hdfs-dn.hadoop.svc.cluster.local</value>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:9864</value>
      </property>

      <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:9866</value>
      </property>

      <property>
        <name>dfs.replication</name>
          <value>3</value>
      </property>

      <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data1,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data2,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data3</value>
        <description>DataNode directory</description>
      </property>

      <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/namenode</value>
        <description>NameNode directory for namespace and transaction logs storage.</description>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>dfs.namenode.rpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

    </configuration>

---
# Source: hadoop/templates/yarn/yarn-rm-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-yarn-rm
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-rm
  minAvailable: 1
---
# Source: hadoop/templates/hadoop-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-hadoop
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
data:
  core-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://hadoop-hadoop-hdfs-nn-0.hadoop-hadoop-hdfs-nn.hadoop.svc.cluster.local:9000/</value>
          <description>NameNode URI</description>
      </property>
      <property>
          <name>hadoop.proxyuser.root.hosts</name>
          <value>*</value>
      </property>
      <property>
          <name>hadoop.proxyuser.root.groups</name>
          <value>*</value>
      </property>
      <property>
          <name>hadoop.proxyuser.hadoop.hosts</name>
          <value>*</value>
      </property>
      <property>
          <name>hadoop.proxyuser.hadoop.groups</name>
          <value>*</value>
      </property>
    </configuration>

  hdfs-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration><property>
          <name>dfs.webhdfs.enabled</name>
          <value>true</value>
      </property><property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <!--
      <property>
        <name>dfs.datanode.hostname</name>
        <value>example.com</value>
      </property>
      -->

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:9864</value>
      </property>

      <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:9866</value>
      </property>

      <property>
        <name>dfs.replication</name>
          <value>3</value>
      </property>

      <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data1,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data2,/opt/apache/hadoop-3.3.6/data/hdfs/datanode/data3</value>
        <description>DataNode directory</description>
      </property>

      <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/hdfs/namenode</value>
        <description>NameNode directory for namespace and transaction logs storage.</description>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>dfs.namenode.rpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

    </configuration>

  mapred-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>
      <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
      </property>
      <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-hadoop-mr-historyserver-0:10020</value>
      </property>
      <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-hadoop-mr-historyserver-0:19888</value>
      </property>
    </configuration>

  yarn-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>
      <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-hadoop-yarn-rm-headless</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>yarn.resourcemanager.bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>yarn.nodemanager.bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>yarn.timeline-service.bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

      <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
      </property>

      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
      </property>

      <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>

      <property>
        <description>List of directories to store localized files in.</description>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/opt/apache/hadoop-3.3.6/data/yarn/local-dirs</value>
      </property>

      <property>
        <description>Where to store container logs.</description>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/opt/apache/hadoop-3.3.6/data/yarn/log-dirs</value>
      </property>

      <property>
        <description>Where to aggregate logs to.</description>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/opt/apache/hadoop-3.3.6/data/yarn/apps</value>
      </property>

      <property>
        <name>yarn.web-proxy.address</name>
        <value>hadoop-hadoop-yarn-proxyserver-0:9111</value>
      </property>

      <property>
        <name>yarn.application.classpath</name>
        <value>
          /opt/apache/hadoop-3.3.6/etc/hadoop,
          /opt/apache/hadoop-3.3.6/share/hadoop/common/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/common/lib/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/hdfs/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/hdfs/lib/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/mapreduce/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/mapreduce/lib/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/yarn/*,
          /opt/apache/hadoop-3.3.6/share/hadoop/yarn/lib/*
        </value>
      </property>
    </configuration>
  dfs-hosts.includes: |
    hadoop-hadoop-hdfs-dn-0.hadoop-hadoop-hdfs-dn.hadoop.svc.cluster.local
    hadoop-hadoop-hdfs-dn-1.hadoop-hadoop-hdfs-dn.hadoop.svc.cluster.local
    hadoop-hadoop-hdfs-dn-2.hadoop-hadoop-hdfs-dn.hadoop.svc.cluster.local
  dfs-hosts.excludes: |
  yarn-hosts.includes: |
    hadoop-hadoop-yarn-nm-0.hadoop-hadoop-yarn-nm.hadoop.svc.cluster.local
    hadoop-hadoop-yarn-nm-1.hadoop-hadoop-yarn-nm.hadoop.svc.cluster.local
    hadoop-hadoop-yarn-nm-2.hadoop-hadoop-yarn-nm.hadoop.svc.cluster.local
  yarn-hosts.excludes: |
---
# Source: hadoop/templates/hive/hive-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-hadoop-hive
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop-hive
data:
  hive-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
        <!-- 配置hdfs存储目录 -->
        <property>
                <name>hive.metastore.warehouse.dir</name>
                <value>/user/hive_remote/warehouse</value>
        </property>

        <property>
                <name>hive.metastore.local</name>
                <value>false</value>
        </property>

        <!-- 所连接的 MySQL 数据库的地址，hive_local是数据库，程序会自动创建，自定义就行 -->
        <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://192.168.110.125:30653/hive_metastore?createDatabaseIfNotExist=true&amp;useSSL=false&amp;serverTimezone=Asia/Shanghai</value>
        </property>

        <!-- MySQL 驱动 -->
        <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <!--<value>com.mysql.cj.jdbc.Driver</value>-->
                <value>com.mysql.jdbc.Driver</value>
        </property>

        <!-- mysql连接用户 -->
        <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>root</value>
        </property>

        <!-- mysql连接密码 -->
        <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>qwe123</value>
        </property>

        <!--元数据是否校验-->
        <property>
                <name>hive.metastore.schema.verification</name>
                <value>false</value>
        </property>

        <property>
                <name>system:user.name</name>
                <value>root</value>
                <description>user name</description>
        </property>

        <property>
                <name>hive.metastore.uris</name>
                <value>thrift://hadoop-hadoop-hive-metastore-0.hadoop-hadoop-hive-metastore:9083</value>
        </property>

        <!-- host -->
        <property>
                <name>hive.server2.thrift.bind.host</name>
                <value>0.0.0.0</value>
                <description>Bind host on which to run the HiveServer2 Thrift service.</description>
        </property>

        <!-- hs2端口 默认是10000-->
        <property>
                <name>hive.server2.thrift.port</name>
                <value>10000</value>
        </property>

        <property>
                <name>hive.server2.active.passive.ha.enable</name>
                <value>true</value>
        </property>

    </configuration>
---
# Source: hadoop/templates/hdfs/hdfs-dn-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hdfs-dn-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
spec:
  ports:
  - name: webhdfs
    port: 9864
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
---
# Source: hadoop/templates/hdfs/hdfs-dn-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hdfs-dn
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  ports:
  - name: webhdfs
    port: 9864
    nodePort: 30864
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
---
# Source: hadoop/templates/hdfs/hdfs-nn-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hdfs-nn-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  ports:
  - name: dfs
    port: 9000
    protocol: TCP
  - name: webhdfs
    port: 9870
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
---
# Source: hadoop/templates/hdfs/hdfs-nn-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hdfs-nn
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  ports:
  - name: dfs
    port: 9000
    protocol: TCP
    nodePort: 30900
  - name: webhdfs
    port: 9870
    nodePort: 30870
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
---
# Source: hadoop/templates/hive/hive-hiveserver2-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hive-hiveserver2-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-hiveserver2
spec:
  ports:
  - port: 10000
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-hiveserver2
---
# Source: hadoop/templates/hive/hive-hiveserver2-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hive-hiveserver2
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-hiveserver2
spec:
  ports:
  - port: 10000
    name: web
    nodePort: 30000
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-hiveserver2
---
# Source: hadoop/templates/hive/hive-metastore-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hive-metastore-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-metastore
spec:
  ports:
  - port: 9083
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-metastore
---
# Source: hadoop/templates/hive/hive-metastore-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hive-metastore
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-metastore
spec:
  ports:
  - port: 9083
    name: web
    nodePort: 31183
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-metastore
---
# Source: hadoop/templates/mr-historyserver/mr-historyserver-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-mr-historyserver-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: mr-historyserver
spec:
  ports:
  - port: 19888
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: mr-historyserver
---
# Source: hadoop/templates/mr-historyserver/mr-historyserver-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-mr-historyserver
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: mr-historyserver
spec:
  ports:
  - port: 19888
    name: web
    nodePort: 30888
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: mr-historyserver
---
# Source: hadoop/templates/yarn-proxyserver/yarn-proxyserver-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-proxyserver-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-proxyserver
spec:
  ports:
  - port: 9111
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-proxyserver
---
# Source: hadoop/templates/yarn-proxyserver/yarn-proxyserver-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-proxyserver
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-proxyserver
spec:
  ports:
  - port: 9111
    name: web
    nodePort: 30911
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-proxyserver
---
# Source: hadoop/templates/yarn/yarn-nm-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-nm-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  ports:
  - port: 8042
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
---
# Source: hadoop/templates/yarn/yarn-nm-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-nm
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  ports:
  - port: 8042
    name: web
    nodePort: 30042
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
---
# Source: hadoop/templates/yarn/yarn-rm-svc-headless.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-rm-headless
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  ports:
  - port: 8088
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
---
# Source: hadoop/templates/yarn/yarn-rm-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-rm
  namespace: hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  ports:
  - port: 8088
    name: web
    nodePort: 30088
  type: NodePort
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
---
# Source: hadoop/templates/hdfs/hdfs-dn-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-hdfs-dn
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-dn
  serviceName: hadoop-hadoop-hdfs-dn
  replicas: 3
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: hdfs-dn
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: hdfs-dn
      terminationGracePeriodSeconds: 0
      initContainers:
      - name: wait-for-nn
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        command: ['sh', '-c', "until curl -m 3 -sI http://hadoop-hadoop-hdfs-nn-0.hadoop-hadoop-hdfs-nn.hadoop.svc.cluster.local:9870 | egrep --silent 'HTTP/1.1 200 OK|HTTP/1.1 302 Found'; do echo waiting for nn; sleep 1; done"]
      containers:
      - name: hdfs-dn
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-hdfs-dn"]
        env:
          - name: POD_INDEX
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']
        resources:
          limits:
            cpu: 1000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          httpGet:
            path: /
            port: 9864
          initialDelaySeconds: 10
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /
            port: 9864
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        - name: dfs1
          mountPath: /opt/apache/hdfs/datanode1
        - name: dfs2
          mountPath: /opt/apache/hdfs/datanode2
        - name: dfs3
          mountPath: /opt/apache/hdfs/datanode3
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: dfs1
        hostPath:
          path: /opt/bigdata/servers/hadoop/dn/data/data1
          type: Directory
      - name: dfs2
        hostPath:
          path: /opt/bigdata/servers/hadoop/dn/data/data2
          type: Directory
      - name: dfs3
        hostPath:
          path: /opt/bigdata/servers/hadoop/dn/data/data3
          type: Directory
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
---
# Source: hadoop/templates/hdfs/hdfs-nn-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-hdfs-nn
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-nn
  serviceName: hadoop-hadoop-hdfs-nn
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: hdfs-nn
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: hdfs-nn
      terminationGracePeriodSeconds: 0
      containers:
      - name: hdfs-nn
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-hdfs-nn"]
        resources:
          limits:
            cpu: 1000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          httpGet:
            path: /
            port: 9870
          initialDelaySeconds: 10
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /
            port: 9870
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        - name: nn1
          mountPath: /opt/apache/hadoop-3.3.6/data/hdfs/namenode
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: nn1
        hostPath:
          path: /opt/bigdata/servers/hadoop/nn/data/data1
          type: Directory
---
# Source: hadoop/templates/hive/hive-hiveserver2-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-hive-hiveserver2
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-hiveserver2
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hive-hiveserver2
  serviceName: hadoop-hadoop-hive-hiveserver2
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: hive-hiveserver2
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: hive-hiveserver2
      terminationGracePeriodSeconds: 0
      initContainers:
      - name: wait-hive-metastore
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        command: ['sh', '-c', "until (echo 'q')|telnet -e 'q' hadoop-hadoop-hive-metastore.hadoop.svc.cluster.local 9083 >/dev/null 2>&1; do echo waiting for hive metastore; sleep 1; done"]
      containers:
      - name: hive-hiveserver2
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 10000
          name: web
        command: ["sh","-c","/opt/apache/bootstrap.sh hive-hiveserver2"]
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          tcpSocket:
            port: 10000
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          tcpSocket:
            port: 10000
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        - name: hive-site
          mountPath: /opt/apache/hive/conf/hive-site.xml
          subPath: hive-site.xml
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: hive-site
        configMap:
          name: hadoop-hadoop-hive
---
# Source: hadoop/templates/hive/hive-metastore-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-hive-metastore
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hive-metastore
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hive-metastore
  serviceName: hadoop-hadoop-hive-metastore
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: hive-metastore
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: hive-metastore
      terminationGracePeriodSeconds: 0
      initContainers:
      - name: wait-for-nn
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        command: ['sh', '-c', "until curl -m 3 -sI http://hadoop-hadoop-hdfs-nn-0.hadoop-hadoop-hdfs-nn.hadoop.svc.cluster.local:9870 | egrep --silent 'HTTP/1.1 200 OK|HTTP/1.1 302 Found'; do echo waiting for nn; sleep 1; done"]
      containers:
      - name: hive-metastore
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 9083
          name: web
        command: ["sh","-c","/opt/apache/bootstrap.sh hive-metastore"]
        resources:
          limits:
            cpu: 1000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          tcpSocket:
            port: 9083
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          tcpSocket:
            port: 9083
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        - name: hive-site
          mountPath: /opt/apache/hive/conf/hive-site.xml
          subPath: hive-site.xml
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: hive-site
        configMap:
          name: hadoop-hadoop-hive
---
# Source: hadoop/templates/mr-historyserver/mr-historyserver-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-mr-historyserver
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: mr-historyserver
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: mr-historyserver
  serviceName: hadoop-hadoop-mr-historyserver
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: mr-historyserver
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: mr-historyserver
      terminationGracePeriodSeconds: 0
      initContainers:
      - name: wait-for-rm
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        command: ['sh', '-c', "until curl -m 3 -sI http://hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.hadoop.svc.cluster.local:8088 | egrep --silent 'HTTP/1.1 200 OK|HTTP/1.1 302 Found'; do echo waiting for rm; sleep 1; done"]
      containers:
      - name: mr-historyserver
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 19888
          name: web
        command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-mr-historyserver"]
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          httpGet:
            path: /ws/v1/history/info
            port: 19888
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ws/v1/history/info
            port: 19888
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
---
# Source: hadoop/templates/yarn-proxyserver/yarn-proxyserver-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-yarn-proxyserver
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-proxyserver
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-proxyserver
  serviceName: hadoop-hadoop-yarn-proxyserver
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: yarn-proxyserver
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: yarn-proxyserver
      terminationGracePeriodSeconds: 0
      initContainers:
      - name: wait-for-rm
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        command: ['sh', '-c', "until curl -m 3 -sI http://hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.hadoop.svc.cluster.local:8088 | egrep --silent 'HTTP/1.1 200 OK|HTTP/1.1 302 Found'; do echo waiting for rm; sleep 1; done"]
      containers:
      - name: yarn-proxyserver
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 9111
          name: web
        command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-proxyserver"]
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          tcpSocket:
            port: 9111
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          tcpSocket:
            port: 9111
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
---
# Source: hadoop/templates/yarn/yarn-nm-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-yarn-nm
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-nm
  serviceName: hadoop-hadoop-yarn-nm
  replicas: 3
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: yarn-nm
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: yarn-nm
      terminationGracePeriodSeconds: 0
      initContainers:
      - name: wait-for-rm
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        command: ['sh', '-c', "until curl -m 3 -sI http://hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.hadoop.svc.cluster.local:8088 | egrep --silent 'HTTP/1.1 200 OK|HTTP/1.1 302 Found'; do echo waiting for rm; sleep 1; done"]
      containers:
      - name: yarn-nm
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 8042
          name: web
        command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-nm hadoop-hadoop-yarn-rm.hadoop.svc.cluster.local 8088"]
        resources:
          limits:
            cpu: 1000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          httpGet:
            path: /node
            port: 8042
          initialDelaySeconds: 10
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /node
            port: 8042
          initialDelaySeconds: 10
          timeoutSeconds: 2
        env:
        - name: MY_CPU_LIMIT
          valueFrom:
            resourceFieldRef:
              containerName: yarn-nm
              resource: limits.cpu
              divisor: 1
        - name: MY_MEM_LIMIT
          valueFrom:
            resourceFieldRef:
              containerName: yarn-nm
              resource: limits.memory
              divisor: 1M
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop
---
# Source: hadoop/templates/yarn/yarn-rm-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-yarn-rm
  namespace: hadoop
  annotations:
    checksum/config: 36697e54c58cb45f147f354343692c9d77539232b308ec5f728d9c5e361acd54
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-rm
  serviceName: hadoop-hadoop-yarn-rm
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: yarn-rm
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: yarn-rm
      terminationGracePeriodSeconds: 0
      containers:
      - name: yarn-rm
        image: "harbor.yourdomain.com:8443/hadoop/hadoop_hive:arm"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 8088
          name: web
        command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-rm"]
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        readinessProbe:
          httpGet:
            path: /ws/v1/cluster/info
            port: 8088
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ws/v1/cluster/info
            port: 8088
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: core-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/core-site.xml
          subPath: core-site.xml
        - name: hdfs-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
          subPath: hdfs-site.xml
        - name: mapred-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/mapred-site.xml
          subPath: mapred-site.xml
        - name: yarn-site
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-site.xml
          subPath: yarn-site.xml
        - name: dfs-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.includes
          subPath: dfs-hosts.includes
        - name: dfs-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/dfs-hosts.excludes
          subPath: dfs-hosts.excludes
        - name: yarn-hosts-includes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.includes
          subPath: yarn-hosts.includes
        - name: yarn-hosts-excludes
          mountPath: /opt/apache/hadoop-3.3.6/etc/hadoop/yarn-hosts.excludes
          subPath: yarn-hosts.excludes
        securityContext:
          runAsUser: 10000
          privileged: true
      volumes:
      - name: core-site
        configMap:
          name: hadoop-hadoop
      - name: hdfs-site
        configMap:
          name: hadoop-hadoop
      - name: mapred-site
        configMap:
          name: hadoop-hadoop
      - name: yarn-site
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: dfs-hosts-excludes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-includes
        configMap:
          name: hadoop-hadoop
      - name: yarn-hosts-excludes
        configMap:
          name: hadoop-hadoop

部署后查看nn 节点端口， hadoop 部署成功
在这里插入图片描述

对于大文件需要多线程的方式分块上传再合并为一个文件，已经投产效率不错

def upload_file_to_hdfs(file):
    host, port = HDFS_HOST, HDFS_PORT
    path = '/upload'
    script_directory = os.path.dirname(os.path.realpath(__file__))
    local_file_path = os.path.join(script_directory, '../file_upload', file.filename)
    file.save(local_file_path)

    hdfs_file_path = os.path.join(path, file.filename)
    client = InsecureClient(url=f'http://{host}:{int(port)}', user='hadoop')

    file_size = os.path.getsize(local_file_path)
    chunk_size = 5 * 1024 * 1024  # 每块 5MB
    num_chunks = (file_size + chunk_size - 1) // chunk_size  # 计算分块数量
    futures = []

    with concurrent.futures.ThreadPoolExecutor() as executor:
        for offset in range(0, file_size, chunk_size):
            futures.append(
                executor.submit(upload_hdfs_chunk, client, hdfs_file_path, local_file_path, offset, chunk_size))

        concurrent.futures.wait(futures)

    # 合并上传的分块
    merge_hdfs_chunks(client, hdfs_file_path, num_chunks)

    return hdfs_file_path


def merge_hdfs_chunks(client, hdfs_file_path, num_chunks):
    # 创建一个新的文件并写入
    with client.write(hdfs_file_path, overwrite=True) as writer:
        for i in range(num_chunks):
            part_file_path = f"{hdfs_file_path}.part{i}"
            try:
                with client.read(part_file_path) as reader:
                    writer.write(reader.read())
                # 删除已合并的分块文件
                client.delete(part_file_path)
            except Exception as e:
                print(f"Error processing {part_file_path}: {e}")

    print(f'Merged file created at: {hdfs_file_path}')

小结

成功实现了国产arm kylin的hadoop搭建和使用python 多线程分块方式上传文件到hdfs

纯手动搭建大数据集群架构_记录021_麒麟系统v10版_x86/64架构安装hadoop集群_修改SSH端口号_调整脚本端口---大数据之Hadoop3.x工作笔记0181
添柴程序猿的专栏
05-31 402
如果是arm架构的系统,就需要安装arm架构的java,jdk,对应的hadoop应该也需要arm架构的,这个。hadoop集群,那么这时候,还是有些地方需要修改的,大部分,跟之前使用centos,去安装。hadoop集群没有区别,直接参考以前的博文就可以了,并且,有区别的地方,已经在。但是这一次,安装的时候,ssh端口不是默认的22,给改成了22022了,这个时候。首先,如果想要修改ssh的端口,可以看到按照上面这样修改就可以了,然后。还没有尝试,尝试以后,给大家说.对应的博文中,记录了.
Kylin启动失败||启动成功但Web界面显示404
CSDN2022博客之星Top39；华为云博主7&，CSDN、稀土掘金、微信公众号、阿里云开发者社区、腾讯云社区、博客园、知乎、51CTO同名博主WHYBIGDATA；
09-14 1010
Kylin启动失败||启动成功但Web界面显示404
基于Hadoop架构的国产化分布式集群平台.pdf
08-09
#资源达人分享计划#
CDH/HDP国产替换CMP（CDH大数据平台国产替换的要求和原因是什么？）
最新发布
yumgpkpm的专栏
03-16 433
CDH（Cloudera Distribution including Apache Hadoop）大数据平台的国产替换需求源于多重因素，其核心要求和原因可归纳如下：一、国产替换的核心原因二、国产替换的核心要求三、国产替代方案示例总结CDH 的国产替换是技术自主可控、安全合规、成本优化的必然选择。国产平台通过技术创新、生态适配和本地化服务，为企业提供了可靠的替代方案，助力数字化转型与国家信创战略落地。
国内Hadoop相关的开源项目
weixin_30485291的博客
12-27 362
1、BC-Hadoop：中国移动Hadoop工具链打包 https://github.com/cmri/bc-hadoop2.0 孵化阶段，将成为一个通用的开源Hadoop平台 2、BC-BSP：中国移动图计算平台 https://github.com/cmri/BC-BSP 孵化阶段，将成为通用的开源BSP平台 3、小米Minos：部署和管理Hadoop生态链工具 Mi...
龙芯平台下安装配置hadoop
weixin_30708329的博客
09-04 334
一、 hadoop简介 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。 用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进行高速运算和存储。 Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的（low-cost）硬...
Ubuntu Kylin16.04 安装Hadoop2.7.4 伪分布式
lucyLee的博客
11-12 733
最近，因为课程设计需要在Ubuntu上安装Hadoop。本人使用的是Ubuntu kylin 16.04，安装Hadoop2.7.4。安装过程都是参考别人的博客，自己一步一步摸索过来的。如果您在安装的过程中遇到了什么问题，请多自己查阅网上资料，毕竟每个人会遇到的问题可能是不一样的。一、安装JDKHadoop 2.7及更新版本需要JAVA 7支持，本人安装的是Oracle jdk1.8_151为例。
银河麒麟安装—jdk和hadoop
m0_67403188的博客
04-12 4114
一、jdk安装 下载jdk，官网下载需要注册账号。 oracle官网下载链接 我下载的是 jdk1.8.0_241。 因为我没有安装xftp这类软件，无法直接向linux上传下载好的jdk安装包，所以我把下载好的 jdk-8u241-linux-x64.tar.gz 直接放在共享文件夹（/mnt/hgfs/），这样linux就可以用了。 解压 #创建Java文件夹，便于管理 root@Kylin:mkdir /opt/software/Java #解压到指定目录 root@Kylin:/mnt/
Kylin实践（一）--Hadoop环境搭建
isscici06的专栏
06-11 3024
做了半年kylin，基本停留在操作Web UI的层面，当然，期间也遇到一些坑，但是大多是社区好心人帮忙解决的，尤其跟环境相关的，一出问题，很长时间都解决不了，现在项目的kylin服务已经完成并且对外提供服务了，有了一些基础，想更深入一点学习使用kylin，学习需要理论结合实践，生产环境肯定不能随便动了，那就自己动手搭一个环境吧。三台Centos7的虚拟机，内存8G，8核CUP，200G硬盘。10....
《Linux运维总结：基于银河麒麟V10+ARM64架构CPU部署zookeeper 3.8.4二进制分布式集群》
东城绝神
07-19 1161
《Linux运维总结：基于Ubuntu 22.04+ARM64架构CPU部署zookeeper 3.8.4二进制分布式集群》
Kylin学习笔记（一）安装指南
勿忘始终
08-15 1066
Kylin学习笔记（一）安装指南 标签（空格分隔）： Kylin 一、软件要求 Hadoop: 2.7+ Hive: 0.13 - 1.2.1+ HBase: 1.1+ Spark (可选) 2.1.1+ Kafka (可选) 0.10.0+ JDK: 1.7+ OS: Linux only, CentOS 6.5+ or Ubuntu 16.0.4+ 二、硬件要求 ...
Kylin 集群搭建
金玉良缘
04-20 1847
1.环境说明 1.1 版本选择 1.2Kylin 安装与配置 1.2.1 软件下载 下载地址：https://archive.apache.org/dist/kylin/apache-kylin-2.3.2/ 1.2.2 解压 tar -zxvf apache-kylin-2.3.2-bin-cdh57.tar.gz 1.2.3 建立软连接 ln -s apache-kylin-2.3.2-...
大数据不等于Hadoop 国内Hadoop生态系统亟待完善
IT时代周刊
05-12 454
近期发布的《中国Hadoop MapReduce生态系统分析》报告指出，在中国，Hadoop 应用正在从互联网企业，逐渐拓展到电信，金融，政府，医疗这些传统行业。虽然目前Hadoop应用场景还是以日志存储、查询和非结构化数据处理为主，但是Hadoop技术的不断成熟以及生态系统相关产品的完善，包括Hadoop对SQL不断加强的支持，以及主流商业软件厂商对Hadoop支持的不断增强，会带动Hadoop...
kylin 集群部署
qq_36079077的博客
05-18 548
1. 集群节点规划与说明 rzx1 all rzx2 query rzx3 query 说明： Kylin节点角色有三种： all： 包含query和job query： 查询节点 job： 工作节点 3. Kylin依赖的其他大数据组件非常多，下列列表是安装kylin需要的组件 JDK 1.8<必须项> HADOOP<必须项，hdfs作为数据存储基础，这里版本是hadoop-2.7.7> ZOOKEERER<必须项，集群协调，这里版本zookeeper-3.4.1
虚拟机银河麒麟（Kylin）桌面操作系统搭建Kubernetes集群
XXisCC的博客
08-03 2645
三台虚拟机上安装银河麒麟V10操作系统，取得root权限注意：Centos7使用yum，而Kylin使用apt记得关闭应用保护，总共3项，不然在以后运行应用的时候会反复弹窗。
Kylin集群安装部署
忄凝^的博客
08-18 669
这里目录标题3、`Kylin`集群安装部署3.1. 下载上传3.1.2 修改 hive 配置文件3.1.3 修改 hbase 配置文件3.1.4 配置 kylin 相关环境变量3.1.5 修改配置文件3.1.6 拷贝kylin到其他节点3.1.7 启动3.1.8 关机拍照 3、Kylin集群安装部署 安装前提 OS: CentOS 7 java 1.8 zookeeper 3.4.5 https://blog.csdn.net/weixin_43660536/article/details/1183
麒麟操作系统-03-Hadoop3.5.5安装与部署
Jason1511的博客
06-01 1157
虚拟机也可以，但是要确保运行内存是够用的，建议主节点node1给4G，其他最少2G。注意修改ip们可以参照我的另一篇博文：http://t.csdn.cn/6ohFp。下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.5/注意：以上的配置都是些简单配置，很多参数都是可以根据情况去修改的，包括端口号。清华源下载的快，点进去选择 hadoop-3.3.5.tar.gz 下载，674M的，不要选择错了。
三、kylin读写分离集群部署
zxf126126的博客
04-30 1049
1、部署图 注意： 1、在kylin服务机器上hbase shell 作为存储Hbase集群的客户端，配置需要存储集群的客户端配置，cdh 的hbase配置路径在/etc/hbase/conf下，hbase-site.xml 中不能包含hdfs的信息，不然计算集群会使用Hbase的hdfs配置，只包含zk的信息即可 2、hadoop 的配置包含计算集群的所有配置，需要在hdfs-s...
纯手动搭建大数据集群架构_记录022_麒麟系统v10_ARM架构集群搭建01_ARM架构jdk安装_集群架构_NameNode is stilling---大数据之Hadoop3.x工作笔记0182
添柴程序猿的专栏
06-12 272
最近客户给了个集群,三台机器,但是,却是ARM架构的,第一次安装arm架构的机器,对于软件,有一定的要求,这个过程中,主要是.我打算把,hadoop,每个机器,安装到对应的文件夹中去.那么这个时候,配置hadoop的etc/hadoop文件夹中的。这个位置,其实就是安装的openjdk. 这里要注意openjdk是有问题的, 他默认安装上了,但是没有清晰的指定。并且要指出的是,这三台机器,自身的硬盘都很小,只有一块9.8T的盘,但是这三台机器都挂到这个盘上面去了。
Hadoop集群搭建及生态系统集成优化指南
通过上述文件列表的分析，可以得知该压缩包中包含的文档和指南，将详细阐述Hadoop集群搭建过程中的各个步骤，以及如何通过集成ZooKeeper、HBase、Hive、Kylin来扩展Hadoop的处理能力，并通过集成Kerberos来保证集群...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

CHEN_RUI_2200

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值