离线数据仓库从0到1-阶段二软件安装

已于 2022-12-08 16:52:58 修改
阅读量299 收藏
点赞数
分类专栏： 数据仓库 文章标签： linux 运维 服务器
于 2022-07-25 15:22:42 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/S1124654/article/details/125967476
版权

安装计划图

jdk安装

解压到自定的目录

tar -zxvf jdk-8u161-linux-x64.tar.gz -C ../software/

配置环境变量

 vi /etc/profile.d/my_env.conf
#JAVA_HOME
export JAVA_HOME=/home/bigdata/software/jdk1.8.0_161
export PATH=$JAVA_HOME/bin:$PATH

 同步配置文件

sudo ./xsync /etc/profile.d/my_env.conf

同步jdk 

./xsync ../software/jdk1.8.0_161/
source /etc/profile.d/my_env.conf

输入java

zk集群安装

tar -zxvf apache-zookeeper-3.5.7-bin.tar.gz

修改配置文件

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/home/bigdata/software/zookeeper-3.5.7/zkData
#######################cluster##########################
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

对应存储数据的地方创建一个myid 对应上面的server.x

配置环境变量

#ZK_HOME
export ZK_HOME=/home/bigdata/software/zookeeper-3.5.7
export PATH=$ZK_HOME/bin:$PATH

source /etc/profile.d/my_env.conf

启停脚本

#!/bin/bash

                #检验参数
                if [ $# -lt 1 ]
                then
                 echo '参数不能为空！！！'
                 exit
                fi

                #循环遍历每一台机器，分别启动或者停止ZK服务

                for host in node1 node2 node3
                do
                  case $1 in
                   "start")
                        echo "*****************$1 $host zookeeper****************"
                                ssh $host /home/bigdata/software/zookeeper-3.5.7/bin/zkServer.sh $1
                   ;;

                   "stop")
                        echo "*****************$1 $host zookeeper****************"
                                ssh $host /home/bigdata/software/zookeeper-3.5.7/bin/zkServer.sh $1
                   ;;

                   "status")
                        echo "*****************$1 $host zookeeper****************"
                                ssh $host /home/bigdata/software/zookeeper-3.5.7/bin/zkServer.sh $1
                   ;;

                   *)
                           echo '参数有误！！！'
                           exit
                   ;;
                  esac
                done

 

hadoopHA安装

core-site.xml

<configuration>
    <property>
        <!--指定 namenode 的 hdfs 协议文件系统的通信地址-->
        <name>fs.defaultFS</name>
		<!--指定hdfs高可用的集群名称-->
        <value>hdfs://bigdatacluster</value>
    </property>
    <property>
        <!--指定 hadoop 集群存储临时文件的目录-->
        <name>hadoop.tmp.dir</name>
        <value>/home/bigdata/module/hadoop-3.1.3/data</value>
    </property>
	
	<!-- 配置HDFS网页登录使用的静态用户为bigdata -->
  <property>
    <name>hadoop.http.staticuser.user</name>
    <value>bigdata</value>
  </property>
 
 <!-- 回收站 -->
    <property>
        <name>fs.trash.interval</name>
        <value>1</value>
    </property>
 
    <property>
        <name>fs.trash.checkpoint.interval</name>
        <value>1</value>
    </property>
 
    <!-- 配置该bigdata(superUser)允许通过代理访问的主机节点 -->
    <property>
        <name>hadoop.proxyuser.bigdata.hosts</name>
        <value>*</value>
    </property>
    <!-- 配置该bigdata(superUser)允许通过代理用户所属组 -->
    <property>
        <name>hadoop.proxyuser.bigdata.groups</name>
        <value>*</value>
    </property>
    <!-- 配置该bigdata(superUser)允许通过代理的用户-->
    <property>
        <name>hadoop.proxyuser.bigdata.users</name>
        <value>*</value>
    </property>
 
 <!-- 指定zkfc要连接的zkServer地址 -->
<property>
	<name>ha.zookeeper.quorum</name>
	<value>node1:2181,node2:2181,node3:2181</value>
</property>
</configuration>

hdfs-site.xml

<configuration>
<!-- NameNode数据存储目录 -->
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file://${hadoop.tmp.dir}/name</value>
  </property>
<!-- DataNode数据存储目录 -->
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file://${hadoop.tmp.dir}/data</value>
  </property>
<!-- JournalNode数据存储目录 -->
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>${hadoop.tmp.dir}/jn</value>
  </property>
<!-- 完全分布式集群名称 对应core.xml里面的fs.defaultFS-->
  <property>
    <name>dfs.nameservices</name>
    <value>bigdatacluster</value>
  </property>
<!-- 集群中NameNode节点都有哪些 -->
  <property>
    <name>dfs.ha.namenodes.bigdatacluster</name>
    <value>nn1,nn2</value>
  </property>
<!-- NameNode的RPC通信地址 -->
  <property>
    <name>dfs.namenode.rpc-address.bigdatacluster.nn1</name>
    <value>master1:8020</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.bigdatacluster.nn2</name>
    <value>master2:8020</value>
  </property>
<!-- NameNode的http通信地址 -->
  <property>
    <name>dfs.namenode.http-address.bigdatacluster.nn1</name>
    <value>master1:9870</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.bigdatacluster.nn2</name>
    <value>master2:9870</value>
  </property>
<!-- 指定NameNode元数据在JournalNode上的存放位置 -->
  <property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node1:8485;node2:8485;node3:8485/bigdatacluster</value>
  </property>
<!-- 访问代理类：client用于确定哪个NameNode为Active -->
  <property>
    <name>dfs.client.failover.proxy.provider.bigdatacluster</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
<!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 -->
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>sshfence</value>
  </property>
<!-- 使用隔离机制时需要ssh秘钥登录-->
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/home/bigdata/.ssh/id_rsa</value>
  </property>
 
  <!-- 配置黑名单 -->
     <property>
        <name>dfs.hosts.exclude</name>
        <value>/home/bigdata/module/hadoop-3.1.3/etc/blacklist</value>
    </property>
	
	<!-- 启用nn故障自动转移 -->
<property>
	<name>dfs.ha.automatic-failover.enabled</name>
	<value>true</value>
</property>
</configuration>

yarn-site.xml

<configuration>
 
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
 
    <!-- 启用resourcemanager ha -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
 
    <!-- 声明两台resourcemanager的地址 -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>cluster-yarn1</value>
    </property>
    <!--指定resourcemanager的逻辑列表-->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
</property>
<!-- ========== rm1的配置 ========== -->
<!-- 指定rm1的主机名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>master1</value>
</property>
<!-- 指定rm1的web端地址 -->
<property>
     <name>yarn.resourcemanager.webapp.address.rm1</name>
     <value>master1:8088</value>
</property>
<!-- 指定rm1的内部通信地址 -->
<property>
     <name>yarn.resourcemanager.address.rm1</name>
     <value>master1:8032</value>
</property>
<!-- 指定AM向rm1申请资源的地址 -->
<property>
     <name>yarn.resourcemanager.scheduler.address.rm1</name>  
     <value>master1:8030</value>
</property>
<!-- 指定供NM连接的地址 -->  
<property>
     <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
     <value>master1:8031</value>
</property>
<!-- ========== rm2的配置 ========== -->
    <!-- 指定rm2的主机名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>master2</value>
</property>
<property>
     <name>yarn.resourcemanager.webapp.address.rm2</name>
     <value>master2:8088</value>
</property>
<property>
     <name>yarn.resourcemanager.address.rm2</name>
     <value>master2:8032</value>
</property>
<property>
     <name>yarn.resourcemanager.scheduler.address.rm2</name>
     <value>master2:8030</value>
</property>
<property>
     <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
     <value>master2:8031</value>
</property>
 
    <!-- 指定zookeeper集群的地址 --> 
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>node1:2181,node2:2181,node3:2181</value>
    </property>
 
    <!-- 启用自动恢复 --> 
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>
 
    <!-- 指定resourcemanager的状态信息存储在zookeeper集群 --> 
    <property>
        <name>yarn.resourcemanager.store.class</name>     
		<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<!-- 环境变量的继承 -->
 <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
 
	 <!-- 开启日志聚集功能 -->
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <!-- 设置日志聚集服务器地址 -->
 <!-- 设置日志聚集服务器地址 -->
  <property>  
    <name>yarn.log.server.url</name>  
    <value>http://master1:19888/jobhistory/logs</value>
  </property>
  <!-- 设置日志保留时间为7天 -->
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>
 
    <!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
 
    <!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

    <property>
      <name>yarn.nodemanager.resource.memory-mb</name>
      <value>24576</value>
    </property>
 
</configuration>

mapred-site.xml

<configuration>
<!-- 启用jvm重用 -->
<property>
    <name>mapreduce.job.jvm.numtasks</name>
    <value>10</value>
    <description>How many tasks to run per jvm,if set to -1 ,there is  no limit</description>
</property>
    
<property>
        <!--指定 mapreduce 作业运行在 yarn 上-->
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

	
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=/home/bigdata/module/hadoop-3.1.3</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=/home/bigdata/module/hadoop-3.1.3</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=/home/bigdata/module/hadoop-3.1.3</value>
</property>
 
<!-- 历史服务器端地址 -->
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>master1:10020</value>
  </property>
  <!-- 历史服务器web端地址 -->
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master1:19888</value>
  </property>
 
 
</configuration>

workers

192.168.0.217
192.168.0.218
192.168.0.219

capacity-scheduler.xml 

<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.3</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare 
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare 
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>high,low</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>
<!--
  队列占比
-->
  <property>
    <name>yarn.scheduler.capacity.root.high.capacity</name>
    <value>70</value>
    <description>Default queue target capacity.</description>
  </property>

    <property>
    <name>yarn.scheduler.capacity.root.low.capacity</name>
    <value>30</value>
    <description>Default queue target capacity.</description>
  </property>

  
<!--
  百分比
-->
  <property>
    <name>yarn.scheduler.capacity.root.high.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

    <property>
    <name>yarn.scheduler.capacity.root.low.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>


<!--
  运行状态
-->
  <property>
    <name>yarn.scheduler.capacity.root.high.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.low.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

<!--
  权限
-->
  <property>
    <name>yarn.scheduler.capacity.root.high.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

    <property>
    <name>yarn.scheduler.capacity.root.low.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>


<!--
  权限
-->
  <property>
    <name>yarn.scheduler.capacity.root.high.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

    <property>
    <name>yarn.scheduler.capacity.root.low.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

<!--
  权限
-->
  <property>
    <name>yarn.scheduler.capacity.root.high.acl_application_max_priority</name>
    <value>*</value>
    <description>
      The ACL of who can submit applications with configured priority.
      For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
    </description>
  </property>

    <property>
    <name>yarn.scheduler.capacity.root.low.acl_application_max_priority</name>
    <value>*</value>
    <description>
      The ACL of who can submit applications with configured priority.
      For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
    </description>
  </property>

<!--
  权限
-->
   <property>
     <name>yarn.scheduler.capacity.root.high.maximum-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Maximum lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        This will be a hard time limit for all applications in this
        queue. If positive value is configured then any application submitted
        to this queue will be killed after exceeds the configured lifetime.
        User can also specify lifetime per application basis in
        application submission context. But user lifetime will be
        overridden if it exceeds queue maximum lifetime. It is point-in-time
        configuration.
        Note : Configuring too low value will result in killing application
        sooner. This feature is applicable only for leaf queue.
     </description>
   </property>

      <property>
     <name>yarn.scheduler.capacity.root.low.maximum-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Maximum lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        This will be a hard time limit for all applications in this
        queue. If positive value is configured then any application submitted
        to this queue will be killed after exceeds the configured lifetime.
        User can also specify lifetime per application basis in
        application submission context. But user lifetime will be
        overridden if it exceeds queue maximum lifetime. It is point-in-time
        configuration.
        Note : Configuring too low value will result in killing application
        sooner. This feature is applicable only for leaf queue.
     </description>
   </property>


<!--
  生命周期
-->
   <property>
     <name>yarn.scheduler.capacity.root.high.default-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Default lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        If the user has not submitted application with lifetime value then this
        value will be taken. It is point-in-time configuration.
        Note : Default lifetime can't exceed maximum lifetime. This feature is
        applicable only for leaf queue.
     </description>
   </property>
<property>
        <name>yarn.scheduler.capacity.root.low.default-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Default lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        If the user has not submitted application with lifetime value then this
        value will be taken. It is point-in-time configuration.
        Note : Default lifetime can't exceed maximum lifetime. This feature is
        applicable only for leaf queue.
     </description>
   </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers.
      When setting this parameter, the size of the cluster should be taken into account.
      We use 40 as the default value, which is approximately the number of nodes in one rack.
      Note, if this value is -1, the locality constraint in the container request
      will be ignored, which disables the delay scheduling.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.rack-locality-additional-delay</name>
    <value>-1</value>
    <description>
      Number of additional missed scheduling opportunities over the node-locality-delay
      ones, after which the CapacityScheduler attempts to schedule off-switch containers,
      instead of rack-local ones.
      Example: with node-locality-delay=40 and rack-locality-delay=20, the scheduler will
      attempt rack-local assignments after 40 missed opportunities, and off-switch assignments
      after 40+20=60 missed opportunities.
      When setting this parameter, the size of the cluster should be taken into account.
      We use -1 as the default value, which disables this feature. In this case, the number
      of missed opportunities for assigning off-switch containers is calculated based on
      the number of containers and unique locations specified in the resource request,
      as well as the size of the cluster.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to assign jobs to queues
      The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
      Typically this list will be used to map users to queues,
      for example, u:%user:%user maps all users to queues with the same name
      as the user.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name>
    <value>1</value>
    <description>
      Controls the number of OFF_SWITCH assignments allowed
      during a node's heartbeat. Increasing this value can improve
      scheduling rate for OFF_SWITCH containers. Lower values reduce
      "clumping" of applications on particular nodes. The default is 1.
      Legal values are 1-MAX_INT. This config is refreshable.
    </description>
  </property>


  <property>
    <name>yarn.scheduler.capacity.application.fail-fast</name>
    <value>false</value>
    <description>
      Whether RM should fail during recovery if previous applications'
      queue is no longer valid.
    </description>
  </property>

</configuration>

架构如图

启动注意事项

先关闭防火墙

sudo systemctl status firewalld
setenforce 0
sudo vi /etc/selinux/config
SELINUXTYPE=disabled

#启动Zookeeper以后，然后再初始化HA在Zookeeper中状态(在对应的namenode上面其中一台格式化就行，最好这种情况是master1主节点)：

hdfs zkfc -formatZK

#启动journalnode他是8485端口(在datanode的所有机器上面启动)

hdfs --daemon start journalnode

#然后格式化（namenode：主节点格式化，这里也就是master1）

hdfs namenode -format

master1启动namenode (master2先不用启动)

 hdfs --daemon start namenode

master2然后同步namenode数据

hdfs namenode -bootstrapStandby 

运行集群的时候注意：如果有JAVA_HOME找不到，那么就把hadoop里面的bin和sbin里面的执行脚本都加一个JAVA_HOME变量，比如：

JAVA_HOME="/home/bigdata/module/jdk1.8.0_161"加入到脚本靠前的位置就行了。

启动集群脚本

#!/bin/bash
if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit ;
fi
case $1 in
"start")
        echo " =================== 启动 hadoop集群 ==================="
		echo "node1的journalnode启动"
        ssh node1 "hdfs --daemon start journalnode"
        echo "node2的journalnode启动"
        ssh node2 "hdfs --daemon start journalnode"
		echo "node3的journalnode启动"
        ssh node3 "hdfs --daemon start journalnode"
		
		
        echo " --------------- 启动 hdfs ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/sbin/start-dfs.sh"
        echo " --------------- 启动 yarn ---------------"
        ssh master2 "/home/bigdata/software/hadoop-3.1.3/sbin/start-yarn.sh"
		
        echo " --------------- 启动 historyserver ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/bin/mapred --daemon start historyserver"  
;;
"stop")
        echo " =================== 关闭 hadoop集群 ==================="
        echo " --------------- 关闭 historyserver ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
		
        echo " --------------- 关闭 yarn ---------------"
        ssh master2 "/home/bigdata/software/hadoop-3.1.3/sbin/stop-yarn.sh"
        echo " --------------- 关闭 hdfs ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/sbin/stop-dfs.sh"
		
		echo "node1的journalnode关闭"
        ssh node1 "hdfs --daemon stop journalnode"
        echo "node2的journalnode关闭"
        ssh node2 "hdfs --daemon stop journalnode"
		echo "node3的journalnode关闭"
        ssh node3 "hdfs --daemon stop journalnode"
;;
*)
    echo "Input Args Error..."
;;
esac

测试集群

hadoop dfs -mkdir /data

hadoop jar /home/bigdata/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount  -Dmapreduce.job.queuename=yarn队列名 /datadata /output

测试resourcemanger高可用是否可用你

yarn-daemon.sh start resourcemanager
yarn-daemon.sh stop resourcemanager

测试namenode高可用是否可用

hdfs --daemon start namenode
hdfs --daemon stop namenode

kafka安装

集群规划

集群部署 

1）解压安装包

tar -zxvf kafka_2.11-2.4.1.tgz -C /home/bigdata/software/

2）修改解压后的文件名称 （可改可不改，注意一下后面配置即可 我没改）

mv kafka_2.11-2.4.1/ kafka

3）在/opt/module/kafka目录下创建logs文件夹

mkdir logs

4）修改配置文件

cd config/
vim server.properties

修改或者增加以下内容：
#broker的全局唯一编号，不能重复  这个非常重要
broker.id=0
#删除topic功能使能
delete.topic.enable=true
#kafka运行日志存放的路径
log.dirs=/home/bigdata/software/kafka_2.11-2.4.1/data
#配置连接Zookeeper集群地址
zookeeper.connect=node1:2181,node2:2181,node3:2181/kafka

5）配置环境变量

 sudo vim /etc/profile.d/my_env.sh

#KAFKA_HOME
export KAFKA_HOME=/home/bigdata/software/kafka_2.11-2.4.1/
export PATH=$PATH:$KAFKA_HOME/bin

环境变量改完之后source一下，让它生效

source /etc/profile.d/my_env.sh

6）分发安装包

xsync kafka/

注意：分发之后记得配置其他机器的环境变量

7）分别在其它节点上修改配置文件/home/bigdata/software/kafka_2.11-2.4.1/config/server.properties中的broker.id=1、broker.id=2

注：broker.id不得重复

8）启动集群

依次在node1/node2/node3节点上启动kafka

bin/kafka-server-start.sh -daemon /home/bigdata/software/kafka_2.11-2.4.1/config/server.properties

bin/kafka-server-start.sh -daemon /home/bigdata/software/kafka_2.11-2.4.1/config/server.properties

bin/kafka-server-start.sh -daemon /home/bigdata/software/kafka_2.11-2.4.1/server.properties

9）关闭集群

bin/kafka-server-stop.sh

bin/kafka-server-stop.sh

bin/kafka-server-stop.sh

kafka群起脚本

#!/bin/bash
 
case $1 in
"start"){
    for i in hadoop102 hadoop103 hadoop104
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/opt/module/kafka/kafka_2.12-3.0.0/bin/kafka-server-start.sh -daemon /opt/module/kafka/kafka_2.12-3.0.0/config/server.properties "
    done
};;
"stop"){
    for i in hadoop102 hadoop103 hadoop104
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/opt/module/kafka/kafka_2.12-3.0.0/bin/kafka-server-stop.sh stop"
    done
};;
esac

flume 3.0安装

安装地址

（1） Flume官网地址：Welcome to Apache Flume — Apache Flume

（2）文档查看地址：Flume 1.10.0 User Guide — Apache Flume

（3）下载地址：http://archive.apache.org/dist/flume/

安装部署

（1）将apache-flume-1.9.0-bin.tar.gz上传到linux的/home/bigdata/package目录下

（2）解压apache-flume-1.9.0-bin.tar.gz到/home/bigdata/software/目录下

[bigdata@master1 flume-1.9.0]$ tar -zxvf /home/bigdata/package /apache-flume-1.9.0-bin.tar.gz -C /home/bigdata/software/

（3）修改apache-flume-1.9.0-bin的名称为flume

[bigdata@master1 lib]$ mv apache-flume-1.9.0-bin/ flume-1.9.0

（4）将lib文件夹下的guava-11.0.2.jar删除以兼容Hadoop 3.1.3

[bigdata@master1 flume-1.9.0]$ rm /home/bigdata/software/flume-1.9.0/lib/guava-11.0.2.jar 

注意：删除guava-11.0.2.jar的服务器节点，一定要配置hadoop环境变量。否则会报如下异常。

Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Lists

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 1 more

（5）修改conf目录下的log4j.properties配置文件，配置日志文件路径

[bigdata@master1 conf]$ vim log4j.properties

flume.log.dir=/home/bigdata/software/flume-1.9.0/logs

将flume分发到master2上

rsync -av /home/bigdata/software/flume-1.9.0 root@master2:/home/bigdata/software

项目经验

1）堆内存调整

Flume堆内存通常设置为4G或更高，配置方式如下：

修改/opt/module/flume/conf/flume-env.sh文件，配置如下参数

export JAVA_OPTS="-Xms4096m -Xmx4096m -Dcom.sun.management.jmxremote"

注：

-Xms表示JVM Heap(堆内存)最小尺寸，初始分配；

-Xmx 表示JVM Heap(堆内存)最大允许的尺寸，按需分配。

2）方便起见，此处编写一个日志采集Flume进程的启停脚本

在master1节点的/home/bigdata/bin目录下创建脚本f1.sh

[bigdata@master1 bin]$ vim f1.sh

在脚本中填写如下内容

脚本

#!/bin/bash
 
case $1 in
"start"){
        for i in hadoop102 hadoop103
        do
                echo " --------启动 $i 采集flume-------"
                ssh $i "nohup /opt/module/flume/apache-flume-1.9.0-bin/bin/flume-ng agent --conf /opt/module/flume/apache-flume-1.9.0-bin/conf --conf-file /opt/module/flume/apache-flume-1.9.0-bin/myconf/file_to_kafka.conf --name a1 -Dflume.root.logger=INFO,console >/dev/null 2>&1 &"
        done
};; 
"stop"){
        for i in hadoop102 hadoop103
        do
                echo " --------停止 $i 采集flume-------"
                ssh $i "ps -ef | grep file_to_kafka.conf | grep -v grep |awk  '{print \$2}' | xargs -n1 kill -9 "
        done
 
};;
esac

集群脚本

hadoop

#!/bin/bash
if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit ;
fi
case $1 in
"start")
        echo " =================== 启动 hadoop集群 ==================="
		echo "node1的journalnode启动"
        ssh node1 "hdfs --daemon start journalnode"
        echo "node2的journalnode启动"
        ssh node2 "hdfs --daemon start journalnode"
		echo "node3的journalnode启动"
        ssh node3 "hdfs --daemon start journalnode"
		
		
        echo " --------------- 启动 hdfs ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/sbin/start-dfs.sh"
        echo " --------------- 启动 yarn ---------------"
        ssh master2 "/home/bigdata/software/hadoop-3.1.3/sbin/start-yarn.sh"
		
        echo " --------------- 启动 historyserver ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/bin/mapred --daemon start historyserver"  
;;
"stop")
        echo " =================== 关闭 hadoop集群 ==================="
        echo " --------------- 关闭 historyserver ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
		
        echo " --------------- 关闭 yarn ---------------"
        ssh master2 "/home/bigdata/software/hadoop-3.1.3/sbin/stop-yarn.sh"
        echo " --------------- 关闭 hdfs ---------------"
        ssh master1 "/home/bigdata/software/hadoop-3.1.3/sbin/stop-dfs.sh"
		
		echo "node1的journalnode关闭"
        ssh node1 "hdfs --daemon stop journalnode"
        echo "node2的journalnode关闭"
        ssh node2 "hdfs --daemon stop journalnode"
		echo "node3的journalnode关闭"
        ssh node3 "hdfs --daemon stop journalnode"
;;
*)
    echo "Input Args Error..."
;;
esac

jpsall.sh 

#! /bin/bash

for i in master1 master2 node1 node2 node3
do
    echo --------- $i ----------
    ssh $i "jps"
done

kafka

#!/bin/bash

case $1 in
"start"){
    for i in node1 node2 node3
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/home/bigdata/software/kafka_2.11-2.4.1/bin/kafka-server-start.sh -daemon /home/bigdata/software/kafka_2.11-2.4.1/config/server.properties "
    done
};;
"stop"){
    for i in node1 node2 node3
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/home/bigdata/software/kafka_2.11-2.4.1/bin/kafka-server-stop.sh stop"
    done
};;
esac

xsync

#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
  echo Not Enough Arguement!
  exit;
fi
#2. 遍历集群所有机器
for host in master1 master2 node1 node2 node3
do
  echo ====================  $host  ====================
  #3. 遍历所有目录，挨个发送
  for file in $@
  do
    #4 判断文件是否存在
    if [ -e $file ]
    then
      #5. 获取父目录
      pdir=$(cd -P $(dirname $file); pwd)
      #6. 获取当前文件的名称
      fname=$(basename $file)
      ssh $host "mkdir -p $pdir"
      rsync -av $pdir/$fname $host:$pdir
    else
      echo $file does not exists!
    fi
  done
done

zk

#!/bin/bash

                #检验参数
                if [ $# -lt 1 ]
                then
                 echo '参数不能为空！！！'
                 exit
                fi

                #循环遍历每一台机器，分别启动或者停止ZK服务

                for host in node1 node2 node3
                do
                  case $1 in
                   "start")
                        echo "*****************$1 $host zookeeper****************"
                                ssh $host /home/bigdata/software/zookeeper-3.5.7/bin/zkServer.sh $1
                   ;;

                   "stop")
                        echo "*****************$1 $host zookeeper****************"
                                ssh $host /home/bigdata/software/zookeeper-3.5.7/bin/zkServer.sh $1
                   ;;

                   "status")
                        echo "*****************$1 $host zookeeper****************"
                                ssh $host /home/bigdata/software/zookeeper-3.5.7/bin/zkServer.sh $1
                   ;;

                   *)
                           echo '参数有误！！！'
                           exit
                   ;;
                  esac
                done

如果启动mapreduce提交到Yarn

 注解掉hosts的127.0.0.1 还有 

然后重启hadoop，一定要重启！！！ 

工作变成艺术
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
电商离线数仓从0到1系列之二 hive，flume环境搭建
manshiai的博客
11-15 1813
hive安装 hive内置元数据库Derby，但一般采用自有的元数据库，这里用mysql hive集群分配： 软件 warehouse101 warehouse102 warehouse103 hadoop √ √ √ hive √ √ √ mysql √ 元数据（metastore）采用远程模式配置，配置规划： 服务 warehouse101 warehouse102 warehouse103 metastore √ √ HiveServer2 √
大数据项目离线数仓（全）三（可视化工具版）
qq_55906442的博客
06-03 1168
本文仅仅包含可视化的工具版，其他内容请关注我的博客！在专栏里！！！本篇文章参考尚硅谷大数据项目写成！可视化的代码版后期更新！敬请期待！！！目录工具可视化（superset）介绍：一、安装准备1.1安装Miniconda二、安装配置2.1安装Miniconda2.2安装python3.6环境 2.3配置superset2.4启动superset三、superset制作可视化3.1数据源配置 3.2 制作仪表盘 3.3创建图表 3.4整体可视化配置信息完成样式Superset概述Apache Super....
大数据项目离线数仓（全 ）一（数据采集平台）
qq_55906442的博客
05-24 4247
一、用户行为数据采集平台 二、业务数据采集平台 三、数据仓库系统 四、可视化报表
ubuntu17.10安装JDK1.8.0_161和Maven3.5.3
CSDN 正在崛起的新人---小麦
04-15 1087
官网下载JDK：http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html下载完成后解压到/opt/soft/java/jdk1.8.0_161然后设置环境变量：sudo vi ~/.bashrc在最后加入下面的内容后，按esc，shift+：，wq，保存文件更改。重启SSH终端输入java...
大数据项目离线数仓（全 ）二（数仓系统）
qq_55906442的博客
05-28 2837
一、数据仓库系统 1.1基础概念 3.1.1数据分层的好处 1.把复杂问题简单化，将复杂的任务分解成多层来完成，每一层只处理简单任务，方便定位问题。 2.减少重复开发，规范数据分层，通过中间层数据，能够减少极大的重复计算，增加一次计算结果的复用性。 3.隔离原始数据：不论是数据的异常还是数据的敏感性，使真实数据与统计数据解耦开。 3.1.2数据分层 ODS：原始数据层,存放原始数据,直接加载原始日志、数据,数据保持原貌不做处理。 DWD：对ODS层数据进行清洗(去除空值,脏数据,超过极限范围
离线数据仓库从0到1-阶段一资源购买配置
S1124654的博客
07-24 210
离线数据仓库从0到1-阶段一资源购买配置
O2O行业数据平台实战从监控到诊断的数据产品搭建共33页
11-19
通常，我们会采用如Hive、HBase等大数据存储系统，构建离线数据分析环境，配合ETL（Extract, Transform, Load）流程，对历史数据进行清洗、整合，形成结构化的数据仓库，为后续的深度分析提供基础。 接着，数据产品...
离线计算项目案例--版本轨迹统计1
08-08
离线计算项目通常涉及到大数据处理，本案例是一个关于版本轨迹统计的项目，主要目标是对网站和APP的用户行为数据进行深度分析，以便了解产品的运营状况。在这个过程中，Hadoop技术被应用来处理海量数据。 首先，...
2-5+大数据平台从0到1之后.pdf
03-18
【大数据平台从0到1之后】的演进与关键技术 在大数据领域，构建一个高效、稳定的数据平台是企业数字化转型的关键。本报告以链家为例，详细介绍了大数据平台从无到有的过程，以及在此过程中遇到的挑战和解决方案，...
快速实现 sr 集群之间的数据迁移（离线迁移）.zip
01-03
总之，StarRocks集群间的离线数据迁移是一个复杂但必要的过程，涉及多个阶段和注意事项。正确执行迁移可以确保业务的连续性和数据的安全性。通过使用提供的工具和方法，可以有效地管理和执行这样的迁移任务。
如何从0到1构建用户画像系统.pptx
03-23
1. **目标解读**：在构建用户画像系统的初始阶段，需要明确画像的业务目标，例如提高用户粘性、增加转化率或优化个性化推荐。这需要与业务团队紧密合作，理解他们的需求和期望效果。 2. **需求分解与调研**：了解...
从0到1建设数据仓库—数仓基础篇
cdfunlove的博客
04-13 1355
数据仓库建设主题是系列篇，目的是带大家从了解数据仓库的基础知识开始，循序渐进，学会数据仓库的建设，本篇是数仓建设的第一篇，主要讲讲数据仓库的基础知识。 01 什么是数据仓库 数据仓库，英文名称为DataWarehouse，可简写为DW或DWH。数据仓库，是为企业所有级别的决策制定过程，提供所有类型数据支持的战略集合。它出于分析性报告和决策支持目的而创建。为需要业务智能的企业，提供指导业务流程改进、监视时间、成本、质量以及控制。 02 数据仓库的特点 面向主题性 传统数据库面向应
数仓02-从0到1设计和搭建
DCHAO的博客
02-02 4318
数仓的知识
记一波数据仓库从0到1的实战经历
weixin_37944880的博客
05-29 4536
之前算是经历了一个团队的数据体系从比较原始的状态到相对比较完善的数据仓库的演变过程，大概记录下，有个方向和思路。 1.原始状态 业务上 纯需求驱动，无明确分工； 根据需求，联系原数据方，通过各种技术手段得到结果。 技术上 数据存储 主要依赖oracle和mysql，绝大多数存储在oracle中。 任务调度 大部分通过可执行jar包放到服务器上起crontab任务定时执行jar包，部分写在自己的j...
正式邀测！ OS Copilot——一款基于大模型构建的 Linux 智能操作系统助手
weixin_60347558的博客
07-02 525
重塑操作系统交互新体验，开启您的智能化系统管理之旅
Linux存储管理I
m0_54563444的博客
07-03 774
主要知识点: 基本分区、逻辑卷LVM、EXT3/4/XFS文件系统、RAID。
RedHat运维-LinuxSELinux基础4-端口绑定SELinux上下文
最新发布
a15735748145a的博客
07-06 384
12. 将60001/tcp端口上的SELinux上下文由http_port_t调整为gopher_port_t的方法是______________________；13. 将60002/tcp端口上的SELinux上下文由gopher_port_t调整为ssh_port_t的方法是______________________；14. 将60003/tcp端口上的SELinux上下文由ssh_port_t调整为http_port_t的方法是_______________________；
Linux--V4L2摄像头驱动框架及UVC浅析
qq_47258284的博客
07-04 912
对于一个usb摄像头，它的内核驱动源码位于/drivers/media/usb/uvc/核心层：V4L2_dev.c文件硬件相关层： uvc_driver.c文件本篇记录基于对6.8.8.8内核下vivid-core.c文件（虚拟视频驱动程序）的分析，梳理Linux系统中vedio视频设备的驱动框架。
Linux 常用指令详解
TheJustice_的博客
07-03 433
Linux 是一个强大而灵活的操作系统，掌握常用的 Linux 指令是使用和管理 Linux 系统的基础。本文将介绍一些常用的 Linux 指令，并附上 Vim 和 g++ 的常用指令说明，帮助你更好地进行开发和操作。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

工作变成艺术

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值