hadoop完全分布式+hive+sqoop+hbase+spark+zookeeper

于 2021-06-20 15:40:47 发布
阅读量7.2k 收藏 49
点赞数 9
文章标签： hadoop hive 大数据 sqoop 运维
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/su_mingyang/article/details/118070573
版权

================

IP地址用户名密码主机名
10.1.1.101rootpasswordmaster
10.1.1.102rootpasswordslave1
10.1.1.103rootpasswordslave2
主机名IP服务名用户名密码
master10.1.1.101mysqlrootpassword
组件版本Linux版本下载地址
java8https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html
hadoop2.7.1https://archive.apache.org/dist/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz
hive2.0.0https://archive.apache.org/dist/hive/hive-2.0.0/apache-hive-2.0.0-bin.tar.gz
sqoop1.4.7https://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
zookeeper3.4.8https://archive.apache.org/dist/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
spark2.0.0https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz
hbase1.2.1https://archive.apache.org/dist/hbase/1.2.1/hbase-1.2.1-bin.tar.gz

注:所有安装包都在/h3cu下,本环境已经安装好mysql

1、虚拟机基础配置

1.1、修改主机名

  1. master

    [root@localhost ~]# hostnamectl set-hostname master 
[root@localhost ~]# bash
[root@master ~]#

  2. slave1

    [root@localhost ~]# hostnamectl set-hostname slave1
[root@localhost ~]# bash 
[root@slave1 ~]#

  3. slave2

    [root@localhost ~]# hostnamectl set-hostname slave2
[root@localhost ~]# bash 
[root@slave2 ~]#

1.2、配置hosts域名解析

1.2.1、查看三台主机ip地址

  1. master

    [root@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5a:51:a2 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.101/24 brd 10.1.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe5a:51a2/64 scope link 
       valid_lft forever preferred_lft forever

  2. slave1

    [root@slave1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:c8:8c:15 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.102/24 brd 10.1.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fec8:8c15/64 scope link 
       valid_lft forever preferred_lft forever

  3. slave2

    [root@slave2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:43:d4:07 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.103/24 brd 10.1.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe43:d407/64 scope link 
       valid_lft forever preferred_lft forever
1.2.2、master配置hosts文件
[root@master ~]# vi /etc/hosts
10.1.1.101 master
10.1.1.102 slave1
10.1.1.103 slave2
1.2.3、测试连接性
[root@master ~]# ping master -c 2
PING master (10.1.1.101) 56(84) bytes of data.
64 bytes from master (10.1.1.101): icmp_seq=1 ttl=64 time=0.137 ms
64 bytes from master (10.1.1.101): icmp_seq=2 ttl=64 time=0.084 ms

--- master ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 0.084/0.110/0.137/0.028 ms
[root@master ~]# ping slave1 -c 2
PING slave1 (10.1.1.102) 56(84) bytes of data.
64 bytes from slave1 (10.1.1.102): icmp_seq=1 ttl=64 time=0.497 ms
64 bytes from slave1 (10.1.1.102): icmp_seq=2 ttl=64 time=0.388 ms

--- slave1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.388/0.442/0.497/0.058 ms
[root@master ~]# ping slave2 -c 2
PING slave2 (10.1.1.103) 56(84) bytes of data.
64 bytes from slave2 (10.1.1.103): icmp_seq=1 ttl=64 time=0.430 ms
64 bytes from slave2 (10.1.1.103): icmp_seq=2 ttl=64 time=0.214 ms

--- slave2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.214/0.322/0.430/0.108 ms

  1. 将hosts文件分发给slave1和slave2

    [root@master ~]# scp /etc/hosts slave1:/etc/
The authenticity of host 'slave1 (10.1.1.102)' can't be established.
ECDSA key fingerprint is a2:ec:2f:3a:b9:33:d3:a7:fd:51:9d:d7:cf:ce:fb:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,10.1.1.102' (ECDSA) to the list of known hosts.
root@slave1's password: 
hosts                                                                                      100%  212     0.2KB/s   00:00    
[root@master ~]# scp /etc/hosts slave2:/etc/
The authenticity of host 'slave2 (10.1.1.103)' can't be established.
ECDSA key fingerprint is a2:ec:2f:3a:b9:33:d3:a7:fd:51:9d:d7:cf:ce:fb:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,10.1.1.103' (ECDSA) to the list of known hosts.
root@slave2's password: 
hosts                                                                                      100%  212     0.2KB/s   00:00

  2. 关闭防火墙

    1. master

      [root@master ~]# systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-06-19 03:14:26 CST; 5h 7min ago
 Main PID: 872 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─872 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Jun 19 03:14:25 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Jun 19 03:14:26 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
[root@master ~]# systemctl stop firewalld 
[root@master ~]# systemctl disable firewalld 
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

    2. slave1

      [root@slave1 ~]# systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-06-19 03:14:50 CST; 6h ago
 Main PID: 875 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─875 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Jun 19 03:14:50 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Jun 19 03:14:50 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
[root@slave1 ~]# systemctl stop firewalld 
[root@slave1 ~]# systemctl disable firewalld 
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

    3. slave2

      [root@slave2 ~]# systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-06-19 03:15:00 CST; 6h ago
 Main PID: 873 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─873 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Jun 19 03:15:00 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Jun 19 03:15:00 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
[root@slave2 ~]# systemctl stop firewalld 
[root@slave2 ~]# systemctl disable firewalld 
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

2、配置SSH

2.1、生成秘钥文件

  1. master

    [root@master ~]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
42:04:1c:9e:1d:ad:f2:ed:ae:4c:21:92:30:9d:3b:f8 root@master
The key's randomart image is:
+--[ RSA 2048]----+
|   .ooo.         |
| . o.+ ..        |
|o o o o.         |
| + o...          |
|. = .oo.S        |
| . o ..o.        |
|  E   ..         |
|     o  .        |
|      oo.        |
+-----------------+

  2. slave1

    [root@slave1 ~]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
30:22:3d:06:35:2d:ac:65:33:b4:0d:43:c2:ee:d0:c1 root@slave1
The key's randomart image is:
+--[ RSA 2048]----+
| oo=B.           |
|  E+B*.          |
| o.=*++          |
|. +o o o         |
| o      S        |
|  .              |
|                 |
|                 |
|                 |
+-----------------+

  3. slave2

    [root@slave2 ~]# ssh-keygetn -t rsa -P ''
bash: ssh-keygetn: command not found
[root@slave2 ~]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a3:01:98:3c:a6:ec:20:d0:57:0b:4b:56:0c:27:cf:c3 root@slave2
The key's randomart image is:
+--[ RSA 2048]----+
|    *++          |
| o = X..         |
|. B + E          |
|oo o . .         |
|+.    . S        |
|+      o .       |
| .    .          |
|                 |
|                 |
+-----------------+

2.2、master分发公钥

[root@master ~]# ssh-copy-id -i master 
The authenticity of host 'master (10.1.1.101)' can't be established.
ECDSA key fingerprint is a2:ec:2f:3a:b9:33:d3:a7:fd:51:9d:d7:cf:ce:fb:ea.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@master's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'master'"
and check to make sure that only the key(s) you wanted were added.

[root@master ~]# ssh-copy-id -i slave1 
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@slave1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'slave1'"
and check to make sure that only the key(s) you wanted were added.

[root@master ~]# ssh-copy-id -i slave2
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@slave2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'slave2'"
and check to make sure that only the key(s) you wanted were added.

2.3、测试master登录slave1

[root@master ~]# ssh slave1
Last login: Sat Jun 19 03:21:18 2021 from 10.1.1.1

3、配置java环境

3.1、卸载openjdk

[root@master ~]# rpm -qa |grep openjdk
通过rpm -e --nodeps "查询出来的rpm包" 去卸载

3.2、安装java

  1. 将/h3cu下面的java安装到/usr/local/src下

    [root@master ~]# tar -xzf /h3cu/jdk-8u144-linux-x64.tar.gz -C /usr/local/src/

  2. 将解压后的java文件重命名为java

    [root@master ~]# mv /usr/local/src/jdk1.8.0_144 /usr/local/src/java

  3. 配置java环境变量，仅使当前用户生效

    [root@master ~]# vi /root/.bash_profile
export JAVA_HOME=/usr/local/src/java
export PATH=$PATH:$JAVA_HOME/bin

  4. 加载环境变量，查看java的版本信息

    [root@master ~]# source /root/.bash_profile 
[root@master ~]# java -version 
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

  5. 将java分发给slave1和slave2

    [root@master ~]# scp -r /usr/local/src/java slave1:/usr/local/src/
[root@master ~]# scp -r /usr/local/src/java slave2:/usr/local/src/
[root@master ~]# scp /root/.bash_profile slave1:/root/
[root@master ~]# scp /root/.bash_profile slave2:/root/

4、配置hadoop完全分布式

  1. 将/h3cu下的hadoop解压到/usr/lcoal/src下(master上操作)

    [root@master ~]# tar -xzf /h3cu/hadoop-2.7.1.tar.gz -C /usr/local/src/

  2. 将解压后的hadoop文件重命名为hadoop

    [root@master ~]# ll /usr/local/src/
total 8
drwxr-xr-x. 9 10021 10021 4096 Jun 29  2015 hadoop-2.7.1
drwxr-xr-x. 8    10   143 4096 Jul 22  2017 java
[root@master ~]# mv /usr/local/src/hadoop-2.7.1 /usr/local/src/hadoop
[root@master ~]# ll /usr/local/src/
total 8
drwxr-xr-x. 9 10021 10021 4096 Jun 29  2015 hadoop
drwxr-xr-x. 8    10   143 4096 Jul 22  2017 java

  3. 配置hadoop环境变量，仅当前用户生效

    [root@master ~]# vi /root/.bash_profile 
export HADOOP_HOME=/usr/local/src/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

  4. 加载环境变量，查看hadoop版本

    [root@master ~]# source /root/.bash_profile 
[root@master ~]# hadoop version 
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using /usr/local/src/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

  5. 配置hadoop-env.sh

    [root@master ~]# vi /usr/local/src/hadoop/etc/hadoop/hadoop-env.sh 
export JAVA_HOME=/usr/local/src/java

  6. 配置core-site.xml

    1. 命令

      [root@master ~]# vi /usr/local/src/hadoop/etc/hadoop/core-site.xml

    2. 配置文件添加内容

      <property>
  <!--namenode的URL地址(必须写)-->
  <name>fs.defaultFS</name>
  <value>hdfs://master:9000</value>
</property>
<property>
  <!--SequenceFiles中使用的读/写缓冲区的大小，单位为KB,131072KB默认为64M(该配置可选)-->
  <name>io.file.buffer.size</name>
  <value>131072</value>
</property>
<property>
  <!--hadoop临时文件路径(可选配置)-->
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/src/hadoop/dfs/tmp</value>
</property>

  7. 配置hdfs-site.xml

    1. 命令

      [root@master ~]# vi /usr/local/src/hadoop/etc/hadoop/hdfs-site.xml

    2. 配置文件添加内容

      <property>
  <!--hadoop的副本数量，默认为3(必须写)-->
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <!--在本地文件系统所在的NameNode的存储空间和持续化处理日志(必须写)-->
  <name>dfs.namenode.name.dir</name>
  <value>/usr/local/src/hadoop/dfs/name</value>
</property>
<property>
  <!--在本地文件系统所在的DataNode的存储空间和持续化处理日志(必须写)-->
  <name>dfs.datanode.data.dir</name>
  <value>/usr/local/src/hadoop/dfs/data</value>
</property>
<property>
  <!--设置namenode线程，处理datanode发出rpc请求数量（可选配置）-->
  <name>dfs.namenode.handler.count</name>
  <value>100</value>
</property>

  8. 配置mapred-site.xml

    1. 命令

      [root@master ~]# cp /usr/local/src/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/src/hadoop/etc/hadoop/mapred-site.xml
[root@master ~]# vi /usr/local/src/hadoop/etc/hadoop/mapred-site.xml

    2. 配置文件添加内容

      <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

  9. 配置yarn-site.xml

    1. 命令

      [root@master ~]# vi /usr/local/src/hadoop/etc/hadoop/yarn-site.xml

    2. 配置文件添加内容(全部必须写入)

      <property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>  
    <name>yarn.resourcemanager.address</name>  
    <value>master:8032</value>  
</property> 
<property>
    <name>yarn.resourcemanager.scheduler.address</name>  
    <value>master:8030</value>  
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>  
    <value>master:8031</value>  
</property>

  10. 配置slaves

[root@master ~]# vi /usr/local/src/hadoop/etc/hadoop/slaves
master
slave1
slave2

  1. 将hadoop文件和环境变量文件分发给slave1和slave2

    [root@master ~]# scp -r /usr/local/src/hadoop slave1:/usr/local/src/
[root@slave2 ~]# scp -r /usr/local/src/hadoop slave2:/usr/local/src/
[root@slave1 ~]# scp /root/.bash_profile slave1:/root
[root@slave1 ~]# scp /root/.bash_profile slave2:/root

  2. namenode进行格式化

    [root@master ~]# hdfs namenode -format 
21/06/19 08:27:08 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/10.1.1.101
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.1
STARTUP_MSG:   classpath = /usr/local/src/hadoop/etc/hadoo(此处省略)
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a; compiled by 'jenkins' on 2015-06-29T06:04Z
STARTUP_MSG:   java = 1.8.0_144
************************************************************/
...省略
21/06/19 08:27:09 INFO namenode.FSImage: Allocated new BlockPoolId: BP-83508879-10.1.1.101-1624062429594
21/06/19 08:27:09 INFO common.Storage: Storage directory /usr/local/src/hadoop/dfs/name has been successfully formatted.
21/06/19 08:27:09 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
21/06/19 08:27:09 INFO util.ExitUtil: Exiting with status 0
21/06/19 08:27:09 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/10.1.1.101
************************************************************/

  3. 启动hadoop集群,查看守护进程

    1. 启动

      [root@master ~]# start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-root-namenode-master.out
slave1: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-root-datanode-slave1.out
slave2: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-root-datanode-slave2.out
master: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-root-datanode-master.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-root-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-root-resourcemanager-master.out
slave1: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-root-nodemanager-slave1.out
slave2: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-root-nodemanager-slave2.out
master: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-root-nodemanager-master.out

    2. jps查看

      1. master

        [root@master ~]# jps
10802 SecondaryNameNode
11042 NodeManager
10948 ResourceManager
10535 NameNode
10651 DataNode
11326 Jps

      2. slave1

        [root@slave1 ~]# jps
2660 NodeManager
2777 Jps
2555 DataNode

5、部署hive组件

  1. 将/h3cu下的hive解压到/usr/lcoal/src下(master上操作)

    [root@master ~]# tar -xzvf /h3cu/apache-hive-2.0.0-bin.tar.gz -C /usr/local/src/

  2. 将解压后的hadoop文件重命名为hadoop

    [root@master ~]# mv /usr/local/src/apache-hive-2.0.0-bin /usr/local/src/hive

  3. 配置hadoop环境变量，仅当前用户生效

    [root@master ~]# vi /root/.bash_profile 
export HIVE_HOME=/usr/local/src/hive
export PATH=$PATH:$HIVE_HOME/bin
[root@master ~]# source /root/.bash_profile

  4. 解压mysql-connect驱动包，将jar复制到hive的lib目录下

    [root@master ~]# tar -xzf /h3cu/mysql-connector-java-5.1.27.tar.gz -C /usr/local/src/
[root@master ~]# cp /usr/local/src/mysql-connector-java-5.1.27/mysql-connector-java-5.1.27-bin.jar /usr/local/src/hive/lib/

  5. 配置hive-site.xml

    1. 命令

      [root@master ~]# cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml
[root@master ~]# vi /usr/local/src/hive/conf/hive-site.xml

    2. 内容

      <property>
  <!--连接数据库URL(必选参数)-->
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
</property>
<property>
  <!--连接数据驱动(必选参数)-->
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<property>
  <!--数据库连接用户名(必选参数)-->
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
</property>
<property>
  <!--数据库连接密码(必选参数)-->
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>password</value>
</property>
<property>
  <!--验证元数据的一致性，默认为false(可选参数)-->
  <name>hive.metastore.schema.verification</name>
  <value>false</value>
</property>
<property>
  <!--指定HDFS内hive数据临时文件存放目录,启动hive>，HDFS自动创建(可选参数)-->
  <name>hive.exec.scratchdir</name>
  <value>/hive/warehouse/tmp</value>
</property>
<property>
  <!--指定HDFS内hive数据的存放目录，HDFS自动创建(可选参数)-->
  <name>hive.metastore.warehouse.dir</name>
  <value>/hive/warehouse/home</value>
</property>
<property>
  <!--客户端显示当前数据库名称信息(可选参数)-->
  <name>hive.cli.print.header</name>
  <value>true</value>
</property>
<property>
  <!--客户端显示当前数据库名称(可选参数)-->
  <name>hive.cli.print.current.db</name>
  <value>true</value>
</property>
<property>
  <!--支持正则匹配(可选参数)-->
  <name>hive.support.quoted.identifiers</name>
  <value>none</value>
</property>

  6. hive初始化

    [root@master ~]# schematool -dbType mysql -initSchema
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/mysql/bin:/root/bin:/usr/local/src/java/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/root/bin:/usr/local/src/java/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/src/hive/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 root
Starting metastore schema initialization to 2.0.0
Initialization script hive-schema-2.0.0.mysql.sql
Initialization script completed
schemaTool completed

  7. 进入hive shell

    [root@master ~]# hive 
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/mysql/bin:/root/bin:/usr/local/src/java/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/root/bin:/usr/local/src/java/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/src/hive/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
cHive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive (default)> show databases;
OK
database_name
default
Time taken: 0.707 seconds, Fetched: 1 row(s)
hive (default)> create database test;
OK
Time taken: 0.178 seconds
hive (default)> show databases;
OK
database_name
default
test
Time taken: 0.021 seconds, Fetched: 2 row(s)

6、部署sqoop组件

  1. 将/h3cu下的sqoop解压到/usr/local/src下

    [root@master ~]# tar -xzvf /h3cu/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C   /usr/local/src/

  2. 将解压后的sqoop文件重命名为sqoop

    [root@master ~]# mv /usr/local/src/sqoop-1.4.7.bin__hadoop-2.6.0 /usr/local/src/sqoop

  3. 配置sqoop环境变量

    [root@master ~]# vi /root/.bash_profile 
export SQOOP_HOME=/usr/local/src/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

  4. 加载sqoop环境变量，查看sqoop版本信息

    [root@master ~]# source /root/.bash_profile 
[root@master ~]# sqoop version 
Warning: /usr/local/src/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/src/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
21/06/21 02:13:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

  5. 添加mysql驱动

    [root@master ~]# cp /usr/local/src/mysql-connector-java-5.1.27/mysql-connector-java-5.1.27-bin.jar /usr/local/src/sqoop/lib/

  6. 配置sqoop-env.sh

    [root@master ~]# cp /usr/local/src/sqoop/conf/sqoop-env-template.sh /usr/local/src/sqoop/conf/sqoop-env.sh
[root@master ~]# vi /usr/local/src/sqoop/conf/sqoop-env.sh
export HADOOP_COMMON_HOME=/usr/local/src/hadoop
export HADOOP_MAPRED_HOME=/usr/local/src/hadoop
export HIVE_HOME=/usr/local/src/hive

  7. 测试sqoop

    [root@master ~]# sqoop list-databases --connect jdbc:mysql://master:3306 --username root --password password
Warning: /usr/local/src/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/src/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
21/06/21 02:20:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/06/21 02:20:05 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/06/21 02:20:05 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
performance_schema
sys
test

7、部署zookeeper集群

  1. 将/h3cu下的zookeeper解压到/usr/local/src

    [root@master ~]# tar -xvzf /h3cu/zookeeper-3.4.8.tar.gz -C /usr/local/src/

  2. 将解压后文件重命名为zookeeper

    [root@master ~]# mv /usr/local/src/zookeeper-3.4.8 /usr/local/src/zookeeper

  3. 配置zookeeper环境变量，加载环境变量，仅对当前用户生效

    [root@master ~]# vi /root/.bash_profile 
export ZOOKEEPER_HOME=/usr/local/src/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin
[root@master ~]# source /root/.bash_profile

  4. 配置zoo.cfg配置文件

    dataDir进行修改，server三行写入进去

    [root@master ~]# cp /usr/local/src/zookeeper/conf/zoo_sample.cfg /usr/local/src/zookeeper/conf/zoo.cfg
[root@master ~]# vi /usr/local/src/zookeeper/conf/zoo.cfg 
dataDir=/usr/local/src/zookeeper/data
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888

  5. 配置myid文件

    [root@master ~]# mkdir /usr/local/src/zookeeper/data
[root@master ~]# echo "1" > /usr/local/src/zookeeper/data/myid

  6. 将文件分发给slave1和slave2

    [root@master ~]# scp -r /usr/local/src/zookeeper slave1:/usr/local/src/ 
[root@master ~]# scp -r /usr/local/src/zookeeper slave2:/usr/local/src/
[root@master ~]# scp /root/.bash_profile slave1:/root/
[root@master ~]# scp /root/.bash_profile slave2:/root/

  7. 修改slave1和slave2的myid文件

    • slave1

      [root@slave1 ~]# echo 2 > /usr/local/src/zookeeper/data/myid

    • slave2

      [root@slave2 ~]# echo 3 > /usr/local/src/zookeeper/data/myid

  8. 分别启动zk集群

    • master

      [root@master ~]# source /root/.bash_profile 
[root@master ~]# zkServer.sh start 
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

    • slave1

      [root@slave1 ~]# source /root/.bash_profile 
[root@slave1 ~]# zkServer.sh start 
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

    • slave2

      [root@slave2 ~]# source /root/.bash_profile 
[root@slave2 ~]# zkServer.sh start 
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

  9. 分别查看zk集群的状态

    注意：leader和follower是选举出来不是固定在某台机器上

    [root@master ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@master ~]# jps
17120 QuorumPeerMain
16449 SecondaryNameNode
16594 ResourceManager
16695 NodeManager
16297 DataNode
17230 Jps
16175 NameNode
[root@slave1 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[root@slave1 ~]# jps
15587 NodeManager
15481 DataNode
15721 Jps
15050 QuorumPeerMain
[root@slave2 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@slave2 ~]# jps
14965 QuorumPeerMain
15480 NodeManager
15374 DataNode
15647 Jps

8、部署hbase完全分布式

  1. 将/h3cu下的hbase解压到/usr/local/src

    [root@master ~]# tar -xvzf /h3cu/hbase-1.2.1-bin.tar.gz -C /usr/local/src/

  2. 将解压后的文件重命名为hbase

    [root@master ~]# mv /usr/local/src/hbase-1.2.1/ /usr/local/src/hbase

  3. 配置hbase的环境变量

    [root@master ~]# vi /root/.bash_profile 
export HBASE_HOME=/usr/local/src/hbase
export PATH=$PATH:$HBASE_HOME/bin
[root@master ~]# source /root/.bash_profile

  4. 配置hbase-site.xml

    • 命令

    • 配置文件内容

      <property>
  <!--是否分布式部署（必选）-->
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <!--hbase存放数据目录（必选）-->
  <name>hbase.rootdir</name>
  <value>hdfs://master:9000/hbase</value>
</property>
<property>
  <!--zookooper配置、日志等的存储位置（必选）-->
  <name>hbase.zookeeper.property.dataDir</name>
  <value>/usr/local/src/zookeeper/ZKdata</value>
</property>
<property>
  <!--配置zk端口（必选）-->
  <name>hbase.zookeeper.property.clientPort</name>
  <value>2181</value>
</property>
<property>
  <!--zookeeper地址（必选）-->
  <name>hbase.zookeeper.quorum</name>
  <value>master,slave1,slave2</value>
</property>
<property>
  <!--设置hbase端口（必选）-->
  <name>hbase.master.info.port</name>
  <value>16010</value>
</property>

  5. 配置hbase-env.sh

    [root@master ~]# vim /usr/local/src/hbase/conf/hbase-env.sh 
export JAVA_HOME=/usr/local/src/java
export HBASE_MANAGES_ZK=false

  6. 配置regionservers

    [root@master ~]# vim /usr/local/src/hbase/conf/regionservers 
master
slave1
slave2

  7. 配置备用backup-master

    [root@master ~]# vim /usr/local/src/hbase/conf/backup-masters
slave1

  8. 分发给slave1和slave1

    [root@master ~]# scp /root/.bash_profile slave1:/root/
[root@master ~]# scp /root/.bash_profile slave2:/root/
[root@master ~]# scp -r /usr/local/src/hbase slave1:/usr/local/src/ 
[root@master ~]# scp -r /usr/local/src/hbase slave2:/usr/local/src/

  9. 启动hbase

    [root@master ~]# start-hbase.sh 
starting master, logging to /usr/local/src/hbase/logs/hbase-root-master-localhost.localdomain.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
slave2: starting regionserver, logging to /usr/local/src/hbase/bin/../logs/hbase-root-regionserver-slave2.out
slave1: starting regionserver, logging to /usr/local/src/hbase/bin/../logs/hbase-root-regionserver-slave1.out
master: starting regionserver, logging to /usr/local/src/hbase/bin/../logs/hbase-root-regionserver-master.out
slave1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
slave1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
slave2: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
slave2: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
master: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
master: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
slave1: starting master, logging to /usr/local/src/hbase/bin/../logs/hbase-root-master-slave1.out
slave1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
slave1: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
[root@master ~]# jps
17120 QuorumPeerMain
18273 SecondaryNameNode
18114 DataNode
19602 HRegionServer
17987 NameNode
18423 ResourceManager
18537 NodeManager
19465 HMaster
19803 Jps
[root@slave1 ~]# jps
15587 NodeManager
16099 HRegionServer
16184 HMaster
15481 DataNode
15050 QuorumPeerMain
16412 Jps
[root@slave2 ~]# jps
14965 QuorumPeerMain
15480 NodeManager
16090 Jps
15374 DataNode
15935 HRegionServer

9、部署spark完全分布式

  1. 将/h3cu下的spark解压到/usr/local/src

    [root@master ~]# tar -xvzf /h3cu/spark-2.0.0-bin-hadoop2.7.tgz -C /usr/local/src/

  2. 将解压后的文件重名为spark

    [root@master ~]# mv /usr/local/src/spark-2.0.0-bin-hadoop2.7 /usr/local/src/spark

  3. 配置spark-env.sh

    [root@master ~]# cp  /usr/local/src/spark/conf/spark-env.sh.template  /usr/local/src/spark/conf/spark-env.sh
[root@master ~]# vi /usr/local/src/spark/conf/spark-env.sh
# java位置
export JAVA_HOME=/usr/local/src/java
# master节点IP或域名
export SPARK_MASTER_IP=master
# worker内存大小
export SPARK_WORKER_MEMORY=1G
# Worker的cpu核数
SPARK_WORKER_CORES=1
# hadoop配置文件路径
export HADOOP_CONF_DIR=/usr/local/src/hadoop/etc/hadoop

  4. 配置slaves

    [root@master ~]# cp /usr/local/src/spark/conf/slaves.template /usr/local/src/spark/conf/slaves
[root@master ~]# vi /usr/local/src/spark/conf/slaves
master
slave1
slave2

  5. 分发文件

    [root@master ~]# scp -r /usr/local/src/spark slave1:/usr/local/src/  
[root@master ~]# scp -r /usr/local/src/spark slave2:/usr/local/src/

  6. 启动spark集群

    [root@master ~]# /usr/local/src/spark/sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/local/src/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-localhost.localdomain.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/src/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/src/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out
master: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/src/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out
[root@master ~]# jps
17120 QuorumPeerMain
18273 SecondaryNameNode
18114 DataNode
19602 HRegionServer
17987 NameNode
20949 Worker
18423 ResourceManager
20999 Jps
18537 NodeManager
19465 HMaster
20860 Master
[root@slave1 ~]# jps
15587 NodeManager
16099 HRegionServer
17091 Worker
17141 Jps
16184 HMaster
15481 DataNode
15050 QuorumPeerMain
Hadoop2.2.0 HA高可用分布式集群搭建(hbase,hive,sqoop,spark)
edent-ON THE WAY
09-07 4777
1 需要软件 Hadoop-2.2.0 Hbase-0.96.2（这里就用这个版本，跟Hadoop-2.2.0是配套的，不用覆盖jar包什么的） Hive-0.13.1 Zookeepr-3.4.6（建议使用Zookeepr-3.4.5，这样就不用替换storm和hive里面的zookeepr-3.4.5.jar了） Sqoop1.4.5 Scala-2.10.4 Spark-1.0
最全图文教你怎么搭建分布式hadoop+zkp+hbase+mysql+hive+sqoop
我的博客
03-19 1009
转载需说明出处！！！                               分布式集群安装所用版本： hive/lib 下放的mysql是5.1.29版本sqoop/lib 下放的mysql也是5.1.29版本hadoop:2.6.0zookeeper:3.4.5hbase :0.96.2mysql:5.1.73删除原有的mysql 再进行安装新的mysql安装在 /usr/bin/mysq...
使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟
weixin_30500473的博客
12-08 797
使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟Sqoop大数据HiveHBaseETL使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟基础环境Sqool和Hive、HBase简介SqoopHiveHBase测试Sqoop使用Sqoop从MySQL导入数据到Hive使用复杂SQL调整Hive数据类型不断更新使用Sqoop从MySQL导入数据到HBas...
Hadoop相关项目Hive-Pig-Spark-Storm-HBase-Sqoop
weixin_30437847的博客
01-13 266
Hadoop相关项目Hive-Pig-Spark-Storm-HBase-Sqoop的相关介绍。 Hive Pig和Hive的对比 摘要: Pig Pig是一种编程语言，它简化了Hadoop常见的工作任务。Pig可加载数据、表达转换数据以及存储最终结果。Pig内置的操作使得半结构化数据变得有意义（如日志文件）。同时Pig可扩展使用Java中添加的自定义数据类型并支持数据转换。 Hiv...
大数据伪分布式环境搭建hadoop+zookeeper+hive+sqoop+hbase
Linzhongyilisha的专栏
08-19 3221
最近开始学习大数据，自己在虚拟机中搭建了一套环境，因为参考的博客写的很详细，自己不再重写，只给出链接：http://blog.csdn.net/pucao_cug?viewmode=contents 自己在搭建过程中，出的一些问题： 1.配置authorized_keys文件     生成key文件后，在hserver1上拷贝时少了开头的一个字母，结果hserver1通过ssh跳转hs
zookeeper+hadoop+hbase+hive(集成hbase)安装部署教程（超详细）.docx
06-01
jdk1.8.0_131、apache-zookeeper-3.8.0、hadoop-3.3.2、hbase-2.4.12 mysql5.7.38、mysql jdbc驱动mysql-connector-java-8.0.8-dmr-bin.jar、 apache-hive-3.1.3 2.本文软件均安装在自建的目录/export/server/下 ...
Hadoop+Hive+Spark+Kafka+Zookeeper+Flume+Sqoop+Azkaban+Scala
最新发布
05-05
基于 Zookeeper 搭建 Hadoop 高可用集群 二、Hive 简介及核心概念 Linux 环境下 Hive 的安装部署 CLI 和 Beeline 命令行的基本使用 常用 DDL 操作 分区表和分桶表 视图和索引 常用 DML 操作 数据查询详解 三、Spark ...
Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+ES+Redash等详细安装部署
06-22
在大数据领域，构建一个完整的生态系统是至关重要的，其中包括多个组件，如Hadoop、Spark、Hive、HBase、Oozie、Kafka、Flume、Flink、Elasticsearch和Redash。这些组件协同工作，提供了数据存储、处理、调度、流...
Hadoop集群的完整搭建(Hadoop完全分布式+ZooKeeper+Mysql+Hive+Hbase完全分布式+Spark完全分布式)
HR的博客
12-19 2814
Hadoop集群的完整搭建准备工作网络环境的配置修改网关、IP配置映射、主机名重启网络服务，测试ssh的配置jdk的安装Hadoop完全分布式的安装准备工作修改配置文件slaveshadoop-env.shyarn-env.shcore-site.xmlhdfs-site.xmlmapred-site.xmlyarn-site.xml分发到其他节点测试安装Zookeeper准备工作修改配置文件分发即修改文件测试 准备工作 网络环境的配置 修改网关、IP 打开要修改的文件(最后一个文件名可能每个人的不一样)
hadoop+hive+spark+zookeeper+hbase大数据环境搭建
fengwuxichen的博客
04-17 933
1. 硬件环境 主机名 IP地址 操作系统 master 172.16.34.101 CentOS Linux release 7.3.1611 slave01 172.16.34.102 CentOS Linux release 7.3.1611 slave03 172.16.34.103 CentOS Linux release 7.3.1611 2. 软件版本 ...
Hadoop hbase hive sqoop集群环境安装配置及使用文档
10-25
Hadoop hbase hive sqoop集群环境安装配置及使用文档
Hadoop+Hbase+Spark+Hive搭建
08-12
全套的Hadoop+Hbase+Spark+Hive搭建指导手册
Spark+Hbase 亿级流量分析实战（数据结构设计）
SmallCatBaby的博客
03-23 284
靠文章生存的大厂们简书/小红书/CSDN(PS:好吧你们仨记得给我广告费)，对优秀的文章进行大数据分析的工作必不可以少了，本系列文章将会从关于文章的各个维度进行实战分析，这系列文章新手可借着踏入大数据研发的大门，至于大数据的大佬们可以一起来相互伤害，至少为什么取名为 ''亿级流量分析实战'' 看完后整个系列的文章你就知道了，相信大家都是会举一反三的孩子们...
hadoop、hbase、hive、spark分布式系统架构详细搭建过程
雨花石
03-20 2103
参考文档：http://www.shareditor.com/blogshow/?blogId=96，在这里表示感谢！以下内容用到的测试软件版本及下载地址：操作系统：Centos 6.9 Hadoop2.7.5 下载地址：http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.5/hadoop-2.7.5.tar.gzHbase1.2.3 ...
Hadoop+hbase+zookeeper+spark+sqoop
danevc的博客
02-10 477
单机环境安装Hadoop+hbase+zookeeper+spark+sqoop软件：maven bin  version:3.3.9 hadoop bin version:2.7.1hbase bin version:1.1.2ojdbc6scala scala-2.11.7.tgzSpark spark-1.5.2-bin-hadoop2.6.tgzzookeeper 3.4.7jdk
hadoop、hive、sqoop、spark、livy、jdk单节点伪分布式集群一键部署shell脚本
weixin_42240930的博客
05-07 597
ards.1.0安装说明 注：脚本tgz包后续上传，欢迎留言与我交流讨论 一、使用说明 本脚本可实现快速自动安装（hadoop-2.7.3/hive-1.1.0/sqoop-1.4.6/spark-2.3.0/livy/jdk1.8）集群功能，提前阅读以下说明（脚本内亦有提示）有助于您使用此脚本 1、脚本经centos6.5、centos7.4/7.5测试安装正常,脚本内输入错误可使用Ctr...
Hadoop、hive、sqoop入门及完整小例子
fight4gold的专栏
07-27 2656
Hadoop、hive、sqoop入门及小例子
hadoop、hive、spark、hbase综述
yangzixuan_0608的博客
12-03 491
hadoop综述 1 namenode 一般在master节点上，用来存储文件和目录的所有元数据信息 常见面试问题："namenode和datanode可以在同一个节点吗？" -- 原理上可以在一个节点，但是这样做有很多弊端，比如datanode是数据的实际执行节点，对资源的占用会影响namenode节点的性能; 2 datanode 一般在...
大数据集群搭建jdk,hadoop,mysql,hive,spark,sqoop,flume......
qq_38220334的博客
09-20 367
基础环境： jdk版本： Hadoop版本： vi /etc/sysconfig/network-scripts/ifcfg-eno16777736 修改机器为静态IP 原来的配置 TYPE=Ethernet BOOTPROTO=dhcp DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no NAME=eno16777736 UUID=25b39fc8
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值