项目0单节点的虚拟机做大数据开发（四万字全）

南潇如梦

已于 2022-03-27 22:13:04 修改

阅读量2.7k

点赞数 1

分类专栏：运维大数据那些事大数据项目文章标签： hadoop big data hive

于 2022-03-21 01:33:40 首次发布

本文链接：https://blog.csdn.net/weixin_45682261/article/details/123625762

版权

本文提供了一个详尽的教程，指导如何在单节点虚拟机上进行大数据开发。从新建虚拟机开始，逐步讲解如何配置网络、设置免密登录、同步时间，并详细介绍了安装 JDK、MySQL、Hadoop、Hive、Zookeeper、Hbase、Presto、Sqoop等大数据相关软件的步骤。此外，还涉及到了Kafka、Grafana、Openresty等组件的安装，为全面的大数据环境构建提供了清晰的指南。

摘要由CSDN通过智能技术生成

1新建虚拟机

1.1保姆级教程建虚拟机

1.2设置网络ip和主机名（设置好了才能使用远程工具）

2.11.2.2azkaban-web-server的安装

说明：这个项目是面向大数据开发的，要涉及到多个软件的安装。以后还会更新整个项目的流程包括代码。

1新建虚拟机

1.1保姆级教程建虚拟机

上面这个镜像文件

链接：https://pan.baidu.com/s/1bdrE2pFtVXQt07oOrqw-KA
提取码：0fht
--来自百度网盘超级会员V2的分享

上图只需要点击安装位置和网络和主机名就可以开始安装

网络和主机名：打开，然后点击完成

点击开始安装

Root的密码

点击设置密码

完成

点击重启

输入root和密码就可以登录了。下面步骤全在虚拟机中完成。

1.2设置网络ip和主机名（设置好了才能使用远程工具）

编辑-》虚拟网络编辑器-》选择VMnet8，然后选择更改设置

选择VMnet8，点击确定

回到虚拟机终端界面：

[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet
BOOTPROTO=static         #三个值dhcp，static，none可选， dhcp表示动态分配，其他两个为静态
NAME=ens33
DEVICE=ens33
ONBOOT=yes        #开机后是否自动联网，yes为自动，no为手动
IPADDR=192.168.10.101  #IP
NETMASK=255.255.255.0   #子网掩码
GATEWAY=192.168.10.2        # 网关
DNS1=192.168.10.2
DNS2=8.8.8.8
DNS3=114.114.114.114

写完之后按esc，然后输入:wq

[root@localhost ~]# systemctl restart network

[root@localhost ~]# ip addr

修改主机名

[root@localhost ~]# hostnamectl set-hostname qianfeng01

主机名的查看
[root@localhost ~]# hostname

下面是修改映射文件

[root@qianfeng01 ~]# vi /etc/hosts

在后面添加 192.168.10.101 qianfeng01

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.101 qianfeng01

这边可以使用远程工具了

链接：https://pan.baidu.com/s/1OYtRB_1PxbQgHgS3S6yUOg
提取码：7qkt
--来自百度网盘超级会员V2的分享

傻瓜式安装，下图是创建一个ssh登录的步骤

然后是对虚拟机中防火墙的关闭

[root@qianfeng01 ~]# systemctl stop firewalld
[root@qianfeng01 ~]# systemctl stop NetworkManager

1.3免密登录的设置

[root@qianfeng01 ~]# ssh-keygen -t rsa   #一路回车
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:vIPh8966rpsZrBmE95dmpbV7vX/oyrOpXP09dw88/4o root@qianfeng01
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|                 |
|   .   .         |
|  . o . S o      |
|   o + o * . o   |
|    . B O . ..=. |
|     + X + +ooo*=|
|    o ==*o=oE=++#|
+----[SHA256]-----+

[root@qianfeng01 ~]#  ssh-copy-id -i qianfeng01   #对自己免密登录
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'qianfeng01 (192.168.10.101)' can't be established.
ECDSA key fingerprint is SHA256:k2XidB3kdX5fsbwIeKzqmyDpfC1JJ3kND7NBKPuuwA0.
ECDSA key fingerprint is MD5:0c:da:d1:98:c5:a7:e2:03:86:cb:30:8b:75:24:9c:80.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@qianfeng01's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'qianfeng01'"
and check to make sure that only the key(s) you wanted were added.

一个yes，一个密码

1.4同步时间

[root@qianfeng01 ~]# yum -y install ntpdate.x86_64

[root@qianfeng01 ~]# ntpdate -u ntp1.aliyun.com #下面不是代码，是执行完了之后的显示
26 Mar 09:38:49 ntpdate[12441]: adjust time server 120.25.115.20 offset 0.002909 sec

2、安装软件

在安装之前，我们先安装一下需要的软件环境，不然许多命令执行不了

不在意是在那个路径以及主机名，只要执行命令就可以了

[root@pure ~]# yum install -y curl.x86_64
[root@pure ~]# yum install -y wget.x86_64
[root@pure ~]# yum -y install net-tools.x86_64
[root@pure ~]# yum -y install bzip2.x86_64
[root@pure ~]# yum -y install unzip.x86_64
[root@pure ~]# yum -y install perl
[root@pure ~]# yum -y install zip.x86_64
[root@pure ~]# yum -y install psmisc
[root@pure ~]# yum -y install vim
[root@pure ~]# yum -y install gcc gcc-c++
[root@pure ~]# yum -y install ntpdate

2.1jdk的安装

[root@qianfeng01 ~]# mkdir -p /usr/local/

先把jdk的安装包拖进远程工具

安装包：

链接：https://pan.baidu.com/s/1AmHDfg_KpQL9Gm2sMGrAcQ
提取码：btpv
--来自百度网盘超级会员V2的分享

下面安装

[root@qianfeng01 ~]# tar -zxvf ./jdk-8u221-linux-x64.tar.gz -C /usr/local

[root@qianfeng01 ~]# cd /usr/local
[root@qianfeng01 local]# mv jdk1.8.0_221/ jdk

[root@qianfeng01 ~]#vi /etc/profile

在最后按一个o，添加内容,添加完了之后按ase键，按:wq

# JDK environment
JAVA_HOME=/usr/local/jdk
PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

[root@qianfeng01 ~]# source /etc/profile

验证

[root@qianfeng01 ~]# java -version
[root@qianfeng01 ~]# javac

2.3安装mysql

安装包

链接：https://pan.baidu.com/s/1vM0JdJFIGsDzLDq0ODQI9w
提取码：cmzp
--来自百度网盘超级会员V2的分享

把安装包拖到远程工具中（就是root目录下就可以了）

[root@qianfeng01 ~]# tar -xvf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar
[root@qianfeng01 ~]# ll    #下面不是代码

mysql-community-client-5.7.28-1.el7.x86_64.rpm
mysql-community-common-5.7.28-1.el7.x86_64.rpm
mysql-community-devel-5.7.28-1.el7.x86_64.rpm
mysql-community-embedded-5.7.28-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.28-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.28-1.el7.x86_64.rpm
mysql-community-libs-5.7.28-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm
mysql-community-server-5.7.28-1.el7.x86_64.rpm
mysql-community-test-5.7.28-1.el7.x86_64.rpm

[root@qianfeng01 ~]# rpm -qa | grep mariadb #下面这行不是代码
mariadb-libs-5.5.64-1.el7.x86_64
[root@qianfeng01 ~]# rpm -e mariadb-libs-5.5.64-1.el7.x86_64 --nodeps
2. 安装mysql-common
[root@qianfeng01 ~]# rpm -ivh mysql-community-common-5.7.28-1.el7.x86_64.rpm
3. 安装mysql-lib
[root@qianfeng01 ~]# rpm -ivh mysql-community-libs-5.7.28-1.el7.x86_64.rpm
4. 安装mysql-client
[root@qianfeng01 ~]# rpm -ivh mysql-community-client-5.7.28-1.el7.x86_64.rpm
5. 安装mysql-server
[root@qianfeng01 ~]# rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm

systemctl start mysqld

systemctl status mysqld

cat /var/log/mysqld.log | grep password #查看初始密码

[root@qianfeng01 ~]# cat /var/log/mysqld.log | grep password
2022-03-26T01:55:07.835729Z 1 [Note] A temporary password is generated for root@localhost: 45aO!haG/F3j

mysql -uroot -p'45aO!haG/F3j'

alter user root@'localhost' identified by '@Mmforu45';

grant all privileges on *.* to root@'%' identified by '@Mmforu45'; #远程授权登录

使用navicat去看看有没有远程成功

特别说明，windows系统需要改一下映射！不然可能后面浏览器无法访问软件的web界面

2.3hadoop的搭建

安装包：

链接：https://pan.baidu.com/s/1TvJ8KHMlZ9p3FLezTkA-uA
提取码：oy1v
--来自百度网盘超级会员V2的分享

因为考虑到后面的spark是2.7版本的，就换了一下。之前的hadoop是2.8.1

tar -zxvf hadoop-2.7.6.tar.gz -C /usr/local/

cd /usr/local

mv hadoop-2.7.6/ hadoop

vim /etc/profile

export HADOOP_HOME=/usr/local/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

source /etc/profile

改配置文件：重中之重

[root@qianfeng01 local]# cd hadoop
[root@qianfeng01 hadoop]# cd etc/hadoop
[root@qianfeng01 hadoop]# vi hadoop-env.sh

就是把那个文件中的JAVA_HOME路径改成本地的实际路径

export JAVA_HOME=/usr/local/jdk

#检验是否安装好环境

hadoop version

下面就是配置文件

cd /usr/local/hadoop/etc/hadoop/

vi slaves

文件添加一个qianfeng01

vi core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
        <!-- hdfs系统的唯一标识，scheme,ip,port ,内部守护进程的通信地址-->
        <name>fs.defaultFS</name>
        <value>hdfs://qianfeng01:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
    </property>
    <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
</property>
</configuration>

vi hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- namenode守护进程管理的元数据文件fsimage存储的位置-->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///usr/local/hadoop/hdpdata/dfs/name</value>
    </property>
    <!-- 确定DFS数据节点应该将其块存储在本地文件系统的何处-->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///usr/local/hadoop/hdpdata/dfs/data</value>
    </property>
    <!-- 块的副本数-->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <!-- 块的大小(128M),下面的单位是字节-->
    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
    </property>
    <!-- secondarynamenode守护进程的http地址：主机名和端口号。参考守护进程布局-->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>qianfeng01:50090</value>
    </property>
  	<!-- namenode守护进程的http地址：主机名和端口号。参考守护进程布局-->
	<property>
  	  <name>dfs.namenode.http-address</name>
  	  <value>qianfeng01:50070</value>
	</property>  
	<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///usr/local/hadoop/hdpdata/dfs/name</value>
    </property>
        <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>file:///usr/local/hadoop/hdpdata/dfs/cname</value>
    </property>
       <property>
        <name>dfs.namenode.checkpoint.edits.dir</name>
        <value>file:///usr/local/hadoop/hdpdata/dfs/cname</value>
    </property>
</configuration>

vi mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configura

最低0.47元/天解锁文章

南潇如梦

关注

1
点赞
踩
8

收藏

觉得还不错? 一键收藏
0
评论
项目0单节点的虚拟机做大数据开发（四万字全）

说明：这个项目是面向大数据开发的，要涉及到多个软件的安装。以后还会更新整个项目的流程包括代码。1.1新建虚拟机上面这个镜像文件链接：https://pan.baidu.com/s/1bdrE2pFtVXQt07oOrqw-KA提取码：0fht--来自百度网盘超级会员V2的分享上图只需要点击安装位置和网络和主机名就可以开始安装网络和主机名：打开，然后点击完成点击开始安装Root的密码..
复制链接

扫一扫

专栏目录