目录
说明:这个项目是面向大数据开发的,要涉及到多个软件的安装。以后还会更新整个项目的流程包括代码。
1新建虚拟机
1.1保姆级教程建虚拟机
上面这个镜像文件
链接:https://pan.baidu.com/s/1bdrE2pFtVXQt07oOrqw-KA
提取码:0fht
--来自百度网盘超级会员V2的分享
上图只需要点击安装位置和网络和主机名就可以开始安装
网络和主机名:打开,然后点击完成
点击开始安装
Root的密码
点击设置密码
完成
点击重启
输入root和密码就可以登录了。下面步骤全在虚拟机中完成。
1.2设置网络ip和主机名(设置好了才能使用远程工具)
编辑-》虚拟网络编辑器-》选择VMnet8,然后选择更改设置
选择VMnet8,点击确定
回到虚拟机终端界面:
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static #三个值dhcp,static,none可选, dhcp表示动态分配,其他两个为静态
NAME=ens33
DEVICE=ens33
ONBOOT=yes #开机后是否自动联网,yes为自动,no为手动
IPADDR=192.168.10.101 #IP
NETMASK=255.255.255.0 #子网掩码
GATEWAY=192.168.10.2 # 网关
DNS1=192.168.10.2
DNS2=8.8.8.8
DNS3=114.114.114.114
写完之后按esc,然后输入:wq
[root@localhost ~]# systemctl restart network
[root@localhost ~]# ip addr
修改主机名
[root@localhost ~]# hostnamectl set-hostname qianfeng01
主机名的查看
[root@localhost ~]# hostname
下面是修改映射文件
[root@qianfeng01 ~]# vi /etc/hosts
在后面添加 192.168.10.101 qianfeng01
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.101 qianfeng01
这边可以使用远程工具了
链接:https://pan.baidu.com/s/1OYtRB_1PxbQgHgS3S6yUOg
提取码:7qkt
--来自百度网盘超级会员V2的分享
傻瓜式安装,下图是创建一个ssh登录的步骤
然后是对虚拟机中防火墙的关闭
[root@qianfeng01 ~]# systemctl stop firewalld
[root@qianfeng01 ~]# systemctl stop NetworkManager
1.3免密登录的设置
[root@qianfeng01 ~]# ssh-keygen -t rsa #一路回车
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:vIPh8966rpsZrBmE95dmpbV7vX/oyrOpXP09dw88/4o root@qianfeng01
The key's randomart image is:
+---[RSA 2048]----+
| |
| |
| |
| . . |
| . o . S o |
| o + o * . o |
| . B O . ..=. |
| + X + +ooo*=|
| o ==*o=oE=++#|
+----[SHA256]-----+
[root@qianfeng01 ~]# ssh-copy-id -i qianfeng01 #对自己免密登录
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'qianfeng01 (192.168.10.101)' can't be established.
ECDSA key fingerprint is SHA256:k2XidB3kdX5fsbwIeKzqmyDpfC1JJ3kND7NBKPuuwA0.
ECDSA key fingerprint is MD5:0c:da:d1:98:c5:a7:e2:03:86:cb:30:8b:75:24:9c:80.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@qianfeng01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'qianfeng01'"
and check to make sure that only the key(s) you wanted were added.
一个yes,一个密码
1.4同步时间
[root@qianfeng01 ~]# yum -y install ntpdate.x86_64
[root@qianfeng01 ~]# ntpdate -u ntp1.aliyun.com #下面不是代码,是执行完了之后的显示
26 Mar 09:38:49 ntpdate[12441]: adjust time server 120.25.115.20 offset 0.002909 sec
2、安装软件
在安装之前,我们先安装一下需要的软件环境,不然许多命令执行不了
不在意是在那个路径以及主机名,只要执行命令就可以了
[root@pure ~]# yum install -y curl.x86_64
[root@pure ~]# yum install -y wget.x86_64
[root@pure ~]# yum -y install net-tools.x86_64
[root@pure ~]# yum -y install bzip2.x86_64
[root@pure ~]# yum -y install unzip.x86_64
[root@pure ~]# yum -y install perl
[root@pure ~]# yum -y install zip.x86_64
[root@pure ~]# yum -y install psmisc
[root@pure ~]# yum -y install vim
[root@pure ~]# yum -y install gcc gcc-c++
[root@pure ~]# yum -y install ntpdate
2.1jdk的安装
[root@qianfeng01 ~]# mkdir -p /usr/local/
先把jdk的安装包拖进远程工具
安装包:
链接:https://pan.baidu.com/s/1AmHDfg_KpQL9Gm2sMGrAcQ
提取码:btpv
--来自百度网盘超级会员V2的分享
下面安装
[root@qianfeng01 ~]# tar -zxvf ./jdk-8u221-linux-x64.tar.gz -C /usr/local
[root@qianfeng01 ~]# cd /usr/local
[root@qianfeng01 local]# mv jdk1.8.0_221/ jdk[root@qianfeng01 ~]#vi /etc/profile
在最后按一个o,添加内容,添加完了之后按ase键,按:wq
# JDK environment
JAVA_HOME=/usr/local/jdk
PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
[root@qianfeng01 ~]# source /etc/profile
验证
[root@qianfeng01 ~]# java -version
[root@qianfeng01 ~]# javac
2.3安装mysql
安装包
链接:https://pan.baidu.com/s/1vM0JdJFIGsDzLDq0ODQI9w
提取码:cmzp
--来自百度网盘超级会员V2的分享
把安装包拖到远程工具中(就是root目录下就可以了)
[root@qianfeng01 ~]# tar -xvf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar
[root@qianfeng01 ~]# ll #下面不是代码
mysql-community-client-5.7.28-1.el7.x86_64.rpm
mysql-community-common-5.7.28-1.el7.x86_64.rpm
mysql-community-devel-5.7.28-1.el7.x86_64.rpm
mysql-community-embedded-5.7.28-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.28-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.28-1.el7.x86_64.rpm
mysql-community-libs-5.7.28-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm
mysql-community-server-5.7.28-1.el7.x86_64.rpm
mysql-community-test-5.7.28-1.el7.x86_64.rpm
[root@qianfeng01 ~]# rpm -qa | grep mariadb #下面这行不是代码
mariadb-libs-5.5.64-1.el7.x86_64
[root@qianfeng01 ~]# rpm -e mariadb-libs-5.5.64-1.el7.x86_64 --nodeps
2. 安装mysql-common
[root@qianfeng01 ~]# rpm -ivh mysql-community-common-5.7.28-1.el7.x86_64.rpm
3. 安装mysql-lib
[root@qianfeng01 ~]# rpm -ivh mysql-community-libs-5.7.28-1.el7.x86_64.rpm
4. 安装mysql-client
[root@qianfeng01 ~]# rpm -ivh mysql-community-client-5.7.28-1.el7.x86_64.rpm
5. 安装mysql-server
[root@qianfeng01 ~]# rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm
systemctl start mysqld
systemctl status mysqld
cat /var/log/mysqld.log | grep password #查看初始密码
[root@qianfeng01 ~]# cat /var/log/mysqld.log | grep password
2022-03-26T01:55:07.835729Z 1 [Note] A temporary password is generated for root@localhost: 45aO!haG/F3j
mysql -uroot -p'45aO!haG/F3j'
alter user root@'localhost' identified by '@Mmforu45';
grant all privileges on *.* to root@'%' identified by '@Mmforu45'; #远程授权登录
使用navicat去看看有没有远程成功
特别说明,windows系统需要改一下映射!不然可能后面浏览器无法访问软件的web界面
2.3hadoop的搭建
安装包:
链接:https://pan.baidu.com/s/1TvJ8KHMlZ9p3FLezTkA-uA
提取码:oy1v
--来自百度网盘超级会员V2的分享
因为考虑到后面的spark是2.7版本的,就换了一下。之前的hadoop是2.8.1
tar -zxvf hadoop-2.7.6.tar.gz -C /usr/local/
cd /usr/local
mv hadoop-2.7.6/ hadoop
vim /etc/profile
export HADOOP_HOME=/usr/local/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
source /etc/profile
改配置文件:重中之重
[root@qianfeng01 local]# cd hadoop
[root@qianfeng01 hadoop]# cd etc/hadoop
[root@qianfeng01 hadoop]# vi hadoop-env.sh
就是把那个文件中的JAVA_HOME路径改成本地的实际路径
export JAVA_HOME=/usr/local/jdk
#检验是否安装好环境
hadoop version
下面就是配置文件
cd /usr/local/hadoop/etc/hadoop/
vi slaves
文件添加一个qianfeng01
vi core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<!-- hdfs系统的唯一标识,scheme,ip,port ,内部守护进程的通信地址-->
<name>fs.defaultFS</name>
<value>hdfs://qianfeng01:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
</configuration>
vi hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- namenode守护进程管理的元数据文件fsimage存储的位置-->
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop/hdpdata/dfs/name</value>
</property>
<!-- 确定DFS数据节点应该将其块存储在本地文件系统的何处-->
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop/hdpdata/dfs/data</value>
</property>
<!-- 块的副本数-->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!-- 块的大小(128M),下面的单位是字节-->
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<!-- secondarynamenode守护进程的http地址:主机名和端口号。参考守护进程布局-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>qianfeng01:50090</value>
</property>
<!-- namenode守护进程的http地址:主机名和端口号。参考守护进程布局-->
<property>
<name>dfs.namenode.http-address</name>
<value>qianfeng01:50070</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop/hdpdata/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///usr/local/hadoop/hdpdata/dfs/cname</value>
</property>
<property>
<name>dfs.namenode.checkpoint.edits.dir</name>
<value>file:///usr/local/hadoop/hdpdata/dfs/cname</value>
</property>
</configuration>
vi mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configura