hadoop hadoop install (1)

最新推荐文章于 2024-04-25 13:49:06 发布

weixin_30651273

最新推荐文章于 2024-04-25 13:49:06 发布

阅读量219

点赞数

文章标签：大数据运维开发工具

原文链接：http://www.cnblogs.com/lianghong881018/p/10295399.html

版权

vmuser@vmuser-VirtualBox:~$ sudo useradd -m hadoop -s /bin/bash
[sudo] vmuser 的密码：
vmuser@vmuser-VirtualBox:~$ sudo passwd hadoop
输入新的 UNIX 密码：
重新输入新的 UNIX 密码：
passwd：已成功更新密码
vmuser@vmuser-VirtualBox:~$ sudo adduser hadoop sudo
正在添加用户"hadoop"到"sudo"组...
正在将用户“hadoop”加入到“sudo”组中
完成。
vmuser@vmuser-VirtualBox:~$ sudo apt-get install openssh-server
正在读取软件包列表... 完成
正在分析软件包的依赖关系树
正在读取状态信息... 完成
将会同时安装下列软件：
ncurses-term openssh-client openssh-sftp-server ssh-import-id
建议安装：
keychain libpam-ssh monkeysphere ssh-askpass molly-guard rssh
下列【新】软件包将被安装：
ncurses-term openssh-server openssh-sftp-server ssh-import-id
下列软件包将被升级：
openssh-client
升级了 1 个软件包，新安装了 4 个软件包，要卸载 0 个软件包，有 386 个软件包未被升级。
需要下载 637 kB/1,248 kB 的归档。
解压缩后会消耗 5,316 kB 的额外空间。
您希望继续执行吗？ [Y/n] y
获取:1 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 ncurses-term all 6.1-1ubuntu1.18.04 [248 kB]
获取:2 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 openssh-sftp-server amd64 1:7.6p1-4ubuntu0.1 [45.5 kB]
获取:3 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 openssh-server amd64 1:7.6p1-4ubuntu0.1 [332 kB]
获取:4 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 ssh-import-id all 5.7-0ubuntu1.1 [10.9 kB]
已下载 637 kB，耗时 0秒 (3,215 kB/s)
正在预设定软件包 ...
(正在读取数据库 ... 系统当前共安装有 140635 个文件和目录。)
正准备解包 .../openssh-client_1%3a7.6p1-4ubuntu0.1_amd64.deb ...
正在将 openssh-client (1:7.6p1-4ubuntu0.1) 解包到 (1:7.6p1-4) 上 ...
正在选中未选择的软件包 ncurses-term。
正准备解包 .../ncurses-term_6.1-1ubuntu1.18.04_all.deb ...
正在解包 ncurses-term (6.1-1ubuntu1.18.04) ...
正在选中未选择的软件包 openssh-sftp-server。
正准备解包 .../openssh-sftp-server_1%3a7.6p1-4ubuntu0.1_amd64.deb ...
正在解包 openssh-sftp-server (1:7.6p1-4ubuntu0.1) ...
正在选中未选择的软件包 openssh-server。
正准备解包 .../openssh-server_1%3a7.6p1-4ubuntu0.1_amd64.deb ...
正在解包 openssh-server (1:7.6p1-4ubuntu0.1) ...
正在选中未选择的软件包 ssh-import-id。
正准备解包 .../ssh-import-id_5.7-0ubuntu1.1_all.deb ...
正在解包 ssh-import-id (5.7-0ubuntu1.1) ...
正在设置 ncurses-term (6.1-1ubuntu1.18.04) ...
正在处理用于 ufw (0.35-5) 的触发器 ...
正在处理用于 ureadahead (0.100.0-20) 的触发器 ...
正在处理用于 systemd (237-3ubuntu10.3) 的触发器 ...
正在处理用于 man-db (2.8.3-2) 的触发器 ...
正在设置 openssh-client (1:7.6p1-4ubuntu0.1) ...
正在设置 ssh-import-id (5.7-0ubuntu1.1) ...
正在设置 openssh-sftp-server (1:7.6p1-4ubuntu0.1) ...
正在设置 openssh-server (1:7.6p1-4ubuntu0.1) ...

Creating config file /etc/ssh/sshd_config with new version
Creating SSH2 RSA key; this may take some time ...
2048 SHA256:+UzKAYam4shz8eFVxXfjzNthoobWMAQk/ncyjqOwnAY root@vmuser-VirtualBox (RSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:Y2lX2D5tkCgVFTgJ6inqzO0v5o07tgEhD2sgea8w24M root@vmuser-VirtualBox (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:ZcP+F60Vdlb/ddaHog4J/KhZFAgufj4IRyxnnqrUtgM root@vmuser-VirtualBox (ED25519)
Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service.
Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service.
正在处理用于 ureadahead (0.100.0-20) 的触发器 ...
正在处理用于 systemd (237-3ubuntu10.3) 的触发器 ...
hadoop@vmuser-VirtualBox:/home/vmuser$ sudo su hadoop
[sudo] hadoop 的密码：
hadoop@vmuser-VirtualBox:/home/vmuser$ cd ~
hadoop@vmuser-VirtualBox:~$ pwd
/home/hadoop
hadoop@vmuser-VirtualBox:~$ ls
examples.desktop
hadoop@vmuser-VirtualBox:~$ ls
examples.desktop
hadoop@vmuser-VirtualBox:~$ mkdir .ssh
hadoop@vmuser-VirtualBox:~$ cd .ssh/
hadoop@vmuser-VirtualBox:~/.ssh$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:VjU4i/ypqi/K4xRQXmYN1wX2JyE0o23ehNR3W+4PSf8 hadoop@vmuser-VirtualBox
The key's randomart image is:
+---[RSA 2048]----+
| . =o.oO+o.o    |
| o + ..=.*+o... .|
|. .   ..+.=+.. + |
| .     oo+.o o .|
| .     S... . + |
|   .   . o   o o|
| .      .     .o|
| o. .   .       E|
| .+o.+o.         |
+----[SHA256]-----+
hadoop@vmuser-VirtualBox:~/.ssh$ cat id_rsa.pub >> authorized_keys
hadoop@vmuser-VirtualBox:~/.ssh$ sudo apt install openjdk-8-jdk openjdk-8-jre
正在读取软件包列表... 完成
正在分析软件包的依赖关系树
正在读取状态信息... 完成
update-alternatives: 使用 /usr/lib/jvm/java-8-openjdk-amd64/bin/jconsole 来在自动模式中提供 /usr/bin/jconsole (jconsole)
正在处理用于 libc-bin (2.27-3ubuntu1) 的触发器 ...
正在处理用于 ca-certificates (20180409) 的触发器 ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
hadoop@vmuser-VirtualBox:~/.ssh$ dpkg -L openjdk-8-jdk
/.
/usr
/usr/lib
/usr/lib/jvm
/usr/lib/jvm/java-8-openjdk-amd64
/usr/lib/jvm/java-8-openjdk-amd64/bin
/usr/lib/jvm/java-8-openjdk-amd64/bin/appletviewer
/usr/lib/jvm/java-8-openjdk-amd64/bin/jconsole
/usr/lib/jvm/java-8-openjdk-amd64/include
/usr/lib/jvm/java-8-openjdk-amd64/include/jawt.h
/usr/lib/jvm/java-8-openjdk-amd64/include/linux
/usr/lib/jvm/java-8-openjdk-amd64/include/linux/jawt_md.h
/usr/lib/jvm/java-8-openjdk-amd64/lib
/usr/lib/jvm/java-8-openjdk-amd64/lib/jconsole.jar
/usr/lib/jvm/java-8-openjdk-amd64/man
/usr/lib/jvm/java-8-openjdk-amd64/man/ja_JP.UTF-8
/usr/lib/jvm/java-8-openjdk-amd64/man/ja_JP.UTF-8/man1
/usr/lib/jvm/java-8-openjdk-amd64/man/ja_JP.UTF-8/man1/appletviewer.1.gz
/usr/lib/jvm/java-8-openjdk-amd64/man/ja_JP.UTF-8/man1/jconsole.1.gz
/usr/lib/jvm/java-8-openjdk-amd64/man/man1
/usr/lib/jvm/java-8-openjdk-amd64/man/man1/appletviewer.1.gz
/usr/lib/jvm/java-8-openjdk-amd64/man/man1/jconsole.1.gz
/usr/share
/usr/share/doc
/usr/share/doc/openjdk-8-jre-headless
/usr/share/doc/openjdk-8-jre-headless/test-amd64
/usr/share/doc/openjdk-8-jre-headless/test-amd64/check-hotspot-hotspot.log.gz
/usr/share/doc/openjdk-8-jre-headless/test-amd64/check-jdk-hotspot.log.gz
/usr/share/doc/openjdk-8-jre-headless/test-amd64/check-langtools-hotspot.log.gz
/usr/share/doc/openjdk-8-jre-headless/test-amd64/failed_tests-hotspot.tar.gz
/usr/share/doc/openjdk-8-jre-headless/test-amd64/jtreg-summary-hotspot.log.gz
/usr/share/doc/openjdk-8-jre-headless/test-amd64/jtreport-hotspot.tar.gz
/usr/share/doc/openjdk-8-jdk
hadoop@vmuser-VirtualBox:~/.ssh$ which javac
/usr/bin/javac
hadoop@vmuser-VirtualBox:~/.ssh$ ls -al /usr/bin/javac
lrwxrwxrwx 1 root root 23 1月 20 13:29 /usr/bin/javac -> /etc/alternatives/javac
hadoop@vmuser-VirtualBox:~/.ssh$ ls -al /etc/alternatives/javac
lrwxrwxrwx 1 root root 43 1月 20 13:29 /etc/alternatives/javac -> /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
hadoop@vmuser-VirtualBox:~/.ssh$

add in ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

hadoop@vmuser-VirtualBox:~/.ssh$ source ~/.bashrc
hadoop@vmuser-VirtualBox:~/.ssh$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64
hadoop@vmuser-VirtualBox:~/.ssh$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
hadoop@vmuser-VirtualBox:~/.ssh$ $JAVA_HOME/bin/java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

hadoop@vmuser-VirtualBox:~/.ssh$ wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.7.6/hadoop-2.7.6.tar.gz
--2019-01-20 13:38:58-- http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.7.6/hadoop-2.7.6.tar.gz
正在解析主机 mirrors.hust.edu.cn (mirrors.hust.edu.cn)... 202.114.18.160
正在连接 mirrors.hust.edu.cn (mirrors.hust.edu.cn)|202.114.18.160|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 216745683 (207M) [application/octet-stream]
正在保存至: “hadoop-2.7.6.tar.gz”

hadoop-2.7.6.tar.gz 100%[===================>] 206.70M 8.81MB/s 用时 30s

2019-01-20 13:39:28 (6.96 MB/s) - 已保存 “hadoop-2.7.6.tar.gz” [216745683/216745683])

hadoop@vmuser-VirtualBox:/usr/local$ sudo tar zvxf hadoop-2.7.6.tar.gz -C /usr/local
hadoop@vmuser-VirtualBox:~/.ssh$ cd /usr/local/
hadoop@vmuser-VirtualBox:/usr/local$ ls
bin etc games hadoop-2.7.6 include lib man sbin share src
hadoop@vmuser-VirtualBox:/usr/local$ sudo mv ./hadoop-2.7.6 ./hadoop
hadoop@vmuser-VirtualBox:/usr/local$ sudo chown -R hadoop ./hadoop

hadoop@vmuser-VirtualBox:/usr/local$ vim hadoop/etc/hadoop/core-site.xml
hadoop@vmuser-VirtualBox:/usr/local$ vim hadoop/etc/hadoop/hdfs-site.xml
hadoop@vmuser-VirtualBox:/usr/local$ cd hadoop/
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ ./bin/hdfs namenode -format
19/01/20 13:46:18 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
19/01/20 13:46:18 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.
19/01/20 13:46:18 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
19/01/20 13:46:18 INFO util.ExitUtil: Exiting with status 0
19/01/20 13:46:18 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at vmuser-VirtualBox/127.0.1.1
************************************************************/
adoop@vmuser-VirtualBox:/usr/local/hadoop$ ./sbin/start-dfs.sh
Starting namenodes on [localhost]
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:Y2lX2D5tkCgVFTgJ6inqzO0v5o07tgEhD2sgea8w24M.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ vim etc/hadoop/hadoop-env.sh +25

etc/hadoop/hadoop-env.sh:25:export JAVA_HOME=${JAVA_HOME}

etc/hadoop/hadoop-env.sh:25:export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

hadoop@vmuser-VirtualBox:/usr/local/hadoop$ ./sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-vmuser-VirtualBox.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-vmuser-VirtualBox.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:Y2lX2D5tkCgVFTgJ6inqzO0v5o07tgEhD2sgea8w24M.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-vmuser-VirtualBox.out
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ jps
8835 NameNode
9365 Jps
9241 SecondaryNameNode
9019 DataNode

access localhost:50070

hadoop@vmuser-VirtualBox:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir -p /user/hadoop
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ ./bin/hdfs dfs -put ./etc/hadoop/capacity-scheduler.xml input
put: Cannot create file/user/hadoop/input/capacity-scheduler.xml._COPYING_. Name node is in safe mode.
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ jps
15058 DataNode
15574 Jps
14126 NameNode
14542 SecondaryNameNode
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ ./bin/hdfs dfs -put ./etc/hadoop/capacity-scheduler.xml input
put: Cannot create file/user/hadoop/input/capacity-scheduler.xml._COPYING_. Name node is in safe mode.
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ bin/hadoop dfsadmin -safemode leave
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Safe mode is OFF
hadoop@vmuser-VirtualBox:/usr/local/hadoop$ ./bin/hdfs dfs -put ./etc/hadoop/capacity-scheduler.xml input