Redhat下安装启动Hadoop

最新推荐文章于 2023-04-03 15:49:52 发布
最新推荐文章于 2023-04-03 15:49:52 发布
阅读量887 收藏
点赞数
分类专栏： Hadoop 文章标签： Hadoop 安装
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/knuuy/article/details/45621477
版权

很久之前的草稿，一直放着没理，后面忙就没继续了，最近被问起来就发出来，写的不怎么好。

1.下载hadoop

http://hadoop.apache.org/releases.html下载地址版本自行选择

也可以在这里下载http://download.csdn.net/detail/jack5261314/6896011 -->hadoop-1.2.1-bin.tar.gz

2.安装

*使用hadoop的前提是安装了sun_jdk，不是open_jdk，openjdk安装软件是需要他的支持，但是是对于开发不适合。

    2.1）卸载open_jdk

[root@localhost ~]# rpm -qa|grep jdk
java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64

[root@localhost ~]# rpm -qa|grep gcj
java-1.4.2-gcj-compat-1.4.2.0-40jpp.115  
libgcj-4.1.2-48.el5  

[root@localhost ~]# yum -y remove java java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
[root@localhost ~]# yum -y remove java java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
[root@localhost ~]# yum -y remove java java-1.4.2-gcj-compat-1.4.2.0-40jpp.115  
[root@localhost ~]# yum -y remove libgcj-4.1.2-48.el5
java -version 没有东西了，Eclipse也没了。

    2.2) 安装sun_jdk

     rpm -ivh jdk-7-linux-x64.rpm 推荐使用rpm包

JDK默认安装在/usr/java中。

2.3)配置sun_jdk

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45

export CLSSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

2.4)安装hadoop

tar -xzf hadoop-1.2.1-bin.tar.gz -C /usr/local  解压

ln -s /usr/local/hadoop-1.2.1 /opt/hadoop  添加方便使用路径的软连接

vim /etc/profile

export HADOOP_HOME=/opt/hadoop
export JAVA_HOME=/usr/java/jdk1.8.0_45
export CLSSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin

source /etc/profile 使配置生效 


<span style="color:#383838;">[root@kong Desktop]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
[root@kong Desktop]# hadoop
</span><span style="color:#ff6666;">Warning: $HADOOP_HOME is deprecated.</span><span style="color:#383838;">

Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  mradmin              run a Map-Reduce admin client
  fsck                 run a DFS filesystem checking utility
  fs                   run a generic filesystem user client
  balancer             run a cluster balancing utility
  oiv                  apply the offline fsimage viewer to an fsimage
  fetchdt              fetch a delegation token from the NameNode
</span>
上面基本安装完成，但是会发现有个warning 不过无关紧要，不妨碍编程。
如果想去掉也可以。
vim /etc/profile 
export HADOOP_HOME=/usr/local/hadoop-1.2.1
export HADOOP_HOME_WARN_SUPPRESS=1 -->解决关键
export JAVA_HOME=/usr/java/jdk1.8.0_45
export CLSSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin
source /etc/profile 

[root@kong Desktop]# hadoop
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  mradmin              run a Map-Reduce admin cli
warning 去掉了

3.简单测试

3.1)计算圆周率

hadoop jar /opt/hadoop/hadoop-examples-1.2.1.jar pi 4 1000

这个作业是建了4个任务来完成圆周率的计算。简单的mapreduce 作业。

[root@localhost home]# hadoop jar /opt/hadoop/hadoop-examples-1.2.1.jar pi 4 1000
Number of Maps  = 4
Samples per Map = 1000
15/05/17 21:49:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job
15/05/17 21:49:45 INFO mapred.FileInputFormat: Total input paths to process : 4
15/05/17 21:49:46 INFO mapred.JobClient: Running job: job_local630184568_0001
15/05/17 21:49:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/05/17 21:49:46 INFO mapred.LocalJobRunner: Starting task: attempt_local630184568_0001_m_000000_0
15/05/17 21:49:46 INFO util.ProcessTree: setsid exited with exit code 0
15/05/17 21:49:46 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@45f1cdad
15/05/17 21:49:46 INFO mapred.MapTask: Processing split: file:/home/PiEstimator_TMP_3_141592654/in/part0:0+118
15/05/17 21:49:46 INFO mapred.MapTask: numReduceTasks: 1
15/05/17 21:49:46 INFO mapred.MapTask: io.sort.mb = 100
15/05/17 21:49:46 INFO mapred.MapTask: data buffer = 79691776/99614720
15/05/17 21:49:46 INFO mapred.MapTask: record buffer = 262144/327680
15/05/17 21:49:46 INFO mapred.MapTask: Starting flush of map output
15/05/17 21:49:46 INFO mapred.MapTask: Finished spill 0
15/05/17 21:49:46 INFO mapred.Task: Task:attempt_local630184568_0001_m_000000_0 is done. And is in the process of commiting
15/05/17 21:49:46 INFO mapred.LocalJobRunner: Generated 1000 samples.
15/05/17 21:49:46 INFO mapred.Task: Task 'attempt_local630184568_0001_m_000000_0' done.
15/05/17 21:49:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local630184568_0001_m_000000_0
15/05/17 21:49:46 INFO mapred.LocalJobRunner: Starting task: attempt_local630184568_0001_m_000001_0
15/05/17 21:49:46 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1abdca6a
15/05/17 21:49:46 INFO mapred.MapTask: Processing split: file:/home/PiEstimator_TMP_3_141592654/in/part1:0+118
15/05/17 21:49:46 INFO mapred.MapTask: numReduceTasks: 1
15/05/17 21:49:46 INFO mapred.MapTask: io.sort.mb = 100
15/05/17 21:49:47 INFO mapred.JobClient:  map 25% reduce 0%
15/05/17 21:49:47 INFO mapred.MapTask: data buffer = 79691776/99614720
15/05/17 21:49:47 INFO mapred.MapTask: record buffer = 262144/327680
15/05/17 21:49:47 INFO mapred.MapTask: Starting flush of map output
15/05/17 21:49:47 INFO mapred.MapTask: Finished spill 0
15/05/17 21:49:47 INFO mapred.Task: Task:attempt_local630184568_0001_m_000001_0 is done. And is in the process of commiting
15/05/17 21:49:47 INFO mapred.LocalJobRunner: Generated 1000 samples.
15/05/17 21:49:47 INFO mapred.Task: Task 'attempt_local630184568_0001_m_000001_0' done.
15/05/17 21:49:47 INFO mapred.LocalJobRunner: Finishing task: attempt_local630184568_0001_m_000001_0
15/05/17 21:49:47 INFO mapred.LocalJobRunner: Starting task: attempt_local630184568_0001_m_000002_0
15/05/17 21:49:47 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@142af1bd
15/05/17 21:49:47 INFO mapred.MapTask: Processing split: file:/home/PiEstimator_TMP_3_141592654/in/part2:0+118
15/05/17 21:49:47 INFO mapred.MapTask: numReduceTasks: 1
15/05/17 21:49:47 INFO mapred.MapTask: io.sort.mb = 100
15/05/17 21:49:48 INFO mapred.JobClient:  map 50% reduce 0%
15/05/17 21:49:49 INFO mapred.MapTask: data buffer = 79691776/99614720
15/05/17 21:49:49 INFO mapred.MapTask: record buffer = 262144/327680
15/05/17 21:49:49 INFO mapred.MapTask: Starting flush of map output
15/05/17 21:49:49 INFO mapred.MapTask: Finished spill 0
15/05/17 21:49:49 INFO mapred.Task: Task:attempt_local630184568_0001_m_000002_0 is done. And is in the process of commiting
15/05/17 21:49:49 INFO mapred.LocalJobRunner: Generated 1000 samples.
15/05/17 21:49:49 INFO mapred.Task: Task 'attempt_local630184568_0001_m_000002_0' done.
15/05/17 21:49:49 INFO mapred.LocalJobRunner: Finishing task: attempt_local630184568_0001_m_000002_0
15/05/17 21:49:49 INFO mapred.LocalJobRunner: Starting task: attempt_local630184568_0001_m_000003_0
15/05/17 21:49:49 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ff0a79d
15/05/17 21:49:49 INFO mapred.MapTask: Processing split: file:/home/PiEstimator_TMP_3_141592654/in/part3:0+118
15/05/17 21:49:49 INFO mapred.MapTask: numReduceTasks: 1
15/05/17 21:49:49 INFO mapred.MapTask: io.sort.mb = 100
15/05/17 21:49:50 INFO mapred.JobClient:  map 75% reduce 0%
15/05/17 21:49:50 INFO mapred.MapTask: data buffer = 79691776/99614720
15/05/17 21:49:50 INFO mapred.MapTask: record buffer = 262144/327680
15/05/17 21:49:50 INFO mapred.MapTask: Starting flush of map output
15/05/17 21:49:50 INFO mapred.MapTask: Finished spill 0
15/05/17 21:49:50 INFO mapred.Task: Task:attempt_local630184568_0001_m_000003_0 is done. And is in the process of commiting
15/05/17 21:49:50 INFO mapred.LocalJobRunner: Generated 1000 samples.
15/05/17 21:49:50 INFO mapred.Task: Task 'attempt_local630184568_0001_m_000003_0' done.
15/05/17 21:49:50 INFO mapred.LocalJobRunner: Finishing task: attempt_local630184568_0001_m_000003_0
15/05/17 21:49:50 INFO mapred.LocalJobRunner: Map task executor complete.
15/05/17 21:49:50 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47588226
15/05/17 21:49:50 INFO mapred.LocalJobRunner: 
15/05/17 21:49:50 INFO mapred.Merger: Merging 4 sorted segments
15/05/17 21:49:50 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 96 bytes
15/05/17 21:49:50 INFO mapred.LocalJobRunner: 
15/05/17 21:49:50 INFO mapred.Task: Task:attempt_local630184568_0001_r_000000_0 is done. And is in the process of commiting
15/05/17 21:49:50 INFO mapred.LocalJobRunner: 
15/05/17 21:49:50 INFO mapred.Task: Task attempt_local630184568_0001_r_000000_0 is allowed to commit now
15/05/17 21:49:50 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local630184568_0001_r_000000_0' to file:/home/PiEstimator_TMP_3_141592654/out
15/05/17 21:49:50 INFO mapred.LocalJobRunner: reduce > reduce
15/05/17 21:49:50 INFO mapred.Task: Task 'attempt_local630184568_0001_r_000000_0' done.
15/05/17 21:49:51 INFO mapred.JobClient:  map 100% reduce 100%
15/05/17 21:49:51 INFO mapred.JobClient: Job complete: job_local630184568_0001
15/05/17 21:49:51 INFO mapred.JobClient: Counters: 21
15/05/17 21:49:51 INFO mapred.JobClient:   Map-Reduce Framework
15/05/17 21:49:51 INFO mapred.JobClient:     Spilled Records=16
15/05/17 21:49:51 INFO mapred.JobClient:     Map output materialized bytes=112
15/05/17 21:49:51 INFO mapred.JobClient:     Reduce input records=8
15/05/17 21:49:51 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
15/05/17 21:49:51 INFO mapred.JobClient:     Map input records=4
15/05/17 21:49:51 INFO mapred.JobClient:     SPLIT_RAW_BYTES=400
15/05/17 21:49:51 INFO mapred.JobClient:     Map output bytes=72
15/05/17 21:49:51 INFO mapred.JobClient:     Reduce shuffle bytes=0
15/05/17 21:49:51 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
15/05/17 21:49:51 INFO mapred.JobClient:     Map input bytes=96
15/05/17 21:49:51 INFO mapred.JobClient:     Reduce input groups=8
15/05/17 21:49:51 INFO mapred.JobClient:     Combine output records=0
15/05/17 21:49:51 INFO mapred.JobClient:     Reduce output records=0
15/05/17 21:49:51 INFO mapred.JobClient:     Map output records=8
15/05/17 21:49:51 INFO mapred.JobClient:     Combine input records=0
15/05/17 21:49:51 INFO mapred.JobClient:     CPU time spent (ms)=0
15/05/17 21:49:51 INFO mapred.JobClient:     Total committed heap usage (bytes)=1698168832
15/05/17 21:49:51 INFO mapred.JobClient:   File Input Format Counters 
15/05/17 21:49:51 INFO mapred.JobClient:     Bytes Read=520
15/05/17 21:49:51 INFO mapred.JobClient:   FileSystemCounters
15/05/17 21:49:51 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=981494
15/05/17 21:49:51 INFO mapred.JobClient:     FILE_BYTES_READ=721813
15/05/17 21:49:51 INFO mapred.JobClient:   File Output Format Counters 
15/05/17 21:49:51 INFO mapred.JobClient:     Bytes Written=109
Job Finished in 5.846 seconds
Estimated value of Pi is 3.14000000000000000000

3.2）注意事项

运行时要保持网络的通常，楼主我曾在一个host-only的虚拟网络中想连接几台虚拟机做个分布式，网络是不通外网的。结果出错，所以要保持网络通畅

centos fedora rehat 一样可以

Knuuy
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
RedHatlinux下安装hadoop
07-23
资源名称：RedHat linux下安装hadoop内容简介： hadoop安装有三种模式：单机模式，伪分布式，完全分布式。本文档主要讲述的是RedHat linux下安装hadoop 0.20.2, 并在windows下远程连接此hadoop,开发调试;感兴趣的...
Redhat_rhel搭建Hadoop完全分布式配置服务
“等你下课”
03-21 736
Redhat_rhel8.0_搭建Hadoop完全分布式配置服务详细 一、简述： 1.1、
Hadoop分布式集群安装部署（Redhat 6.4 64位操作系统）
飘羽的博客
04-03 578
P 地址是 192.168.222.100，slave 对应的 IP 是 192.168.222.100，而自己在做配置时，需要将这两个 IP 地址改为你的 master 和 slave 对应的 IP 地址。选中要配置的虚拟机，然后点击右键--->设置--->网络适配器--->“自定义（U）：特定虚拟网络”--->VMnet*(NAT)模式--->确定。配置VMWare网络模式，虚拟机中点击“编辑”--->“虚拟网络编辑器”，删除其他网络模式，只保留NAT模式，如下图，出现以下信息表示配置成功。
hadoop集群搭建
qq_52159080的博客
03-20 2601
hadoop集群搭建包括了hadoop的安装，以及hadoop的文件配置还有hadoop的节点配置，启动集群
hadoop安装-redhat
zhangxinrun的专栏
01-01 7154
<br />1.基本准备<br />操作系统：redhat 5.2<br />jdk： jdk-6u21-linux-i586-rpm.bin（安装1.6或以上版本，安装步骤请参考其他文）<br />hadoop：hadoop-0.21.0.tar.gz<br />硬件：三台机器，IP配置如下：<br />node1:192.168.14.77<br />node2:192.168.14.155<br />node3:192.168.14.99<br />2.系统配置<br />(1)以root身份打开/e
redHat linux下安装hadoop 0.20.2
03-26
redHat linux下安装hadoop 0.20.2
RedHat linux下安装hadoop 0.docx
10-30
。。。
RedHat下安装hadoop集群
12-03
在redhat下安装hadoop集群，文件中包括设置静态ip，jdk安装，内含截图，详细指导。
Redhat下安装hadoop
11-22
介绍Redhat下安装hadoop具体步骤
linux下的hadoop安装及配置详解
07-13
linux下的hadoop安装及配置详解
haddoop-1.2.1.tar.gz hadoop基础学习之hadoop jar
10-10
haddoop-1.2.1.tar.gz hadoop基础学习之hadoop jar hadoop 学习
hadoop-1.2.1-bin.tar.gz
02-03
hadoop的安装文件,linux下使用
hadoop-1.2.1.tar.gz-百度网盘下载链接
01-05
hadoop-1.2.1.tar.gz做大数据处理，分布式计算的，安装在linux服务器上。
Hadoop安装教程
m0_55853133的博客
10-29 3039
hadoop集群的安装
RedHat6.5上安装Hadoop单机
chuowen140381的博客
07-21 182
版本号：RedHat6.5 JDK1.8 Hadoop2.7.3 hadoop 说明：从版本2开始加入了Yarn这个资源管理器，Yarn并不需要单独安装。只要在机器上安装了JDK就可以直接安装Hadoop，单纯安装Hadoop并不依赖Zookeeper之类的其他东西。 ...
在redhat7中安装hadoop
xxw_sample的博客
04-28 1673
1、安装Linux、JDK、关闭防火墙、配置主机名 解压：tar -zxvf hadoop-2.7.3.tar.gz -C ~/training/ 设置Hadoop的环境变量:  vi ~/.bash_profile HADOOP_HOME=/root/training/hadoop-2.7.3 export HADOOP_HOME PATH=$HADOOP_HOME/bin:$HA...
RedHat6.5上安装Hadoop集群
四叶草的博客
07-05 2460
版本号：RedHat6.5   JDK1.8   Hadoop2.7.3   hadoop  说明：从版本2开始加入了Yarn这个资源管理器，Yarn并不需要单独安装。只要在机器上安装了JDK就可以直接安装Hadoop，单纯安装Hadoop并不依赖Zookeeper之类的其他东西。 基础环境准备：   RedHat6.5安装：RHEL 6.5系统安装配置图解教
Redhat hadoop2.7.2安装笔记
vs412237401的博客
03-29 2794
本次安装是在windows7环境下安装redhat虚拟机进行的，所需要的软件如下： VirtualBox-5.0.16-105871-Win.exe rhel-server-5.4-x86_64-dvd.iso 首先安装虚拟机软件，然后在此基础上安装redhat，安装redhat的时候记得关闭防火墙还有其它的一些服务都disabled掉。首先在windows7上开一个共享目录，将如下软件放入
redHat6.4-x86_64下Hadoop 2.7.2 伪分布式安装
既然选择了-何必在意的专栏
10-21 1643
今天在虚拟机上搭建hadoop伪分布式环境，以做学习之用，特此记录。 环境：win7 64位、VMware 10.0.1、Red Hat Enterprise Linux 6.4 64bit、jdk1.8.0_73; 1、jdk安装步骤在http://blog.csdn.net/yjb7268888/article/details/51766855中有提到，这里不做赘述。 2
vmware下redhat9.0安装教程
最新发布
08-07
虚拟机启动后，按照 Red Hat 9.0 安装向导的提示进行安装。您需要选择语言、键盘布局、磁盘分区等设置。根据提示完成安装过程。 安装完成后，您可以使用 Red Hat 9.0 的图形界面或命令行界面来操作系统。 注意，...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Knuuy

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值