Hadoop配置运行java程序

[size=large]Ubuntu Hadoop环境搭建：

第一部分：下载JDK和HADOOP安装文件：
1. 下载jdk：
查看操作系统是32位还是64位，输入命令：sudo uname --m 如果显示 i586 则是32位，如果显示x86_64则是64位，得到此信息后，下载相应版本的JDK。下载网址是：http://www.oracle.com/technetwork/java/javase/downloads/
2. 下载hadoop：
网址是：http://hadoop.apache.org/releases.html

第二部分：安装java：
解压文件：tar -zxvf jdk-8u11-linux-x64.tar.gz 这时得到一个文件夹jdk1.8.0_11 （我选择的目录是/opt/jdk1.8.0_11，当然可以任意选择安装目录）
修改~/.bashrc 在该文件末尾加入如下语句：
export JAVA_HOME=/opt/jdk1.8.0_11
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
执行source ~/.bashrc立即生效该配置。注意只需要修改黑体部分也就是java的安装目录。
配置默认的JDK版本：
sudo update-alternatives --install /usr/bin/java java /opt/jdk1.8.0_11/bin/java 300
sudo update-alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_11/bin/javac 300
sudo update-alternatives --install /usr/bin/javap javap /opt/jdk1.8.0_11/bin/javap 300
sudo update-alternatives --install /usr/bin/javadoc javadoc /opt/jdk1.8.0_11/bin/javadoc 300
同样的，只需要修改黑体部分的java的安装目录即可。
测试JAVA安装成功：输入 java 和 java -version测试，出现如下的结果表明安装过程正确并结束：

root@ubuntu:/opt/jdk1.8.0_11# java -version
java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)

第三部分：安装hadoop：

解压hadoop安装包到/opt/下，修改~/.bashrc添加hadoop的相关环境变量：
vi ~/.bashrc

export HADOOP_HOME=/opt/hadoop-1.2.1
export HADOOP_CLASSPATH=$HADOOP_HOME/build/classes
[color=red](说明：将要跑的class,jar包等程序一定要在这个目录下放一份，否则报错：Error: Could not find or load main class com.hadoop.mn.MaxTemperature)[/color]
export PATH=$HADOOP_HOME/bin:${PATH}
黑体部分可以自行定义。

修改$HADOOP_HOME/conf/hadoop-env.sh，添加 export JAVA_HOME=/opt/jdk1.8.0_11


工具一：使得UE能够FTP连接root用户：
在/etc/ftpusers文件中保存了，系统不允许FTP访问的用户列表，在这个列表中将root删掉即可。
# /etc/ftpusers: list of users disallowed FTP access. See ftpusers(5).

daemon
bin
sys
sync
games
man
lp
mail
news
uucp
nobody


工具二：配置和安装SSH
1. 输入命令ssh-keygen，其余的都回车就可以：
root@ubuntu:/# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
be:2c:48:43:9d:c9:67:50:a9:2c:0a:a9:fa:b2:cb:c0 root@ubuntu
The key's randomart image is:
+--[ RSA 2048]----+
| ... |
| . . |
| . + = |
|o o B o |
|.. o . oS |
|o . o . |
|oE . o . |
|= . .. . |
|o*. .o |
+-----------------+

2. 首先cd到目录/root，执行命令：cp .ssh/id_rsa.pub .ssh/authorized_keys
3. 测试 SSH安装， 输入命令 ssh localhost。

工具三：关于vi的实用命令：
/pattern<Enter> ：向下查找pattern匹配字符串
?pattern<Enter>：向上查找pattern匹配字符串
使用了查找命令之后，使用如下两个键快速查找：
n：按照同一方向继续查找
N：按照反方向查找

输出结果：
root@ubuntu:/opt/hadoop-1.2.1/build/classes# hadoop com/hadoop/mn/NewMaxTemperature 1901 output
14/10/23 02:00:05 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/10/23 02:00:05 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/10/23 02:00:05 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/10/23 02:00:05 INFO input.FileInputFormat: Total input paths to process : 1
14/10/23 02:00:05 WARN snappy.LoadSnappy: Snappy native library not loaded
14/10/23 02:00:06 INFO mapred.JobClient: Running job: job_local693177873_0001
14/10/23 02:00:06 INFO mapred.LocalJobRunner: Waiting for map tasks
14/10/23 02:00:06 INFO mapred.LocalJobRunner: Starting task: attempt_local693177873_0001_m_000000_0
14/10/23 02:00:06 INFO util.ProcessTree: setsid exited with exit code 0
14/10/23 02:00:06 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2cc3897e
14/10/23 02:00:06 INFO mapred.MapTask: Processing split: file:/opt/hadoop-1.2.1/build/classes/1901:0+888190
14/10/23 02:00:06 INFO mapred.MapTask: io.sort.mb = 100
14/10/23 02:00:06 INFO mapred.MapTask: data buffer = 79691776/99614720
14/10/23 02:00:06 INFO mapred.MapTask: record buffer = 262144/327680
14/10/23 02:00:07 INFO mapred.JobClient: map 0% reduce 0%
14/10/23 02:00:07 INFO mapred.MapTask: Starting flush of map output
14/10/23 02:00:07 INFO mapred.Task: Task:attempt_local693177873_0001_m_000000_0 is done. And is in the process of commiting
14/10/23 02:00:07 INFO mapred.LocalJobRunner:
14/10/23 02:00:07 INFO mapred.Task: Task 'attempt_local693177873_0001_m_000000_0' done.
14/10/23 02:00:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local693177873_0001_m_000000_0
14/10/23 02:00:07 INFO mapred.LocalJobRunner: Map task executor complete.
14/10/23 02:00:07 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@67d3712a
14/10/23 02:00:07 INFO mapred.LocalJobRunner:
14/10/23 02:00:07 INFO mapred.Merger: Merging 1 sorted segments
14/10/23 02:00:07 INFO mapred.Merger: Down to the last merge-pass, with 0 segments left of total size: 0 bytes
14/10/23 02:00:07 INFO mapred.LocalJobRunner:
14/10/23 02:00:07 INFO mapred.Task: Task:attempt_local693177873_0001_r_000000_0 is done. And is in the process of commiting
14/10/23 02:00:07 INFO mapred.LocalJobRunner:
14/10/23 02:00:07 INFO mapred.Task: Task attempt_local693177873_0001_r_000000_0 is allowed to commit now
14/10/23 02:00:07 INFO output.FileOutputCommitter: Saved output of task 'attempt_local693177873_0001_r_000000_0' to output
14/10/23 02:00:07 INFO mapred.LocalJobRunner: reduce > reduce
14/10/23 02:00:07 INFO mapred.Task: Task 'attempt_local693177873_0001_r_000000_0' done.
14/10/23 02:00:08 INFO mapred.JobClient: map 100% reduce 100%
14/10/23 02:00:08 INFO mapred.JobClient: Job complete: job_local693177873_0001
14/10/23 02:00:08 INFO mapred.JobClient: Counters: 20
14/10/23 02:00:08 INFO mapred.JobClient: Map-Reduce Framework
14/10/23 02:00:08 INFO mapred.JobClient: Spilled Records=0
14/10/23 02:00:08 INFO mapred.JobClient: Map output materialized bytes=6
14/10/23 02:00:08 INFO mapred.JobClient: Reduce input records=0
14/10/23 02:00:08 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
14/10/23 02:00:08 INFO mapred.JobClient: Map input records=6565
14/10/23 02:00:08 INFO mapred.JobClient: SPLIT_RAW_BYTES=106
14/10/23 02:00:08 INFO mapred.JobClient: Map output bytes=0
14/10/23 02:00:08 INFO mapred.JobClient: Reduce shuffle bytes=0
14/10/23 02:00:08 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
14/10/23 02:00:08 INFO mapred.JobClient: Reduce input groups=0
14/10/23 02:00:08 INFO mapred.JobClient: Combine output records=0
14/10/23 02:00:08 INFO mapred.JobClient: Reduce output records=0
14/10/23 02:00:08 INFO mapred.JobClient: Map output records=0
14/10/23 02:00:08 INFO mapred.JobClient: Combine input records=0
14/10/23 02:00:08 INFO mapred.JobClient: CPU time spent (ms)=0
14/10/23 02:00:08 INFO mapred.JobClient: Total committed heap usage (bytes)=505110528
14/10/23 02:00:08 INFO mapred.JobClient: File Input Format Counters
14/10/23 02:00:08 INFO mapred.JobClient: Bytes Read=888190
14/10/23 02:00:08 INFO mapred.JobClient: FileSystemCounters
14/10/23 02:00:08 INFO mapred.JobClient: FILE_BYTES_WRITTEN=100924
14/10/23 02:00:08 INFO mapred.JobClient: FILE_BYTES_READ=1776712
14/10/23 02:00:08 INFO mapred.JobClient: File Output Format Counters
14/10/23 02:00:08 INFO mapred.JobClient: Bytes Written=8
[/size]