一、windows基础环境准备
1.jdk环境配置
jdk安装后好后配置相关环境变量
2.eclipse环境配置
直接解压就可用
3.eclipse连接hadoop工具包
1.jdk环境配置
jdk安装后好后配置相关环境变量
2.eclipse环境配置
直接解压就可用
3.eclipse连接hadoop工具包
hadoop-eclipse-plugin-2.5.2.jar
百度一下就有,很多
4.下载hadoop-2.5.2.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
将hadoop-2.5.2.tar.gz下载解压到任意目录下 ,一会在eclipse里面会引用这个目录。
5.修改本地hosts文件,加入如下内容:
百度一下就有,很多
4.下载hadoop-2.5.2.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
将hadoop-2.5.2.tar.gz下载解压到任意目录下 ,一会在eclipse里面会引用这个目录。
5.修改本地hosts文件,加入如下内容:
192.168.153.128 master--这个是你的虚拟机的ip地址,后面的是主机名,可以随意写。
二、eclipse环境配置
1.将下载的hadoop-eclipse-plugin-2.5.2.jar拷贝至eclipse的plugins目录下,然后重启eclipse
2.打开菜单Window--Preference--Hadoop Map/Reduce进行配置,如下图所示:
这里引用上面解压的hadoop安装目录。
3.显示Hadoop连接配置窗口:Window--Show View--Other-MapReduce Tools,如下图所示:
二、eclipse环境配置
1.将下载的hadoop-eclipse-plugin-2.5.2.jar拷贝至eclipse的plugins目录下,然后重启eclipse
2.打开菜单Window--Preference--Hadoop Map/Reduce进行配置,如下图所示:
这里引用上面解压的hadoop安装目录。
3.显示Hadoop连接配置窗口:Window--Show View--Other-MapReduce Tools,如下图所示:
4.添加配置连接Hadoop,如下图所示:
host可以直接写hadoop启动的虚拟机的ip也可以配置host里面配置的主机名,端口是hadoop的启动端口,我的是9000,在core-site.xml文件设置的。
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
查看是否连接成功,能看到如下信息,则表示连接成功:
三、hadoop集群环境添加测试文件
启动hadoop集群,然后添加input目录
使用dfs命令创建input目录
hadoop fs -mkdir /user/自己的用户名/input
四、创建一个Map/Reduce Project
1.新建项目 File--New--Other--Map/Reduce Project 命名随意,
然后创建类org.apache.hadoop.examples.WordCount,从hadoop-2.5.2-src中拷贝覆盖
路径是:(hadoop-2.5.2-src\hadoop-mapreduce-project\hadoop-mapreduce-examples\src\main\java\org\apache\hadoop\examples\WordCount.java)
源码百度一下也有
hadoop fs -mkdir /user/自己的用户名/input
四、创建一个Map/Reduce Project
1.新建项目 File--New--Other--Map/Reduce Project 命名随意,
然后创建类org.apache.hadoop.examples.WordCount,从hadoop-2.5.2-src中拷贝覆盖
路径是:(hadoop-2.5.2-src\hadoop-mapreduce-project\hadoop-mapreduce-examples\src\main\java\org\apache\hadoop\examples\WordCount.java)
源码百度一下也有
2、win下运行mapreduce需要winutils.exe和hadoop.dll。
百度下载之后把
winutils.exe放到hadoop安装目录下的bin目录下,hadoop.dll放到C盘下system32目录下就行了。
3.创建log4j.properties文件
在src目录下创建log4j.properties文件,内容如下:
log4j.rootLogger=CONSOLE,FILE
log4j.addivity.org.apache=true
# \u5E94\u7528\u4E8E\u63A7\u5236\u53F0
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.Target=System.out
log4j.appender.CONSOLE.Encoding=UTF-8
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
# \u6BCF\u5929\u65B0\u5EFA\u65E5\u5FD7
log4j.appender.A1=org.apache.log4j.DailyRollingFileAppender
log4j.appender.A1.File=C:/log4j/log
log4j.appender.A1.Encoding=UTF-8
log4j.appender.A1.Threshold=DEBUG
log4j.appender.A1.DatePattern='.'yyyy-MM-dd
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L : %m%n
#\u5E94\u7528\u4E8E\u6587\u4EF6
log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.File=C:/log4j/file.log
log4j.appender.FILE.Append=false
log4j.appender.FILE.Encoding=UTF-8
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
# \u5E94\u7528\u4E8E\u6587\u4EF6\u56DE\u6EDA
log4j.appender.ROLLING_FILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLING_FILE.Threshold=ERROR
log4j.appender.ROLLING_FILE.File=rolling.log
log4j.appender.ROLLING_FILE.Append=true
log4j.appender.CONSOLE_FILE.Encoding=UTF-8
log4j.appender.ROLLING_FILE.MaxFileSize=10KB
log4j.appender.ROLLING_FILE.MaxBackupIndex=1
log4j.appender.ROLLING_FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLING_FILE.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
#\u81EA\u5B9A\u4E49Appender
log4j.appender.im = net.cybercorlin.util.logger.appender.IMAppender
log4j.appender.im.host = mail.cybercorlin.net
log4j.appender.im.username = username
log4j.appender.im.password = password
log4j.appender.im.recipient = yyflyons@163.com
log4j.appender.im.layout=org.apache.log4j.PatternLayout
log4j.appender.im.layout.ConversionPattern =[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
#\u5E94\u7528\u4E8Esocket
log4j.appender.SOCKET=org.apache.log4j.RollingFileAppender
log4j.appender.SOCKET.RemoteHost=localhost
log4j.appender.SOCKET.Port=5001
log4j.appender.SOCKET.LocationInfo=true
# Set up for Log Facter 5
log4j.appender.SOCKET.layout=org.apache.log4j.PatternLayout
log4j.appender.SOCET.layout.ConversionPattern=[start]%d{DATE}[DATE]%n%p[PRIORITY]%n%x[NDC]%n%t[THREAD]%n%c[CATEGORY]%n%m[MESSAGE]%n%n
# Log Factor 5 Appender
log4j.appender.LF5_APPENDER=org.apache.log4j.lf5.LF5Appender
log4j.appender.LF5_APPENDER.MaxNumberOfRecords=2000
# \u53D1\u9001\u65E5\u5FD7\u7ED9\u90AE\u4EF6
log4j.appender.MAIL=org.apache.log4j.net.SMTPAppender
log4j.appender.MAIL.Threshold=FATAL
log4j.appender.MAIL.BufferSize=10
log4j.appender.MAIL.From=yyflyons@163.com
log4j.appender.MAIL.SMTPHost=www.wusetu.com
log4j.appender.MAIL.Subject=Log4J Message
log4j.appender.MAIL.To=yyflyons@126.com
log4j.appender.MAIL.layout=org.apache.log4j.PatternLayout
log4j.appender.MAIL.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
3.创建log4j.properties文件
在src目录下创建log4j.properties文件,内容如下:
log4j.rootLogger=CONSOLE,FILE
log4j.addivity.org.apache=true
# \u5E94\u7528\u4E8E\u63A7\u5236\u53F0
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.Target=System.out
log4j.appender.CONSOLE.Encoding=UTF-8
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
# \u6BCF\u5929\u65B0\u5EFA\u65E5\u5FD7
log4j.appender.A1=org.apache.log4j.DailyRollingFileAppender
log4j.appender.A1.File=C:/log4j/log
log4j.appender.A1.Encoding=UTF-8
log4j.appender.A1.Threshold=DEBUG
log4j.appender.A1.DatePattern='.'yyyy-MM-dd
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L : %m%n
#\u5E94\u7528\u4E8E\u6587\u4EF6
log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.File=C:/log4j/file.log
log4j.appender.FILE.Append=false
log4j.appender.FILE.Encoding=UTF-8
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
# \u5E94\u7528\u4E8E\u6587\u4EF6\u56DE\u6EDA
log4j.appender.ROLLING_FILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLING_FILE.Threshold=ERROR
log4j.appender.ROLLING_FILE.File=rolling.log
log4j.appender.ROLLING_FILE.Append=true
log4j.appender.CONSOLE_FILE.Encoding=UTF-8
log4j.appender.ROLLING_FILE.MaxFileSize=10KB
log4j.appender.ROLLING_FILE.MaxBackupIndex=1
log4j.appender.ROLLING_FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLING_FILE.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
#\u81EA\u5B9A\u4E49Appender
log4j.appender.im = net.cybercorlin.util.logger.appender.IMAppender
log4j.appender.im.host = mail.cybercorlin.net
log4j.appender.im.username = username
log4j.appender.im.password = password
log4j.appender.im.recipient = yyflyons@163.com
log4j.appender.im.layout=org.apache.log4j.PatternLayout
log4j.appender.im.layout.ConversionPattern =[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
#\u5E94\u7528\u4E8Esocket
log4j.appender.SOCKET=org.apache.log4j.RollingFileAppender
log4j.appender.SOCKET.RemoteHost=localhost
log4j.appender.SOCKET.Port=5001
log4j.appender.SOCKET.LocationInfo=true
# Set up for Log Facter 5
log4j.appender.SOCKET.layout=org.apache.log4j.PatternLayout
log4j.appender.SOCET.layout.ConversionPattern=[start]%d{DATE}[DATE]%n%p[PRIORITY]%n%x[NDC]%n%t[THREAD]%n%c[CATEGORY]%n%m[MESSAGE]%n%n
# Log Factor 5 Appender
log4j.appender.LF5_APPENDER=org.apache.log4j.lf5.LF5Appender
log4j.appender.LF5_APPENDER.MaxNumberOfRecords=2000
# \u53D1\u9001\u65E5\u5FD7\u7ED9\u90AE\u4EF6
log4j.appender.MAIL=org.apache.log4j.net.SMTPAppender
log4j.appender.MAIL.Threshold=FATAL
log4j.appender.MAIL.BufferSize=10
log4j.appender.MAIL.From=yyflyons@163.com
log4j.appender.MAIL.SMTPHost=www.wusetu.com
log4j.appender.MAIL.Subject=Log4J Message
log4j.appender.MAIL.To=yyflyons@126.com
log4j.appender.MAIL.layout=org.apache.log4j.PatternLayout
log4j.appender.MAIL.layout.ConversionPattern=[framework] %d - %c -%-4r [%t] %-5p %c %x - %m%n
4、然后需要在run as config的时候配置一下你的hdfs文件目录,一个是hadoop里面创建的input,output一定要不存在才行。
一个简单的mapreduce程序就OK了。