参考我的前一篇文章(ubuntu + hadoop2.5.2分布式环境配置
http://www.cnblogs.com/huligong1234/p/4136331.html)
但本人配置时还修改了如下内容(由于你的环境和我的可能不一致,可以在后面出现相关问题后再进行修改):
a.在master节点上(ubuntu-V01)修改hdfs-site.xml加上以下内容
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
旨在取消权限检查,原因是为了解决我在windows机器上配置eclipse连接hadoop服务器时,配置map/reduce连接后报以下错误,org.apache.hadoop.security.AccessControlException: Permission denied:
b.同样在master节点上(ubuntu-V01)修改hdfs-site.xml加上以下内容
<property>
<name>dfs.web.ugi</name>
<value>jack,supergroup</value>
</property>
原因是运行时,报如下错误 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user jack
应该是我的windows的用户名为jack,无访问权限
更多权限配置可参看官方说明文档:
HDFS权限管理用户指南 http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_permissions_guide.html
配置修改完后重启hadoop集群:
hadoop@ubuntu-V01:~/data$./sbin/stop-dfs.sh
hadoop@ubuntu-V01:~/data$./sbin/stop-yarn.sh
hadoop@ubuntu-V01:~/data$./sbin/start-dfs.sh
hadoop@ubuntu-V01:~/data$./sbin/start-yarn.sh
二.windows基础环境准备
windows7(x64),jdk,ant,eclipse,hadoop
1.jdk环境配置
jdk-6u26-windows-i586.exe安装后好后配置相关JAVA_HOME环境变量,并将bin目录配置到path
2.eclipse环境配置
eclipse-standard-luna-SR1-win32.zip解压到D:\eclipse\目录下并命名eclipse-hadoop
下载地址: http://developer.eclipsesource.com/technology/epp/luna/eclipse-standard-luna-SR1-win32.zip
3.ant环境配置
apache-ant-1.9.4-bin.zip解压到D:\apache\目录下,配置环境变量ANT_HOME,并将bin目录配置到path
下载地址: http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.zip
4.下载hadoop-2.5.2.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
5.下载hadoop-2.5.2-src.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2-src.tar.gz
6.下载hadoop2x-eclipse-plugin
https://github.com/winghc/hadoop2x-eclipse-plugin
7.下载hadoop-common-2.2.0-bin
https://github.com/srccodes/hadoop-common-2.2.0-bin
分别将hadoop-2.5.2.tar.gz、hadoop-2.5.2-src.tar.gz、hadoop2x-eclipse-plugin、hadoop-common-2.2.0-bin下载解压到F:\hadoop\目录下
8.修改本地hosts文件,加入如下内容:
但本人配置时还修改了如下内容(由于你的环境和我的可能不一致,可以在后面出现相关问题后再进行修改):
a.在master节点上(ubuntu-V01)修改hdfs-site.xml加上以下内容
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
旨在取消权限检查,原因是为了解决我在windows机器上配置eclipse连接hadoop服务器时,配置map/reduce连接后报以下错误,org.apache.hadoop.security.AccessControlException: Permission denied:
b.同样在master节点上(ubuntu-V01)修改hdfs-site.xml加上以下内容
<property>
<name>dfs.web.ugi</name>
<value>jack,supergroup</value>
</property>
原因是运行时,报如下错误 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user jack
应该是我的windows的用户名为jack,无访问权限
更多权限配置可参看官方说明文档:
HDFS权限管理用户指南 http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_permissions_guide.html
配置修改完后重启hadoop集群:
hadoop@ubuntu-V01:~/data$./sbin/stop-dfs.sh
hadoop@ubuntu-V01:~/data$./sbin/stop-yarn.sh
hadoop@ubuntu-V01:~/data$./sbin/start-dfs.sh
hadoop@ubuntu-V01:~/data$./sbin/start-yarn.sh
二.windows基础环境准备
windows7(x64),jdk,ant,eclipse,hadoop
1.jdk环境配置
jdk-6u26-windows-i586.exe安装后好后配置相关JAVA_HOME环境变量,并将bin目录配置到path
2.eclipse环境配置
eclipse-standard-luna-SR1-win32.zip解压到D:\eclipse\目录下并命名eclipse-hadoop
下载地址: http://developer.eclipsesource.com/technology/epp/luna/eclipse-standard-luna-SR1-win32.zip
3.ant环境配置
apache-ant-1.9.4-bin.zip解压到D:\apache\目录下,配置环境变量ANT_HOME,并将bin目录配置到path
下载地址: http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.zip
4.下载hadoop-2.5.2.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
5.下载hadoop-2.5.2-src.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2-src.tar.gz
6.下载hadoop2x-eclipse-plugin
https://github.com/winghc/hadoop2x-eclipse-plugin
7.下载hadoop-common-2.2.0-bin
https://github.com/srccodes/hadoop-common-2.2.0-bin
分别将hadoop-2.5.2.tar.gz、hadoop-2.5.2-src.tar.gz、hadoop2x-eclipse-plugin、hadoop-common-2.2.0-bin下载解压到F:\hadoop\目录下
8.修改本地hosts文件,加入如下内容:
192.168.1.112 ubuntu-V01
三、编译hadoop-eclipse-plugin-2.5.2.jar配置
1.添加环境变量HADOOP_HOME=F:\hadoop\hadoop-2.5.2\
追加环境变量path内容:%HADOOP_HOME%/bin
2.修改编译包及依赖包版本信息
修改F:\hadoop\hadoop2x-eclipse-plugin-master\ivy\libraries.properties
hadoop.version=2.5.2
jackson.version=1.9.13
3.ant编译
F:\hadoop\hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin>
ant jar -Dversion=2.5.2 -Declipse.home=D:\eclipse\eclipse-hadoop\eclipse -Dhadoop.home=F:\hadoop\hadoop-2.5.2
编译好后hadoop-eclipse-plugin-2.5.2.jar会在F:\hadoop\hadoop2x-eclipse-plugin-master\build\contrib\eclipse-plugin目录下
三、编译hadoop-eclipse-plugin-2.5.2.jar配置
1.添加环境变量HADOOP_HOME=F:\hadoop\hadoop-2.5.2\
追加环境变量path内容:%HADOOP_HOME%/bin
2.修改编译包及依赖包版本信息
修改F:\hadoop\hadoop2x-eclipse-plugin-master\ivy\libraries.properties
hadoop.version=2.5.2
jackson.version=1.9.13
3.ant编译
F:\hadoop\hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin>
ant jar -Dversion=2.5.2 -Declipse.home=D:\eclipse\eclipse-hadoop\eclipse -Dhadoop.home=F:\hadoop\hadoop-2.5.2
编译好后hadoop-eclipse-plugin-2.5.2.jar会在F:\hadoop\hadoop2x-eclipse-plugin-master\build\contrib\eclipse-plugin目录下
原文网址:http://www.cnblogs.com/huligong1234/p/4137133.html