hadoop远程调试所遇到的一些异常
问题一:
抛异常:java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.;
可能是端口问题
cdh的环境下,hdfs是8020端口,conf.set(“fs.defaultFS”, “hdfs://192.168.0.4:8020”);
普通hadoop环境,hdfs是8020端口,conf.set(“fs.defaultFS”, “hdfs://192.168.0.121:9000”);
问题二:
抛异常:org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop, access=WRITE, inode=”/magina/output”:hdfs:supergroup:drwxr-xr-x
当前系统用户为hadoop,而调用环境是cdh,操作hdfs文件的用户是hdfs,所以要进行覆盖:
(推荐)System.setProperty(“HADOOP_USER_NAME”,”hdfs”);
或者放开目录的权限,命令如下:
$ hadoop fs -chmod 777 /user/hadoop
chmod 777 /hadoop
问题三:
兼容性问题:
Job job = prepareJob()的老方法不在适用
现如今新版2.5
Job job = Job.getInstance();
//定义Driver类
job.setJarByClass(Write2HiveJob.class);
job.setJobName(“write2Hive”);
……
问题四:
解决Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is…
需要在yarn-site.xml中添加写配置:
yarn.resourcemanager.address127.0.0.1:8032yarn.resourcemanager.scheduler.address127.0.0.1:8030yarn.resourcemanager.resource-tracker.address127.0.0.1:8031
重新格式化HDFS的方法
http://blog.csdn.net/yeruby/article/details/21542465
问题五:
如果eclipse连接hadoop出现Connection refused 错误,ping linux的hadoop集群也能ping通,
/etc/init.d/iptables stop #需要切换到root用户
或者 service iptables stop
永久关闭防火墙
chkconfig –level 35 iptables off
配置文件中 xxx.conf 中使用ip地址