ubuntu hadoop_ubuntu环境下安装hadoop分布式试错

集群

1. WARN util.NativeCodeLoader

直接在log4j日志中去除告警信息。在/usr/local/hadoop/etc/hadoop/log4j.properties文件中添加

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

2. Error: JAVA_HOME is not set and could not be found

要在/usr/local/hadoop/etc/hadoop/hadoop-env.sh中更新JAVA_HOME环境变量

3. Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now

在/usr/local/hadoop/etc/hadoop中在hadoop-env.sh和yarn-env.sh两个文件中添加

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

伪分布式

1. 由集群模式改为伪分布式模式需要做的修改

hdfs-site.xml里面的3改成1

/etc/hosts里面注释掉不需要的slave

mapredsite.xml 里面内容删掉

2. 伪分布式web的50070不能访问

hadoop3.0之后的prot config有改动，由50070改成了9870

3. 伪分布式模式启动./sbin/start-dfs.sh的时候 localhost: datanode is running as process 21801. Stop it first.

workers文件里的localhost注释掉会解决此问题

4. 运行hadoop fs -mkdir 或者 hadoop fs -ls 出现error：put: `.': No such file or directory: `hdfs://localhost:9000/user/

Well, your problem regarding ls: '.': No such file or directory' is because, there is no home dir on HDFS for your current user. Try

hadoop fs -mkdir -p /user/[current login user]