hadoop安装流程文档

1 除了ssh无密码登录配置外，防火墙关闭与Selinux 权限 都可能导致java.net.NoRouteToHostException: 没有到主机的路由
org.apache.hadoop.ipc.RemoteException(java.io.IOException):

2.start-dfs.js stop-dfs.js
脚本开头添加
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

3start-yarn.js stop-yarn.js
脚本开头添加
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
4Note: System times on machines may be out of sync. Check system time and time zones
yum -y install ntp ntpdate
ntpdate cn.pool.ntp.org

5 Hadoop MapReduce 任务时错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
修改 mapred-site.xml文件在configuration节点 添加

mapreduce.application.classpath
/opt/hadoop/hadoop-3.1.1/share/hadoop/mapreduce/, /opt/hadoop/hadoop-3.1.1/share/hadoop/mapreduce/lib/

6 hadoop运行流程
hdfs文件系统建立文件夹
hadoop fs -mkdir -p dirname
将本地文件put进hdfs文件夹中
hadoop fs -put localfile hdfsfile