一、版本说明
hadoop 2.7.3
TensorFlow 1.1.0
xlearning 1.1
java 1.8.0
pyton 2.7.5
二、系统环境安装
1. 安装java
下载jdk-8u73-linux-x64.tar.gz包,解压,并放置到指定目录(如:/opt/XLearning/software目录)下
tar -xzvf jdk-7u67-linux-x64.tar.gz -C /opt/XLearning/software
创建软链接
cd /opt/XLearning/software
ln -s jdk1.7.0_67 java
配置环境变量
vim /etc/profile.d/jdk.sh
export JAVA_HOME=/opt/XLearning/software/jdk1.7.0_67
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=$CLASSPATH:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin
环境变量生效
source /etc/profile
2.安装pip
下载pip-9.0.1.tar.gz解压,并安装
tar -xzvf pip-9.0.1.tar.gz
cd pip-9.0.1
python setup.py install
三、Hadoop部署
1.下载、解压
tar -xzvf hadoop-2.7.3.tar.gz
cd hadoop-2.7.3/etc/hadoop
2.修改配置文件
- hadoop-env.sh文件
vim hadoop-env.sh JAVA_HOME = /opt/XLearning/software/java
- yarn-env.sh文件
vim yarn-env.sh JAVA_HOME = /opt/XLearning/software/java
- mapred-env.sh文件
vim mapred-env.sh JAVA_HOME = /opt/XLearning/software/java
- core-site.xml文件
修改_HOST,替换为主机名,如test-1vim core-site.xml
- hdfs-site.xml文件
修改dfs.namenode.name.dir和dfs.datanode.data.dir属性vim hdfs-site.xml
创建目录<property> <name>dfs.namenode.name.dir</name> <value>file:/opt/XLearning/hadoop/data/dfs/name</value> <final>true</final> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/opt/XLearning/hadoop/data/dfs/data</value> <final>true</final> </property>
mkdir /opt/XLearning/hadoop/data/dfs/name
- mapred-site.xml文件
修改_HOST,替换为主机名,如test-1mv mapred-site.xml.template mapred-site.xml vim mapred-site.xml
- yarn-site.xml文件
修改_HOST,替换为主机名,如test-1vim yarn-site.xml
3. 配置HADOOP_HOME
创建软链接
设置环境变量cd /opt/XLearning/software ln -s hadoop-2.7.3 hadoop
vim /etc/profile.d/hadoop.sh export HADOOP_HOME=/opt/XLearning/software/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
4.启动Hadoop各项服务
格式化主节点命名空间
启动存储服务和资源管理服务cd hadoop ./bin/hadoop namenode -format
查看各项服务的状态./sbin/hadoop-daemon.sh start namenode ./sbin/hadoop-daemon.sh start datanode ./sbin/yarn-daemon.sh start resourcemanager ./sbin/yarn-daemon.sh start nodemanager ./sbin/mr-jobhistory-daemon.sh start historyserver
jps -ml
四、安装TensorFlow
利用pip安装
Python下验证pip install tensorflow-1.1.0-cp27-cp27mu-linux_x86_64.whl
>>>import tensorflow as tf >>>tf.__version__
五、XLearning部署
1. 编译、打包
mvn package
2. 解压
tar -xzvf xlearning-1.1-dist.tar.gz
3. 配置
1 log4j.properties文件cd xlearning-1.1 cd conf
去掉注释
2 xlearning-env.sh文件# Settings the HistoryServer logs log4j.logger.net.qihoo.xlearning.jobhistory=DEBUG,RFA log4j.additivity.net.qihoo.xlearning.jobhistory=false log4j.appender.RFA=org.apache.log4j.RollingFileAppender log4j.appender.RFA.File=/tmp/XLearning/logs/XLearningHistoryServer.log log4j.appender.RFA.Encoding=UTF-8 log4j.appender.RFA.Append=true log4j.appender.RFA.MaxFileSize=100MB log4j.appender.RFA.MaxBackupIndex=5 log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
配置JAVA_HOME HADOOP_CONF_DIR
3 xlearning-site.xmlexport JAVA_HOME=/opt/XLearning/software/java export HADOOP_CONF_DIR=/opt/XLearning/software/hadoop/etc/hadoop
配置JobHistory信息,修改0.0.0.0为主机名,如test-14. 启动XLearning JobHistoryServer服务
cd xlearning-1.1 sbin/start-history-server.sh
5. XLearning客户端
配置XLEARNING_HOME
创建目录export XLEARNING_HOME=/opt/XLearning/xlearning-1.1
上传数据到HDFShadoop fs -mkdir -p /tmp/XLearning/history hadoop fs -mkdir -p /tmp/XLearning/eventLog hadoop fs -mkdir -p /tmp/XLearning/staging
执行示例hadoop fs -put data /tmp/
运行结果cd examples/tensorflow sh run.sh
更多精彩文章,详见
红象云腾社区