Hadoop/Yarn/Hive/Spark等大数据环境搭建集成篇

Hadoop/Yarn/Hive/Spark等大数据环境搭建集成篇

一、Hadoop环境搭建与配置

1、前置准备，安装jdk和ssh免密登录

参见https://blog.csdn.net/qq_29269907/article/details/81221430

2、下载hadoop

wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz 
mv hadoop-2.6.0-cdh5.7.0 ~/app
mv hadoop-2.6.0-cdh5.7.0 hadoop

3、配置hadoop

cd ~/app/hadoop/etc/hadoop
vi hadoop-env.sh
	export JAVA_HOME=/usr/java/jdk1.7.0_75
vi core-site.xml

	<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://jussilee:8020</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>