第一次写比较杂,,,只能等有空再整理了。
环境
我的主机环境是Ubuntu16.04,64位操作系统
工具
jdk:jdk-8u151-linux-x64.tar.gz
hadoop:hadoop-2.6.0-cdh5.7.0.tar.gz
hive:hive-1.1.0-cdh5.7.0.tar.gz
spark:spark-2.2.0.tgz
maven:apache-maven-3.5.2-bin.tar.gz
Scala:scala-2.11.8.tgz
新建用户
1)新建hadoop用户:adduser hadoop
2)修改/etc/sudoers文件:添加hadoop ALL=(ALL:ALL) ALL
配置java环境
1)解压:
tar -xvf jdk-8u151-linux-x64.tar.gz -C /usr/java/
2)修改/etc/profile文件
# set java environment
export JAVA_HOME=/usr/java/jdk1.8.0_151/
export JRE_HOME=/usr/java/jdk1.8.0_151/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
安装hadoop:
1)解压安装包到/usr底下
tar -xvf hadoop-2.6.0-cdh5.7.0.tar.gz -C /usr
2)修改细节
#修改文件名
mv hadoop-xxx-xxx hadoop-2.6.0
#把hadoop文件开启hadoop权限
sudo chown -R hadoop:hadoop hadoop-2.6.0
3)修改相应的配置文件 /etc/profile
# set java environment
export HADOOP_HOME=/usr/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_ROOT_LOGGER=INFO,console
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
4)修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_151
5)修改$HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://106.14.32.248:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/hadoop-2.6.0/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
</configuration>
6)修改$HADOOP_HOME/etc/hadoop/hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>106.14.32.248:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/hadoop-2.6.0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/hadoop-2.6.0/hdfs/data</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
7)复制template,生成xml,命令如下:
cp mapred-site.xml.template mapred-site.xml
8)修改$HADOOP_HOME/etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop001:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop001:19888</value>
</property>
9)修改$HADOOP_HOME/etc/hadoop/yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop001:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop001:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop001:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop001:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop001:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>20480</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
安装ssh免密码登录
1)安装ssh
sudo apt-get install openssh-server
2)将公钥导入自己的机子上
cat id_rsa.pub >> ~/.ssh/authorized_keys
hive下载
1)解压安装包
tar -xvf hive-1.1.0-cdh5.7.0.tar.gz -C /usr
2)修改/etc/profile文件
export HIVE_HOME=/usr/hive-1.1.0
export PATH=$HIVE_HOME/bin:$PATH
3)修改细节
mv hive-1.1.0-cdh5.7.0/ hive-1.1.0
sudo chown -R hadoop:hadoop hive-1.1.0/
4)修改hive-env.sh(mv hive-env.sh.template hive-env.sh)
export HADOOP_HOME=/usr/hadoop-2.6.0
5)修改hive-site.xml文件
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://hadoop001:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
<description>password to use against metastore database</description>
</property>
</configuration>
6)修改 bin/hive-config.sh 文件
export JAVA_HOME=/usr/java/jdk1.8.0_151
export HIVE_HOME=/usr/hive-1.1.0
export HADOOP_HOME=/usr/hadoop-2.6.0
安装spark
1)解压maven:
tar -xvf apache-maven-3.5.2-bin.tar.gz -C /usr
2)配置maven环境,MAVEN_OPTS
export MAVEN_HOME =/usr/apache-maven-3.5.2
export PATH=$MAVEN_HOME/bin:$PATH
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
3)修改maven本地仓库的地址:
cd /usr/apache-maven-3.5.2/conf
vi settings.xml
<localRepository>/usr/apache-maven-3.5.2/maven_repo</localRepository>
3)安装scala
tar -xvf scala-2.11.8.tgz -C /usr/spark-2.2.0/build/
4)配置环境
export SCALA_HOME=/usr/spark-2.2.0/build
export PATH=$SCALA_HOME/bin:$PATH
5)解压spark
tar -xvf spark-2.2.0.tgz -C /usr
6)修改pom.xml文件
<repository>
<id>cloudera</id>
<name>cloudera Repository</name> <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
</repository>
7)编译Spark1
./dev/change-scala-version.sh 2.11
./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Dscala-2.11 -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
./dev/make-distribution.sh \
--name 2.6.0-cdh5.7.0 \
--tgz \
-Dscala-2.11 \
-Dhadoop.version=2.6.0-cdh5.7.0 \
-Phadoop-2.6 \
-Phive -Phive-thriftserver \
-Pyarn
或
./build/mvn -Pyarn -Dscala-2.11 -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 -DskipTests clean package
编译spark2
介于maven多次编译不成功,我又尝试了用git下载的安装包进行编译(保佑可以用吧!)
1)在线安装git
apt-get install git
2)创建一个目录复制spark源代码
mkdir -p /usr/projects/opensource
git clone https://github.com/apache/spark.git
spark环境配置
修改spark-env.sh文件
SPARK_MASTER_HOST=hadoop001
SPARK_WORKER_CORES=2
SPARK_WORKER_MEMORY=2g
SPARK_WORKER_INSTANCES=1
export JAVA_HOME=/usr/java/jdk1.8.0_151/
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export SCALA_HOME=/usr/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH
export SPARK_MASTER_IP=106.14.32.248
export HADOOP_CONF_DIR=/usr/hadoop-2.6.0/etc/hadoop
遇到的问题
1、启动namenode和DataNode报错:
在namenode中报错==>java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 50070
在DataNode中报错==>java.net.BindException: Problem binding to [0.0.0.0:50010] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
在secondarynamenode中报错==>java.net.BindException: Port in use: 0.0.0.0:50090
解决:我是通过kill占用的process,netstat -ap | grep 50070
查看端口占用,并用kill 1068
杀掉对应的processid,之后DataNode就能启动了,但namenode还不行,于是我查到了他说要host:port,原来我的hdfs-site.xml中value值写错了只写了50070,应该是hadoop001:50070
,于是我的namenode和DataNode都起起来了。
注意点:如果DataNode访问不到namenode,那么要用iptables -I INPUT -p tcp --dport 8020 -j ACCEPT
添加启动一下端口的监听
2、安装mysql包时遇到E: Unable to locate package mysql-server
Unable to locate packet就是无法找到包嘛,那还不赶紧sudo apt-get update下,于是就执行sudo apt-get update
问题解决~~~
3、启动./hive时出现错误==>com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
==>Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
==>Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
解决方法:尝试把之前自己建的hive用户删了drop user ‘hive’@’%’;
再新建一个CREATE USER 'hive' IDENTIFIED BY 'hive';
并赋权限GRANT ALL PRIVILEGES ON *.* TO 'hive'@'106.14.32.248' identified by 'hive';
然后重启mysqlsudo /etc/init.d/mysql restart
OK,用hive用户就可以登录了。
4、编译spark时报错,,,精神分裂了要 Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-com
,百度上好像关于这个错没啥资料,有也只是说
pile-first) on project spark-core_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed../dev/change-scala-version.sh 2.11
但我在官网上看说这个只要2.10版本才需要这样搞,2.11不需要,但活马当死马医,果然还是不行,不是这个问题。最后果然还是要借助报错文档,在报错的error提供的网址上标注了要在标签底下加入一行代码:<defaultGoal>install</defaultGoal>
,OKcore没问题了。
5、然而catalyst和sql又报错,用他来解决问题mvn clean install -rf :spark-catalyst_2.11 -Dmaven.test.skip=true
参考文章:Spark的安装和配置http://blog.csdn.net/tian_li/article/details/49328517;
大数据躺过的坑https://www.cnblogs.com/zlslch/p/5865707.html;
学习spark之spark编译部署http://blog.csdn.net/yesuhuangsi/article/details/51830459;
感谢大神们的分享!!!