Ochadoop之shark、spark on yarn搭建[转自 AIMP平台wiki]

安装好hadoop、hive、yarn

shark、spark安装包

spark-assembly_0.9.1-hadoop2.3.0-cdh5.0.0-och3.1.0.jar
shark-0.9.1-och3.1.0.tar

安装步骤

1）下载并安装scala: wget http://www.scala-lang.org/files/archive/scala-2.10.4.tgz
2）解包shark-0.9.1-och3.1.0.tar
3）修改conf/shark-env.sh

export SCALA_HOME="/home/cdh5/ochadoop-och3.1.0/scala-2.10.4" 
export HIVE_CONF_DIR="/home/cdh5/app/hive/conf" 
export SHARK_EXEC_MODE=yarn
export SPARK_ASSEMBLY_JAR=/home/cdh5/ochadoop-och3.1.0/spark-assembly_0.9.1-hadoop2.3.0-cdh5.0.0-och3.1.0.jar
export SHARK_ASSEMBLY_JAR=/home/cdh5/ochadoop-och3.1.0/shark-0.9.1-och3.1.0/target/scala-2.10/shark-assembly-0.9.1-och3.1.0-hadoop2.3.0-cdh5.0.0.jar

export HIVE_SERVER2_THRIFT_PORT=10002        #shark server端口
export SPARK_WORKER_INSTANCES=${SPARK_WORKER_INSTANCES:-2}
export SHARK_MEM=${SHARK_MEM:-2g}
export SHARK_MASTER_MEM=${SHARK_MASTER_MEM:-1g}

4）检查yarn-site.xml，需要有以下配置项：
<property>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
</value>
</property>

启用snappy压缩，需要在shark-env.sh中配置SPARK_JAVA_OPTS -Djava.library.path指定native库路径

...
SPARK_JAVA_OPTS+="-Djava.library.path=/home/cdh5/app/hadoop/lib/native " 
...

启动

1) Shark CLI
bin/shark 交互模式
bin/shark -e "select * from test" 执行sql然后退出

2) shark server
./shark --service sharkserver2
端口可以在shark-env.sh 中配置export HIVE_SERVER2_THRIFT_PORT=10002
可以用jdbc连接shark server

启动成功可以在yarn管理页面查看到application type是SPARK的application。

---

Updated by 杨 启虎 , 更新于 5 天 之前
Access count: 6 :since 2012-04-07