CDH3u4环境下部署Spark+Shark

最新推荐文章于 2021-04-20 13:46:13 发布

tinyid

最新推荐文章于 2021-04-20 13:46:13 发布

阅读量9.2k

点赞数

分类专栏： Spark 文章标签： Spark Shark Hadoop CDH3 实时计算

本文链接：https://blog.csdn.net/cnweike/article/details/23248663

版权

Spark 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

Hadoop版本： CDH3u4

Spark版本： spark-0.9.1-rc3

Shark版本：git clone https://github.com/amplab/shark.git -b branch-0.9

下载对应的Spark包，并解压：

wget https://github.com/apache/spark/archive/v0.9.1-rc3.tar.gz

tar xzvf v0.9.1-rc3.tar.gz

进入到spark目录下，并着手编译Spark：

cd spark-0.9.1-rc3/
SPARK_HADOOP_VERSIOIN=0.20.2-cdh3u4 sbt/sbt clean assembly

编译过程，可能会经历一段比较长的时间，需要耐心等待。待编译结束之后，修改spark-0.9.1-rc3/conf下的spark-env.sh，加入以下内容（通过mesos来虚拟集群资源）：

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export MASTER=zk://storm01:2181,monet00:2181,report:2181/mesos

最后，将编译后的spark目录打包，拷贝到所有节点上的相同位置下，解压即可。

可以通过以下测试命令测试Spark是否安装成功：

bin/run-example org.apache.spark.examples.SparkPi zk://storm01:2181,monet00:2181,report:2181/mesos

我这里的输出是：

bin/run-example org.apache.spark.examples.SparkPi zk://storm01:2181,monet00:2181,report:2181/mesos
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/spark-0.9.1-rc3/examples/target/scala-2.10/spark-examples-assembly-0.9.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/spark-0.9.1-rc3/assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop1.0.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
14/04/09 10:17:06 INFO SparkEnv: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/04/09 10:17:06 INFO SparkEnv: Registering BlockManagerMaster
14/04/09 10:17:06 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140409101706-37d6
14/04/09 10:17:06 INFO MemoryStore: MemoryStore started with capacity 8.4 GB.
14/04/09 10:17:06 INFO ConnectionManager: Bound socket to port 52641 with id = ConnectionManagerId(pnn,52641)
14/04/09 10:17:06 INFO BlockManagerMaster: Trying to register BlockManager
14/04/09 10:17:06 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager pnn:52641 with 8.4 GB RAM
14/04/09 10:17:06 INFO BlockManagerMaster: Registered BlockManager
14/04/09 10:17:06 INFO HttpServer: Starting HTTP Server
14/04/09 10:17:06 INFO HttpBroadcast: Broadcast server started at http://192.168.85.66:57102
14/04/09 10:17:06 INFO SparkEnv: Registering MapOutputTracker
14/04/09 10:17:06 INFO HttpFileServer: HTTP File server directory is /tmp/spark-018553e5-fe2e-4809-b4b6-9e752b9b2bbd
14/04/09 10:17:06 INFO HttpServer: Starting HTTP Server
14/04/09 10:17:07 INFO SparkUI: Started Spark Web UI at http://pnn:4040
14/04/09 10:17:08 INFO SparkContext: Added JAR /opt/spark-0.9.1-rc3/examples/target/scala-2.10/spark-examples-assembly-0.9.1.jar at http://192.168.85.66:57002/jars/spark-examples-assembly-0.9.1.jar with timestamp 1397009828152
2014-04-09 10:17:08,314:27705(0x52f6f940):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@716: Client environment:host.name=pnn
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.18-308.8.1.el5
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Tue May 29 14:57:25 EDT 2012
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@733: Client environment:user.name=hoolai
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@log_env@753: Client environment:user.dir=/opt/spark-0.9.1-rc3
2014-04-09 10:17:08,315:27705(0x52f6f940):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=storm01:2181,monet00:2181,report:2181 sessionTimeout=10000 watcher=0x2aaabda150b0 sessionId=0 sessionPasswd=<null> context=0x2aaab45da2b0 flags=0
2014-04-09 10:17:08,317:27705(0x68391940):ZOO_INFO@check_events@1703: initiated connection to server [192.168.85.114:2181]
2014-04-09 10:17:08,340:27705(0x68391940):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.85.114:2181], sessionId=0x143ade5a8aa01a6, negotiated timeout=10000
I0409 10:17:08.341227 27854 group.cpp:310] Group process ((2)@192.168.85.66:48522) connected to ZooKeeper
I0409 10:17:08.341346 27854 group.cpp:752] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0409 10:17:08.341378 27854 group.cpp:367] Trying to create path '/mesos' in ZooKeeper
I0409 10:17:08.343472 27831 detector.cpp:134] Detected a new leader: (id='43')
I0409 10:17:08.343940 27872 group.cpp:629] Trying to get '/mesos/info_0000000043' in ZooKeeper
I0409 10:17:08.344985 27876 detector.cpp:351] A new leading master (UPID=master@192.168.85.66:5050) is detected
I0409 10:17:08.345324 27834 sched.cpp:218] No credentials provided. Attempting to register without authentication
I0409 10:17:08.345569 27834 sched.cpp:230] Detecting new master
14/04/09 10:17:08 INFO MesosSchedulerBackend: Registered as framework ID 201404081047-1112910016-5050-9652-0016
14/04/09 10:17:08 INFO SparkContext: Starting job: reduce at SparkPi.scala:39
14/04/09 10:17:08 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:39) with 2 output partitions (allowLocal=false)
14/04/09 10:17:08 INFO DAGScheduler: Final stage: Stage 0 (reduce at SparkPi.scala:39)
14/04/09 10:17:08 INFO DAGScheduler: Parents of final stage: List()
14/04/09 10:17:08 INFO DAGScheduler: Missing parents: List()
14/04/09 10:17:08 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at map at SparkPi.scala:35), which has no missing parents
14/04/09 10:17:08 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MappedRDD[1] at map at SparkPi.scala:35)
14/04/09 10:17:08 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
14/04/09 10:17:08 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor 201404011147-1112910016-5050-5163-18: pdn28 (PROCESS_LOCAL)
14/04/09 10:17:08 INFO TaskSetManager: Serialized task 0.0:0 as 1407 bytes in 10 ms
14/04/09 10:17:08 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on executor 201404011147-1112910016-5050-5163-14: pdn24 (PROCESS_LOCAL)
14/04/09 10:17:08 INFO TaskSetManager: Serialized task 0.0:1 as 1407 bytes in 0 ms
14/04/09 10:17:24 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager pdn24:54522 with 294.4 MB RAM
14/04/09 10:17:31 INFO TaskSetManager: Finished TID 1 in 22685 ms on pdn24 (progress: 1/2)
14/04/09 10:17:31 INFO DAGScheduler: Completed ResultTask(0, 1)
14/04/09 10:17:38 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager pdn28:41054 with 294.4 MB RAM
14/04/09 10:17:56 INFO DAGScheduler: Completed ResultTask(0, 0)
14/04/09 10:17:56 INFO TaskSetManager: Finished TID 0 in 47380 ms on pdn28 (progress: 2/2)
14/04/09 10:17:56 INFO DAGScheduler: Stage 0 (reduce at SparkPi.scala:39) finished in 47.408 s
14/04/09 10:17:56 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
14/04/09 10:17:56 INFO SparkContext: Job finished: reduce at SparkPi.scala:39, took 47.606845 s
Pi is roughly 3.14476
14/04/09 10:17:56 INFO MesosSchedulerBackend: driver.run() returned with code DRIVER_STOPPED
14/04/09 10:17:57 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/04/09 10:17:57 INFO ConnectionManager: Selector thread was interrupted!
14/04/09 10:17:57 INFO ConnectionManager: ConnectionManager stopped
14/04/09 10:17:57 INFO MemoryStore: MemoryStore cleared
14/04/09 10:17:57 INFO BlockManager: BlockManager stopped
14/04/09 10:17:57 INFO BlockManagerMasterActor: Stopping BlockManagerMaster
14/04/09 10:17:57 INFO BlockManagerMaster: BlockManagerMaster stopped
14/04/09 10:17:57 INFO SparkContext: Successfully stopped SparkContext
14/04/09 10:17:57 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/04/09 10:17:57 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

运行成功。

下一步，安装Shark，首先获取到0.9分支的最新代码：

git clone https://github.com/amplab/shark.git -b branch-0.9

进入目录并编译：

cd shark
SHARK_HADOOP_VERSION=0.20.2-cdh3u4 sbt/sbt clean package

此编译过程也需要耐心等待一段时间。编译过程中可能会出现一堆警告，可以忽略它们。

待编译完成之后，做一些兼容的工作和配置工作，包括：

1. 将 /opt/shark/lib_managed/jars/org.apache.hadoop/hadoop-core/hadoop-core-1.0.4.jar 换成 hadoop-core-0.20.2-cdh3u4.jar

2. 保证以下jar包出现在/opt/shark/lib目录下：

-rw-r--r-- 1 root root  62230 04-03 17:21 hadoop-lzo-0.4.14.jar
-rw-r--r-- 1 root root 832960 04-03 16:29 mysql-connector-java-5.1.22-bin.jar
-rw-rw-r-- 1 root root  26083 04-05 01:08 slf4j-api-1.7.2.jar
-rw-rw-r-- 1 root root   8819 04-05 01:08 slf4j-log4j12-1.7.2.jar
-rw-r--r-- 1 root root 1218645 04-03 17:29 guava-r09-jarjar.jar

这些jar包都可以从当前的Hadoop环境中拷贝过来。

3. JDK7运行环境部署。使用过程中，需要JDK环境来驱动执行，否则jar包的解析会出错，具体的方法很简单，下载并解压一个jdk7的jar包，在shark-env.sh中指定一下具体的JAVA_HOME即可。

4. 拷贝hive-site.xml到/opt/shark/conf下，使shark能够找到Hive的元数据和其它配置信息。这里需要注意，CDH3u4下的Hive版本是0.7的老旧版本，而Shark默认的Hive兼容版本是Hive0.11，所以我在这里是直接指定了shark自己的元数据库，而不和现有的Hive元数据共用一套，以免对Hive元数据造成损坏。

5. 配置shark-env.sh

export JAVA_HOME=/usr/local/src/jdk   # your JDK7 path

export SPARK_MEM=16g

# (Required) Set the master program's memory
export SHARK_MASTER_MEM=1g

# (Required) Point to your Scala installation.
export SCALA_HOME="/opt/scala"

# (Required) Point to the patched Hive binary distribution
#export HIVE_HOME="/opt/hive/build/dist"

# (Optional) Specify the location of Hive's configuration directory. By default,
# it points to $HIVE_HOME/conf
export HIVE_CONF_DIR="/opt/shark/conf"

# For running Shark in distributed mode, set the following:
export HADOOP_HOME="/usr/lib/hadoop"
export SPARK_HOME="/opt/spark-0.9.1-rc3"
export MASTER="zk://storm01:2181,monet00:2181,report:2181/mesos"
# Only required if using Mesos:
export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so

# Only required if run shark with spark on yarn
#export SHARK_EXEC_MODE=yarn
#export SPARK_ASSEMBLY_JAR=
#export SHARK_ASSEMBLY_JAR=

# (Optional) Extra classpath
export SPARK_LIBRARY_PATH="/usr/local/lib:/usr/lib/hadoop/lib/native/Linux-amd64-64"
export SPARK_CLASSPATH="/usr/lib/hadoop"

# Java options
# On EC2, change the local.dir to /mnt/tmp
SPARK_JAVA_OPTS=" -Dspark.local.dir=/tmp "
SPARK_JAVA_OPTS+="-Dspark.kryoserializer.buffer.mb=10 "
SPARK_JAVA_OPTS+="-verbose:gc -XX:-PrintGCDetails -XX:+PrintGCTimeStamps "
export SPARK_JAVA_OPTS
# 如果你的系统中已经存在以下两个环境变量，要在这里消除掉，保证不要和Shark的执行环境混淆，否则会报错。
unset CLASSPATH 
unset HIVE_HOME

将编译后的shark目录打包，拷贝到所有节点的相同目录下。

至此整个spark+shark环境已经在CDH3u4上组件完毕，经过测试，shark在第一次查询某个表的数据的时候，相对较慢，但是之后的查询速度会上来，因为数据已经缓存在内存里了，以下是一些执行的快照:

shark> select count(1) from a_user_history;
39.024: [GC 285408K->28337K(1005568K), 0.1025650 secs]
78.176: [GC 290993K->21738K(1005568K), 0.0980660 secs]
OK
313758633
Time taken: 85.519 seconds
shark> select count(1) from a_user_history;
157.718: [GC 284394K->32173K(1005568K), 0.0856370 secs]
OK
313758633
Time taken: 11.685 seconds
shark> select count(1) from a_user_history;
220.919: [GC 294829K->22793K(1015808K), 0.1138120 secs]
OK
313758633
Time taken: 10.731 seconds
shark> select count(1) from a_user_history;
232.750: [GC 304380K->34649K(1014272K), 0.1254580 secs]
OK
313758633
Time taken: 9.11 seconds
shark> select count(1) from a_user_history;
247.060: [GC 316249K->21154K(1010688K), 0.0349480 secs]
OK
313758633
Time taken: 9.143 seconds
shark>