spark集成

Hive

配置

将hive的配置文件hive-site.xml拷贝到spark conf目录

cp /root/app/hive/conf/hive-site.xml /root/app/spark-alone/conf/



vi hive-site.xml 

<property>

  <name>hive.metastore.uris</name>

  <value>thrift://cdh1:9083</value>

</property>



拷贝hive中的mysql jar包到spark的jar目录下

cp /root/app/hive/lib/mysql-connector-java-5.1.38.jar /root/app/spark-alone/jars/



检查 spark-env.sh 文件中的 hadoop 配置项

cp spark-env.sh.template spark-env.sh

vi spark-env.sh

HADOOP_CONF_DIR=/root/app/hadoop/etc/hadoop

启动相关服务 

1、检查 mysql 是否启动
2、启动 hadoop 集群
cdh1：sbin/start-dfs.sh

cdh1：sbin/start-yarn.sh

cdh2：sbin/yarn-daemon.sh start resourcemanager

3、启动 hive metastore 服务
/root/app/hive/bin

./hive --service metastore



4、启动 spark-shell

./spark-shell

spark.sql("select * from stu3").show

scala> spark.sql("select * from stu3").show

+---+----+---+---+                                                              

|uid|name|sex|age|

+---+----+---+---+

|  1|  张三|  男| 30|

|  2|  赵四|  女| 28|

+---+----+---+---+



5、启动 spark-sql

./spark-sql

select * from stu3 ;

1       张三    男      30

2       赵四    女      28



通过 JDBC/ODBC 连接 ThriftServer来访问 Spark SQL 数据
/root/app/spark-alone/sbin

./start-thriftserver.sh 



/root/app/spark-alone/bin

./beeline 

!connect jdbc:hive2://cdh1:10000

0: jdbc:hive2://cdh1:10000> select * from stu3;

+------+-------+------+------+--+

| uid  | name  | sex  | age  |

+------+-------+------+------+--+

| 1    | 张三    | 男    | 30   |

| 2    | 赵四    | 女    | 28   |

+------+-------+------+------+--+

Mysql 

代码

val df = spark

 .read

 .format("jdbc")

 .option("url", "jdbc:mysql://192.168.230.1:3306/sqoop")

 .option("dbtable", "stu3")

 .option("user", "root")

 .option("password", "123")

 .load()

 

启动服务

spark-shell 命令行中输入 :paste ，然后拷贝上述完整代码。

输入 ctr+d 退出整段输入，然后再输入 df.show 打印读取数据



df: org.apache.spark.sql.DataFrame = [uid: int, name: string ... 2 more fields]



scala> df.show

+---+----+---+---+                                                              

|uid|name|sex|age|

+---+----+---+---+

|  1|  张三|  男| 30|

|  2|  赵四|  女| 28|

+---+----+---+---+

Hbase 

Jar包

1、拷贝 HBase 的包到 spark 的 jars 目录下

cp /root/app/hbase/lib/hbase-client-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/hbase-common-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/hbase-protocol-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/hbase-server-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/htrace-core-3.2.0-incubating.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/metrics-core-2.2.0.jar /root/app/spark-alone/jars/



2、拷贝 hive 的包到 spark 的 jars 目录下

cp /root/app/hive/lib/hive-hbase-handler-1.1.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hive/lib/mysql-connector-java-5.1.38.jar /root/app/spark-alone/jars/ 

启动服务 

 

1、拷贝 HBase 的包到 spark 的 jars 目录下

cp /root/app/hbase/lib/hbase-client-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/hbase-common-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/hbase-protocol-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/hbase-server-1.2.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/htrace-core-3.2.0-incubating.jar /root/app/spark-alone/jars/

cp /root/app/hbase/lib/metrics-core-2.2.0.jar /root/app/spark-alone/jars/



2、拷贝 hive 的包到 spark 的 jars 目录下

cp /root/app/hive/lib/hive-hbase-handler-1.1.0-cdh5.10.0.jar /root/app/spark-alone/jars/

cp /root/app/hive/lib/mysql-connector-java-5.1.38.jar /root/app/spark-alone/jars/