spark集群搭建与mysql元数据管理

最新推荐文章于 2023-11-10 10:26:45 发布

美梦是个可怜虫

最新推荐文章于 2023-11-10 10:26:45 发布

阅读量1.1k

点赞数 1

分类专栏：大数据可怜虫文章标签： spark

本文链接：https://blog.csdn.net/u010128142/article/details/82113505

版权

大数据同时被 2 个专栏收录

17 篇文章 0 订阅

订阅专栏

可怜虫

2 篇文章 0 订阅

订阅专栏

找个spark集群搭建是针对于上一篇hadoop的基础上搭建的。
所以spark的版本也是要按照着hadoop版本进行下载。

1.解压spark，修改spark的/etc/profile的home目录。

2.安装SCALA，并配置SCALA_HOME。

3.修改spark conf目录下的spark-env.sh文件，并添加下列配置
export JAVA_HOME=/root/java/jdk1.8.0_181
export HADOOP_HOME=/root/hadoop/hadoop-2.7.6
export HADOOP_CONF_DIR=/root/hadoop/hadoop-2.7.6/etc/hadoop
export SCALA_HOME=/root/scala/scala-2.11.8
export SPARK_MASTER_IP=192.168.124.132

4.修改spark conf目录下的slavers文件配置：
centos01
centos02
centos03

5.分发spark到其它节点，包括/etc/profile文件
scp -r /etc/profile root@centos02:/etc/
scp -r ~/spark root@centos02:/root/

5.测试spark yarn提交模式，在spark example目录下有个PI运算jar包
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster spark-examples_2.11-2.3.0.jar 10

利用spark-shell --master yarn进行测试，会出现异常：
2018-08-27 01:01:30 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
利用http://192.168.124.132:8088/cluster/app/application_1535337995441_0003：
查看Diagnostics（诊断）发现报错虚拟内存超过了：
is running beyond virtual memory limits.
Current usage: 40.9 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

配置yarn-site.xml:


<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers</description>
</property>

<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4</value>
<description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

6.配置spark sql的metastore到mysql进行管理
查看机器是否安装过mysql：rpm -qa|grep -i mysql
wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
rpm -ivh mysql57-community-release-el7-8.noarch.rpm
yum install mysql-community-server

安装完成后，重启mysql：service mysqld restart
然后查看初始密码：grep "password" /var/log/mysqld.log
登陆mysql，修改密码~！：alter user 'root'@'localhost' identified by 'xxxx2018@spark';
刷新权限：flush privileges;

7.把jdbc的jar包复制到spark jars目录下：（注意：使用5版本的可以避免出现时区问题）
网上下载mysql jdbc驱动，需要注意驱动和mysql的版本~！
   网上下载https://dev.mysql.com/downloads/connector/j/
   rpm包解压：rpm2cpio mysql-connector-java-8.0.12-1.el7.noarch.rpm | cpio -div
   把jdbc驱动复制到spark jars目录：cp mysql-connector-java-8.0.12.jar ~/sparkapp/spark-2.3.0-bin-hadoop2.7/jars/

8.在spark conf目录下创建一个hive-site.xml文件：
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hiveMetastore?createDatabaseIfNotExist=true&characterEncoding=utf8&useSSL=false</value>
<description>hiveMetastore:Metastore sive in mysql</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>e
<description>mysql account</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>xxxx2018@spark</value>
<description>mysql password</description>
</property>
</configuration>

9.在mysql上创建相关的spark元数据数据库，和视图：
mysql:
create database sparkmetastore;
use sparkmetastore;

create view databases_v
as select DBS.*
from hiveMetastore.DBS;

create view tables_v
as select TBLS.*, DBS.NAME
from hiveMetastore.DBS, hiveMetastore.TBLS
where TBLS.DB_ID=DBS.DB_ID AND TBLS.TBL_TYPE!='VIRTUAL_VIEW';

create view views_v
as select TBLS.*, DBS.NAME
from hiveMetastore.DBS, hiveMetastore.TBLS
where TBLS.DB_ID=DBS.DB_ID AND TBLS.TBL_TYPE='VIRTUAL_VIEW';

create view columns_v
as select COLUMNS_V2.*, TBLS.TBL_NAME, DBS.NAME
from hiveMetastore.DBS, hiveMetastore.TBLS, hiveMetastore.COLUMNS_V2
where DBS.DB_ID = TBLS.DB_ID AND COLUMNS_V2.CD_ID = TBLS.TBL_ID;

10.再在mysql上创建一个用户，该用户为spark-sql jdbc元数据查询用户：
CREATE USER 'spark'@'%' IDENTIFIED BY 'xxxx2018@spark';
GRANT SELECT ON sparkmetastore.* TO 'spark'@'%';

11.在spark-sql中进行反向查询元数据：
create database spark;
CREATE TABLE databases_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.databases_v","user" "spark", "password" "xxxx2018@spark")
CREATE TABLE tables_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.tables_v","user" "spark", "password" "xxxx2018@spark")
CREATE TABLE views_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.views_v","user" "spark", "password" "xxxx2018@spark")
CREATE TABLE columns_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.columns_v","user" "spark", "password" "xxxx2018@spark")

12.打开spark sbin./start-thriftserver.sh服务。

13.在远程root访问hdfs的时候，会出现权限问题：
在hdfs-site.xml加入找个配置
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>xxxx.com:50090</value>
</property>
<!-- hdfs用root登陆的时候会出现权限错误>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
<description>
If "true", enable permission checking in HDFS.
If "false", permission checking is turned off,
but all other behavior is unchanged.
Switching from one parameter value to the other does not change the mode,
owner or group of files or directories.
</description>

14.配置spark-default.conf:(一定要把spark-warehouse设置存储到hdfs上，不然会出现错误)
spark.master yarn

spark.sql.warehouse.dir hdfs://centos01:9000/spark-warehouse