编译spark使cdh支持sparksql

最新推荐文章于 2024-05-16 04:21:12 发布
置顶 最新推荐文章于 2024-05-16 04:21:12 发布
阅读量3.5k 收藏 6
点赞数 1
分类专栏： spark cdh 文章标签： Sparksql cdh
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/m0_37618809/article/details/85621799
版权
spark 同时被 2 个专栏收录
14 篇文章 0 订阅
订阅专栏
cdh
12 篇文章 0 订阅
订阅专栏

我的大数据集群环境为cdh5.11.1，在该环境中spark的版本为spark-1.6.0。CDH从5.5开始Spark distro不带Thrift Server分布式SQL引擎、以及spark-sql脚本。Thrift Server是Spark异构数据大融合愿景重要入口之一，spark-sql脚本是测试SQL利器，但CDH优先推自家impala,  SparkSQL虽然不是Spark的主要功能，但SparkSQL是通向Hive和RDB的大门，而且Spark的SQL解析器增加支持一些SQL语法比如注册临时表，这个表可以存在于任何关系数据存储系统（RDB、Hive），只要有驱动就可以，不必编程，还是挺强大的。如果必须要使用SparkSql只能自行编译Spark替换CDH中的jar.

1.Spark源码编译。

下载spark-1.6.0.tgz,地址：https://archive.apache.org/dist/spark/spark-1.6.0/

传输至具有jdk1.8 maven3.3.9 scala2.10.3的环境中，解压准备编译。

进入spark的解压路径，执行

mvn  -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.11.1 -Phive -Phive-thriftserver -Pyarn -Pspark-ganglia-lgpl -DskipTests -Dmaven.test.skip=true -e clean package

过程很慢，耐心等待。

。。。

经过许久 编译成功

进入spark解压路径/spark-1.6.0/assembly/target/scala-2.10路径：

此jar包就是使cdh支持spark支持Sparksql的需要的jar。

这里是博主编译好的jar的下载地址，如需请自行下载，https://download.csdn.net/download/m0_37618809/10891047。

2.cdh集群操作

2.1将编译好的spark-assembly-1.6.0-hadoop2.6.0-cdh5.11.1.jar上传到需要修改的spark主节点的/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/jars/ 路径

不用将原来的spark-assenbly备份删除，这里名称不一样，因此无需备份删除。

2.2替换cdh中的assembly jar

其实就是修改cdh中spark路径下的软链接使其指向新上传jar包所在的路径：

进入/opt/cloudera/parcels/CDH/lib/spark/lib路径：

spark-assembly.jar -> spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

spark-examples-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar-> ../../../jars/spark-examples-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

 

发现spark-assembly.jar包的软链接指向spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

而spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar指向../../../jars/spark-examples-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

 

修改spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar的软链接使其指向新上传的spark assembly jar:

将原有的spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar备份删除或者重命名

 mv spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar_1

spark-assembly.jar会报错因为找不到spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

新加spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar软链接指向spark-assembly-1.6.0-hadoop2.6.0-cdh5.11.1.jar

ln -s ../../../jars/spark-assembly-1.6.0-hadoop2.6.0-cdh5.11.1.jar spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

spark-assembly.jar不报错。

2.3上传spark-sql执行文件

将spark-sql文件上传至/opt/cloudera/parcels/CDH/lib/spark/bin/

并授权可执行： chmod u+x spark-sql

2.4修改环境变量

export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop

export HADOOP_CONF_DIR=/etc/hadoop/conf

export HADOOP_CMD=/opt/cloudera/parcels/CDH/bin/hadoop

export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark

export SCALA_HOME=/opt/lisery/scala-2.10.3

export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:$SCALA_HOME/bin

2.5上传assebly jar至hdfs

su hdfs

hdfs dfs -mkdir -p /user/spark/share/lib

使此目录授予授予用户可访问的权限

hdfs dfs -chmod 777 /user

hdfs dfs -put /opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar  /user/spark/share/lib/

2.6vi /etc/spark/conf/classpath.txt

最末尾添加

/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/jars/spark-lineage_2.10-1.6.0-cdh5.11.1.jar

 

 

2.7cdh web页面中修改

 

 

 

1.spark_jar_hdfs_path 项：/user/spark/share/lib/spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.jar

 

2.

 

spark.yarn.jar=hdfs://192.168.50.240:8020/user/spark/share/lib/spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar

export HIVE_CONF_DIR=/opt/cloudera/parcels/CDH/lib/hive/conf

 

保存修改。

 

点击部署客户端：

完成。

 

在环境中任意一个位置执行spark-sql:

[root@KSJ001 ~]# spark-sql 
WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
WARNING: Running spark-class from user-defined location.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/jars/spark-assembly-1.6.0-hadoop2.6.0-cdh5.11.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/12/29 02:02:58 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/12/29 02:02:58 INFO metastore.ObjectStore: ObjectStore, initialize called
18/12/29 02:02:58 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/12/29 02:02:58 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/12/29 02:03:00 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/12/29 02:03:01 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:01 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:02 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:02 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:02 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/12/29 02:03:02 INFO metastore.ObjectStore: Initialized ObjectStore
18/12/29 02:03:02 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/12/29 02:03:02 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
18/12/29 02:03:03 INFO metastore.HiveMetaStore: Added admin role in metastore
18/12/29 02:03:03 INFO metastore.HiveMetaStore: Added public role in metastore
18/12/29 02:03:03 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
18/12/29 02:03:03 INFO metastore.HiveMetaStore: 0: get_all_databases
18/12/29 02:03:03 INFO HiveMetaStore.audit: ugi=root	ip=unknown-ip-addr	cmd=get_all_databases	
18/12/29 02:03:03 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*
18/12/29 02:03:03 INFO HiveMetaStore.audit: ugi=root	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
18/12/29 02:03:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:04 INFO session.SessionState: Created local directory: /tmp/ce11f047-2b62-4a1c-bda9-06b2a8b51c8b_resources
18/12/29 02:03:04 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/ce11f047-2b62-4a1c-bda9-06b2a8b51c8b
18/12/29 02:03:04 INFO session.SessionState: Created local directory: /tmp/root/ce11f047-2b62-4a1c-bda9-06b2a8b51c8b
18/12/29 02:03:04 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/ce11f047-2b62-4a1c-bda9-06b2a8b51c8b/_tmp_space.db
18/12/29 02:03:04 INFO spark.SparkContext: Running Spark version 1.6.0
18/12/29 02:03:04 INFO spark.SecurityManager: Changing view acls to: root
18/12/29 02:03:04 INFO spark.SecurityManager: Changing modify acls to: root
18/12/29 02:03:04 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/12/29 02:03:04 INFO util.Utils: Successfully started service 'sparkDriver' on port 37925.
18/12/29 02:03:04 INFO slf4j.Slf4jLogger: Slf4jLogger started
18/12/29 02:03:04 INFO Remoting: Starting remoting
18/12/29 02:03:04 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.50.240:34887]
18/12/29 02:03:04 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 34887.
18/12/29 02:03:04 INFO spark.SparkEnv: Registering MapOutputTracker
18/12/29 02:03:04 INFO spark.SparkEnv: Registering BlockManagerMaster
18/12/29 02:03:04 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-4a7f18b5-ce05-4e51-bad0-fb833e3950ac
18/12/29 02:03:04 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB
18/12/29 02:03:05 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/12/29 02:03:05 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
18/12/29 02:03:05 INFO ui.SparkUI: Started SparkUI at http://192.168.50.240:4040
18/12/29 02:03:05 INFO client.RMProxy: Connecting to ResourceManager at KSJ001/192.168.50.240:8032
18/12/29 02:03:05 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
18/12/29 02:03:05 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2564 MB per container)
18/12/29 02:03:05 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
18/12/29 02:03:05 INFO yarn.Client: Setting up container launch context for our AM
18/12/29 02:03:05 INFO yarn.Client: Setting up the launch environment for our AM container
18/12/29 02:03:05 INFO yarn.Client: Preparing resources for our AM container
18/12/29 02:03:05 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://192.168.50.240:8020/user/spark/share/lib/spark-assembly-1.6.0-cdh5.11.1-hadoop2.6.0-cdh5.11.1.jar
18/12/29 02:03:05 INFO yarn.Client: Uploading resource file:/tmp/spark-a16cb29b-ffd1-475a-b1ad-199e356847f1/__spark_conf__6926342778585643878.zip -> hdfs://KSJ001:8020/user/root/.sparkStaging/application_1546050986659_0006/__spark_conf__6926342778585643878.zip
18/12/29 02:03:05 INFO spark.SecurityManager: Changing view acls to: root
18/12/29 02:03:05 INFO spark.SecurityManager: Changing modify acls to: root
18/12/29 02:03:05 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/12/29 02:03:05 INFO yarn.Client: Submitting application 6 to ResourceManager
18/12/29 02:03:05 INFO impl.YarnClientImpl: Submitted application application_1546050986659_0006
18/12/29 02:03:06 INFO yarn.Client: Application report for application_1546050986659_0006 (state: ACCEPTED)
18/12/29 02:03:06 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.users.root
	 start time: 1546066985819
	 final status: UNDEFINED
	 tracking URL: http://KSJ001:8088/proxy/application_1546050986659_0006/
	 user: root
18/12/29 02:03:07 INFO yarn.Client: Application report for application_1546050986659_0006 (state: ACCEPTED)
18/12/29 02:03:08 INFO yarn.Client: Application report for application_1546050986659_0006 (state: ACCEPTED)
18/12/29 02:03:09 INFO yarn.Client: Application report for application_1546050986659_0006 (state: ACCEPTED)
18/12/29 02:03:10 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
18/12/29 02:03:10 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> KSJ001, PROXY_URI_BASES -> http://KSJ001:8088/proxy/application_1546050986659_0006), /proxy/application_1546050986659_0006
18/12/29 02:03:10 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/12/29 02:03:10 INFO yarn.Client: Application report for application_1546050986659_0006 (state: RUNNING)
18/12/29 02:03:10 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.50.242
	 ApplicationMaster RPC port: 0
	 queue: root.users.root
	 start time: 1546066985819
	 final status: UNDEFINED
	 tracking URL: http://KSJ001:8088/proxy/application_1546050986659_0006/
	 user: root
18/12/29 02:03:10 INFO cluster.YarnClientSchedulerBackend: Application application_1546050986659_0006 has started running.
18/12/29 02:03:10 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40814.
18/12/29 02:03:10 INFO netty.NettyBlockTransferService: Server created on 40814
18/12/29 02:03:10 INFO storage.BlockManager: external shuffle service port = 7337
18/12/29 02:03:10 INFO storage.BlockManagerMaster: Trying to register BlockManager
18/12/29 02:03:10 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.50.240:40814 with 511.1 MB RAM, BlockManagerId(driver, 192.168.50.240, 40814)
18/12/29 02:03:10 INFO storage.BlockManagerMaster: Registered BlockManager
18/12/29 02:03:11 INFO scheduler.EventLoggingListener: Logging events to hdfs://KSJ001:8020/user/spark/applicationHistory/application_1546050986659_0006
18/12/29 02:03:11 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.ClouderaNavigatorListener
18/12/29 02:03:11 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
18/12/29 02:03:11 INFO hive.HiveContext: Initializing execution hive, version 1.2.1
18/12/29 02:03:11 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.11.1
18/12/29 02:03:11 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.11.1
18/12/29 02:03:11 INFO hive.metastore: Mestastore configuration hive.metastore.warehouse.dir changed from file:/tmp/spark-2e667ca5-64ab-4890-b8e2-d9f00e9724ef/metastore to file:/tmp/spark-1a392438-a354-45c0-b109-e87bab82f899/metastore
18/12/29 02:03:11 INFO hive.metastore: Mestastore configuration javax.jdo.option.ConnectionURL changed from jdbc:derby:;databaseName=/tmp/spark-2e667ca5-64ab-4890-b8e2-d9f00e9724ef/metastore;create=true to jdbc:derby:;databaseName=/tmp/spark-1a392438-a354-45c0-b109-e87bab82f899/metastore;create=true
18/12/29 02:03:11 INFO metastore.HiveMetaStore: 0: Shutting down the object store...
18/12/29 02:03:11 INFO HiveMetaStore.audit: ugi=root	ip=unknown-ip-addr	cmd=Shutting down the object store...	
18/12/29 02:03:11 INFO metastore.HiveMetaStore: 0: Metastore shutdown complete.
18/12/29 02:03:11 INFO HiveMetaStore.audit: ugi=root	ip=unknown-ip-addr	cmd=Metastore shutdown complete.	
18/12/29 02:03:11 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/12/29 02:03:11 INFO metastore.ObjectStore: ObjectStore, initialize called
18/12/29 02:03:11 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/12/29 02:03:11 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/12/29 02:03:12 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/12/29 02:03:12 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:12 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:12 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:12 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/12/29 02:03:13 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/12/29 02:03:13 INFO metastore.ObjectStore: Initialized ObjectStore
18/12/29 02:03:13 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
18/12/29 02:03:13 INFO metastore.HiveMetaStore: Added admin role in metastore
18/12/29 02:03:13 INFO metastore.HiveMetaStore: Added public role in metastore
18/12/29 02:03:13 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
18/12/29 02:03:13 INFO session.SessionState: Created local directory: /tmp/5d3442f5-4816-4e3f-bd01-7b0a474932f9_resources
18/12/29 02:03:13 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/5d3442f5-4816-4e3f-bd01-7b0a474932f9
18/12/29 02:03:13 INFO session.SessionState: Created local directory: /tmp/root/5d3442f5-4816-4e3f-bd01-7b0a474932f9
18/12/29 02:03:13 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/5d3442f5-4816-4e3f-bd01-7b0a474932f9/_tmp_space.db
18/12/29 02:03:13 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse
18/12/29 02:03:13 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/12/29 02:03:13 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.11.1
18/12/29 02:03:13 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.11.1
18/12/29 02:03:13 INFO hive.metastore: Trying to connect to metastore with URI thrift://KSJ001:9083
18/12/29 02:03:13 INFO hive.metastore: Connected to metastore.
18/12/29 02:03:13 INFO session.SessionState: Created local directory: /tmp/bcd6f57d-2edd-4560-8ea0-e69f604c570c_resources
18/12/29 02:03:13 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/bcd6f57d-2edd-4560-8ea0-e69f604c570c
18/12/29 02:03:13 INFO session.SessionState: Created local directory: /tmp/root/bcd6f57d-2edd-4560-8ea0-e69f604c570c
18/12/29 02:03:13 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/bcd6f57d-2edd-4560-8ea0-e69f604c570c/_tmp_space.db
SET spark.sql.hive.version=1.2.1

 

lisery_nj
关注
  • 1
    点赞
  • 踩
  • 6
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
适配CDH6.3.2的Spark3.2.2
08-16
在使用"spark-3.2.2-bin-3.0.0-cdh6.3.2"这个压缩包时，用户可以直接在CDH6.3.2环境中运行，无需再进行编译，确保了与CDH组件的兼容性和稳定性。需要注意的是，安装和配置Spark时，要遵循CDH的官方文档，以确保所有...
spark-3.2.2-bin-3.0.0-cdh6.3.2
10-20
内容概要：由于cdh6.3.2的spark版本为2.4.0，并且spark-sql被阉割，现基于cdh6.3.2，scala2.12.0，java1.8，maven3.6.3，,对spark-3.2.2源码进行编译 应用：该资源可用于cdh6.3.2集群配置spark客户端，用于spark-sql
CDH5适配spark3.0集成kyuubi详细教程
weixin_44957036的博客
06-22 1705
例如：随着spark的更新迭代，已经官网编译好的版本已经不支持低版本的hive了，由于公司需要使用到网易的开源组件kyuubi，而公司的生产环境还是CDH5，hive版本为1.1.0，所以需要重新编译spark3，对低版本hive进行适配。链接: 自编译Spark3.X，支持CDH 5.16.2(hadoop-2.6.0-cdh5.16.2) 链接: Kyuubi实践 | 编译Spark3.1以适配CDH5并集成Kyuubi 链接: 使用spark源码脚本编译CDH版本spark 二、环境安装 1.mave
服务器编译spark3.3.1源码支持CDH6.3.2
java技术分享
12-10 1621
服务器编译spark3.3.1源码支持CDH6.3.2
最全大数据进阶之路——Spark SQL环境搭建_sparksql环境搭建，2024年最新快速上手
最新发布
2401_84182578的博客
05-16 810
3）机器参数设置配置Java目录5）格式化HDFS注意：这一步操作，，每次如果都格式化的话，那么HDFS上的数据就会被清空看到下面日志表示成功6）启动HDFS7）停止HDFS。
本地IDEA中使用Spark SQL 连接服务器Hive
热门推荐
jzy3711的博客
02-19 28万+
1. 复制hive-site.xml 将Spark-Home/conf目录下的hive-site.xml复制到IDEA工程的resource目录下 2. 修改入口 Spark 2.x版本将SparkContext和HiveContext整合起来，统一使用SparkSession作为连接入口。所以建立连接需要使用如下代码 val conf = new SparkConf() conf.setAp...
hive任务报错SELECT DISTINCT ‘org.apache.hadoop.hive.metastore.model.MPartition‘
2020xyz的博客
04-30 4755
问题描述： hive执行INSERT OVERWRITE TABLE操作时在最后阶段任务出现报错： FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.
Hadoop程序报错，Hive安装报错解决
qq_41888962的博客
10-15 875
Hadoop 报错没有找到某某类的.< init >（） 解决方法：就是在该类中写一个用于实例化初始化方法，如下： // FlowBean是类名 public FlowBean(){ } Hadoop报hadoop warn EBADF: Bad file descriptor错误解决 ...
spark2.3.3-hadoop2.6.0-cdh-5.15.2重新编译
04-27
带编译所需的maven库，spark2.3.3源码基于hadoop2.6.0-cdh-5.15.2进行重新编译
基于CDH 6.3.0 搭建 Hive on Spark 及相关配置和调优
05-25
CDH6针对hive on spark的调优文档，这个是生产的实战经验
spark-2.4.6-cdh5.16.2
06-14
在本主题中，我们关注的是Spark的特定版本——Spark 2.4.6与CDH（Cloudera Distribution Including Apache Hadoop）5.16.2的集成。CDH是一个由Cloudera公司维护的Hadoop生态系统的企业级发行版，它包含了多个Hadoop...
Spark SQL 使用beeline访问hive仓库
weixin_33860722的博客
05-21 1549
一、添加hive-site.xml 在$SPARK_HOME/conf下添加hive-site.xml的配置文件，目的是能正常访问hive的元数据 vim hive-site.xml <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> ...
本地Spark连接Hive异常问题
狂奔的蜗牛
07-08 6947
环境：idea spark2.2 hive1.1 maven3 场景：本地运行Spark代码，连接Hive集群查询数据： 本地设置了SPARK_HOME、SCALA_HOME 将hive-site.xml复制到本地的%SPARK_HOME%/conf/路径下 代码如下： def main(args: Array[String]): Unit = { val wareho...
spark2.2.1+hadoop2.7.1伪分布式安装
Rocky
08-16 1168
环境说明:jdk1.8+scala.2.11.8+spark2.2.1+hadoop.2.7.1 spark2.2.1伪分布式安装说明： 首先要安装好jdk+hadoop+scala 下载。上传、解压 配置spark/conf/spark-env.sh    记得先把原始文件copy一份  cp spark-env.sh.temple spark-env.sh vim spark-env...
Spark Hive在Eclipse代码中直接编译问题
大数据挖掘SparkExpert的博客
12-24 5507
利用Eclipse进行Spark开发过程中，特别是针对Hive开发时，经常会碰到这样一个问题，就是无法找到metastore。而一旦找不到的时候，hive会自动创建一个临时的本地的metastore，其提示INFO信息如下： 15/12/24 20:46:02 INFO HiveContext: Initializing execution hive, version 1.2.1 15/12/
重新编译spark源码，使CDH支持spark sql
鱼归不知处
12-28 1205
1、编辑$MAVEN_HOME/bin/mvn文件，增加配置： MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" 2、执行mvn命令： mvn -Pyarn -PHadoop-2.6 -Dhadoop.version=2.6.0-cdh5.8.3 -Dscala-2.10.5 -Phive -Phi
Spark_CDH版spark的重新编译
weixin_44449054的博客
02-27 320
由于我们所有的环境统一使用CDH的软件版本，并且对应的CDH版本是5.14.0这个版本，所以我们也直接下载使用对应的spark5.14.0版本即可。但是由于spark对应的5.14.0的CDH版本的软件spark的版本还停留在spark1.x，并且CDH版本的spark不支持sparkSQL的功能（sparkSQL模块被阉割掉了），因为Cloudera公司认为有了impala就不需要再使用sparkSQL的功能了，同时也是为了推广impala，所以直接阉割掉了sparkSQL的模块。我们可以使用Apache
重新编译spark 增加spark-sql适配CDH
Dino系我的专栏
12-19 9287
spark-sql
CDH-5.14.0版本编译Spark 2.2.0源码
不管多累生活也要精致
04-12 157
针对CDH-5.14.0版本编译Spark 2.2.0源码，步骤如下： 1、安装JDK及设置环境变量 2、安装MAVEN及设置仓库位置 2.1、仓库压缩包解压 2.2、设置仓库位置 $MAVEN_HOME/conf/settings.xml 增加内容： <localRepository>/opt/modules/mvnrepository</localRepository> 3、下载Spark源码及配置 ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值