Spark:常用JAR包集成（pom.xml）

最新推荐文章于 2022-02-11 09:25:10 发布
最新推荐文章于 2022-02-11 09:25:10 发布
阅读量535 收藏 1
点赞数
文章标签： 大数据 开发工具 scala
原文链接：http://www.cnblogs.com/drl-blogs/p/11086828.html
版权 
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <scala.version>2.11.8</scala.version>
        <spark.version>2.2.2</spark.version>
        <hadoop.version>2.7.3</hadoop.version>
        <mysql.version>8.0.11</mysql.version>
        <encoding>UTF-8</encoding>
    </properties>
    <dependencies>
        <!-- 导入scala的依赖 -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- 导入spark的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- 导入spark-sql的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- 指定hadoop-client API的版本 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector -->
        <dependency>
            <groupId>org.mongodb.spark</groupId>
            <artifactId>mongo-spark-connector_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>

        <!-- 导入spark-hive的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- 导入spark-streaming的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.version}</version>
        </dependency>

        <!--导入配置文件读取依赖-->
        <!-- https://mvnrepository.com/artifact/com.typesafe/config -->
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scalikejdbc</groupId>
            <artifactId>scalikejdbc_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- scalikejdbc-config_2.11 -->
        <dependency>
            <groupId>org.scalikejdbc</groupId>
            <artifactId>scalikejdbc-config_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.47</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flume/flume-ng-core -->
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.8.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.springframework/spring-webmvc -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-webmvc</artifactId>
            <version>5.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>dom4j</groupId>
            <artifactId>dom4j</artifactId>
            <version>1.6.1</version>
        </dependency>

        <dependency>
            <groupId>jaxen</groupId>
            <artifactId>jaxen</artifactId>
            <version>1.1.6</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.mahout</groupId>
            <artifactId>mahout-core</artifactId>
            <version>0.9</version>
        </dependency>
        <dependency>
            <groupId>org.zeromq</groupId>
            <artifactId>jeromq</artifactId>
            <version>0.3.4</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
        <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>3.5.1</version>
        </dependency>
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.1.1</version>
        </dependency>
        <dependency>
            <groupId>net.sf.json-lib</groupId>
            <artifactId>json-lib</artifactId>
            <version>2.3</version>
            <classifier>jdk15</classifier>
        </dependency>
        <dependency>
            <groupId>net.sf.ezmorph</groupId>
            <artifactId>ezmorph</artifactId>
            <version>1.0.6</version>
        </dependency>
      

        <!-- https://mvnrepository.com/artifact/commons-httpclient/commons-httpclient -->
        <dependency>
            <groupId>commons-httpclient</groupId>
            <artifactId>commons-httpclient</artifactId>
            <version>3.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.4</version>
        </dependency>
        <dependency>
            <groupId>ch.hsr</groupId>
            <artifactId>geohash</artifactId>
            <version>1.3.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.4</version>
        </dependency>


        <dependency>
            <groupId>org.apache.mahout</groupId>
            <artifactId>mahout-integration</artifactId>
            <version>0.9</version>
            <scope>compile</scope>
            <exclusions>
                <exclusion>
                    <artifactId>jetty</artifactId>
                    <groupId>org.mortbay.jetty</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>cassandra-all</artifactId>
                    <groupId>org.apache.cassandra</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hector-core</artifactId>
                    <groupId>me.prettyprint</groupId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <!-- 编译scala的插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!-- 编译java的插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>


            <!-- 打jar插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

转载于:https://www.cnblogs.com/drl-blogs/p/11086828.html

weixin_30483697
关注
大数据学习之Spark——00Spark项目的pom.xml文件
bingque6535的博客
07-21 409
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&g
spark_pom.xml
u011263983的博客
05-25 2344
xml version="1.0" encoding="UTF-8"?> xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4
spark的pom.xml
youmianzhou的博客
11-15 780
一个spark任务提交前需要用idea打包代码，pom.xml设置 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> &lt
spark程序pom.xml
念念的博客,记录学习的点点滴滴！
10-21 1509
&lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://m
SparkStream pom.xml配置
bandi4506的博客
03-16 135
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.o...
spark mllib配置pom.xml错误 Multiple markers at this line Could not transfer artifact net.sf.opencsv:ope...
aanuxzio586136的博客
12-29 692
　　刚刚spark mllib，在maven repository网站http://mvnrepository.com/中查询mllib后得到相关库的最新dependence为： <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark...
pom.xml(Idea中用于整合Scala实现Spark代码编写的配置文件)
12-12
这个文件是用来在Idea...JDK规定, Spark-core, SparkSQL, mysql依赖的jar包,SparkStreaming, SparkStreaming + Kafka, 向kafka 生产数据需要包, 连接 Redis 需要的包,Scala 包. 以及使我们能在idea中创建scala类的插件
新建maven后，配置pom.xml文件导入spark的jar包，出现红色叹号
wzz87的博客
03-28 8713
在真正解决红色叹号问题之前，查阅很多相关资料，如配置tomcat，设置jdk等，这也是可能的原因。但是最终解决还是jar包没有成果导入的原因！通过创建maven项目，配置pom.xml文件自动导入spark的jar包 problems显示的报错信息如下图： 报错信息1：Description Resource Path Location Type Archive fo
presto的jdbc使用
qq5132834的专栏
10-21 1762
1、pom文件 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.a...
spark与scala兼容的pom.xml文件
09-29
在IDEA搭建spark集群，遇到与scala冲突，我提供一个比较完整的pom.xml的文件，供大家分享。
IDEA 搭建spark环境配置文件pom.xml
08-01
IDEA 搭建Spark环境时所需的依赖包配置，可以导入spark基本操作依赖包、机器学习依赖包等等。
常用pom.xml文件.zip
12-17
该资源文件中有maven工程和它的模块的pom.xml配置文件，大多数常用的jar包以及放入其中。需要其他的jar包可以去https://mvnrepository.com/中寻找
spark-streaming 2.2.0 jar包
12-17
基于Scala2.11的spark-streaming 2.2.0 jar包，用于java开发
org.mongodb.spark:mongo-spark-connector_2.11:1.1.0
11-30
mongodb-spark官方连接器，运行spark-submit --packages org.mongodb.spark:mongo-spark-connector_2.11:1.1.0可以自动下载，国内网络不容易下载成功，解压后保存到~/.ivy2目录下即可。
创建spark项目打包成jar包(maven打包)
wjm_520的博客
05-16 2650
创建spark项目打包成jar包(maven打包) 安装Maven 1.下载安装包 官方：官方链接，点击下载。 百度云：百度云链接 提取码：wy98 2.解压和安装maven 3.6.3 sudo unzip ~/下载/apache-maven-3.6.3-bin.zip -d /usr/local #解压到usr/local目录 cd /usr/local sudo mv apache-maven-3.6.3/ ./maven #重命名 sudo chown -R hadoop ./mave
spark pom.xml配置
王坤华的博客
02-11 1876
IDEA软件scala版本2.12.11 pml.xml <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>2.12.4</version> </dependency> <dependency&
对于maven创建spark项目的pom.xml配置文件（图文详解）
weixin_34378767的博客
08-28 1934
    　　不多说，直接上干货！   http://mvnrepository.com/             　　这里，怎么创建，见 Spark编程环境搭建（基于Intellij IDEA的Ultimate版本）（包含Java和Scala版的WordCount）（博主强烈推荐)       　　这里， 我重点说下spark项目，因为，对于hadoop...
spark使用本地local模式时pom.xml的配置
Toby的博客
07-07 909
1、本地local模式启动SparkSession 我写了个脚本做使用本地local模式启动SparkSession，想做个简单的代码测试，如下： //in scala package com.Toby.Gao import org.apache.spark.sql.SparkSession object Test { def main(args:Array[String]){ val spark = SparkSession .builder()
idea搭建spark maven项目的 pom.xml文件配置
最新发布
06-28
在搭建Spark Maven项目时，需要配置pom.xml文件。具体配置如下： 1. 配置项目基本信息，包括groupId、artifactId、version等。 2. 配置依赖项，包括Spark核心依赖、Spark SQL依赖、Hadoop依赖等。 3. 配置插件，...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值