大数据学习所有上的pom文件依赖大全

十二点的泡面

已于 2024-08-29 11:37:49 修改

阅读量663

点赞数 8

分类专栏：大数据文章标签：大数据学习 intellij-idea

于 2024-01-19 14:33:59 首次发布

本文链接：https://blog.csdn.net/2301_78038072/article/details/135696954

版权

大数据专栏收录该内容

27 篇文章 2 订阅

订阅专栏

知道Maven项目和pom模型的小伙伴直接跳转到本文的最后获取pom文件的各种依赖

一、Maven项目

1.了解Maven项目

1.1 简介

项目管理的工具
它通过项目对象模型（POM）来描述项目，并提供了丰富的插件和功能来支持项目的构建、测试、文档生成等各个环节
提供了一个免费的中央仓库，其中几乎可以找到任何的流行开源类库

1.2 主要解决的问题

1.3 Maven的作用

以上关于Maven的内容，我放在了我的另一篇CSDN文章上，点击下方链接直接跳转

↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓

大数据学习之Maven项目

二、pom模型

1.pom模型是什么？

页面对象模型（Page Object Model）的简称，是一种设计模式，用于管理维护一组web元素集的对象库

2.pom模型的优点

以上关于Pom模型的内容，我放在了我的另一篇CSDN文章上，点击下方链接直接跳转

↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓

大数据学习之Pom模型

三、pom文件

1. pom.xml文件是什么？

POM.xml文件是Maven项目中的核心文件，它是一个XML格式的文件，用于描述项目的各种属性和配置信息

2. pom.xml文件的作用

提供了项目的元数据和构建配置信息，使得Maven能够自动化地处理项目的构建、依赖管理、文档生成等任务

3. pom文件的依赖内容

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>untitled</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <scala.version>2.12</scala.version>
        <hdfs.version>3.1.3</hdfs.version>
        <flink.version>1.14.0</flink.version>
        <spark.version>3.1.1</spark.version>
        <hudi.version>0.12.0</hudi.version>
        <hive.version>3.1.2</hive.version>
        <hbase.version>2.2.3</hbase.version>
        <clickhouse.version>0.3.2</clickhouse.version>
        <kafka.version>2.4.1</kafka.version>
        <flink-connector-redis.verion>1.1.5</flink-connector-redis.verion>
        <mysql.version>5.1.47</mysql.version>
        <mysqlconnect.version>5.1.47</mysqlconnect.version>
        <lang3.version>3.9</lang3.version>
        <dom4j.version>2.1.0</dom4j.version>

    </properties>


    <dependencies>
        <!-- hudi-spark3.1 -->
        <dependency>
            <groupId>org.apache.hudi</groupId>
            <artifactId>hudi-spark3.1-bundle_${scala.version}</artifactId>
            <version>${hudi.version}</version>
            <!--            <scope>provided</scope>-->
        </dependency>

        <dependency>
            <groupId>com.esotericsoftware</groupId>
            <artifactId>kryo-shaded</artifactId>
            <version>4.0.2</version>
        </dependency>

        <!--scala-->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-reflect</artifactId>
            <version>${scala.version}.12</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}.12</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}.12</version>
        </dependency>

        <!--kafka-->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_${scala.version}</artifactId>
            <version>${kafka.version}</version>
        </dependency>

        <!--flink 实时处理-->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime-web_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-json</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-scala-bridge_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-redis_2.11</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-shaded-hadoop2</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.commons</groupId>
                    <artifactId>commons-lang3</artifactId>
                </exclusion>
            </exclusions>
            <version>${flink-connector-redis.verion}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>${lang3.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-hive_${scala.version}
            </artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-hbase-2.2_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!--        <dependency>-->
        <!--            <groupId>org.apache.flink</groupId>-->
        <!--            <artifactId>flink-connector-jdbc_${scala.version}</artifactId>-->
        <!--            <version>${flink.version}</version>-->
        <!--        </dependency>-->


        <!--mysql连接器-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysqlconnect.version}</version>
        </dependency>


        <!--spark处理离线-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-exec</artifactId>
                </exclusion>
            </exclusions>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-exec</artifactId>
                </exclusion>
            </exclusions>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>


        <!-- hadoop相关-->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hdfs.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-auth</artifactId>
            <version>${hdfs.version}</version>
        </dependency>


        <!--hbase 相关-->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-mapreduce</artifactId>
            <version>${hbase.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
        </dependency>


        <!--clickhouse-->
        <!-- 连接ClickHouse需要驱动包-->
        <dependency>
            <groupId>ru.yandex.clickhouse</groupId>
            <artifactId>clickhouse-jdbc</artifactId>
            <version>${clickhouse.version}</version>
            <!-- 去除与Spark 冲突的包 -->
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>net.jpountz.lz4</groupId>
                    <artifactId>lz4</artifactId>
                </exclusion>
            </exclusions>
        </dependency>


        <!--处理xml文件-->
        <dependency>
            <groupId>org.dom4j</groupId>
            <artifactId>dom4j</artifactId>
            <version>${dom4j.version}</version>
        </dependency>

    </dependencies>


    <build>
        <resources>
            <resource>
                <directory>src/main/scala</directory>
            </resource>
            <resource>
                <directory>src/main/java</directory>
            </resource>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <configuration>
                    <recompileMode>incremental</recompileMode>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>


        </plugins>
    </build>

</project>