flink-connector-mysql-cdc编译打包

最新推荐文章于 2024-12-04 22:37:09 发布
最新推荐文章于 2024-12-04 22:37:09 发布
阅读量862 收藏 1
点赞数
文章标签： flink mysql java
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u010858669/article/details/130597101
版权

一、下载flink-cdc-connectors源码

git clone -b master https://github.com/ververica/flink-cdc-connectors.git

或者

git clone -b master https://gitee.com/mirrors_ververica/flink-cdc-connectors.git

二、查看集群flink版本

flink --version

Version: 1.14.0

三、flink-cdc-connectors对应flink版本适配

1. flink-cdc-connectors/pom.xml

  1. 注释掉不需要的模块

  2. 修改flink.version版本号

  3. 修改flink-table-api-java-bridge依赖适配

  4. 修改flink-streaming-java依赖适配

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2022 Ververica Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <parent>
        <groupId>org.apache</groupId>
        <artifactId>apache</artifactId>
        <version>20</version>
    </parent>

    <modelVersion>4.0.0</modelVersion>

    <groupId>com.ververica</groupId>
    <artifactId>flink-cdc-connectors</artifactId>
    <version>2.4-SNAPSHOT</version>
    <packaging>pom</packaging>

    <modules>
        <module>flink-cdc-base</module>
        <module>flink-connector-debezium</module>
        <module>flink-connector-test-util</module>
        <module>flink-connector-mysql-cdc</module>
        <module>flink-sql-connector-mysql-cdc</module>
        <!-- 
        <module>flink-connector-postgres-cdc</module>
        <module>flink-connector-oracle-cdc</module>
        <module>flink-connector-mongodb-cdc</module>
        <module>flink-connector-oceanbase-cdc</module>
        <module>flink-connector-sqlserver-cdc</module>
        <module>flink-connector-tidb-cdc</module>
        <module>flink-connector-db2-cdc</module>
        <module>flink-sql-connector-postgres-cdc</module>
        <module>flink-sql-connector-mongodb-cdc</module>
        <module>flink-sql-connector-oracle-cdc</module>
        <module>flink-sql-connector-oceanbase-cdc</module>
        <module>flink-sql-connector-sqlserver-cdc</module>
        <module>flink-sql-connector-tidb-cdc</module>
        <module>flink-sql-connector-db2-cdc</module>
        <module>flink-cdc-e2e-tests</module>
        -->
    </modules>

    <licenses>
        <license>
            <name>The Apache Software License, Version 2.0</name>
            <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <distributionManagement>
        <snapshotRepository>
            <id>oss</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        </snapshotRepository>
        <repository>
            <id>oss</id>
            <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
        </repository>
    </distributionManagement>

    <properties>
        <flink.version>1.14.0</flink.version>
        <debezium.version>1.6.4.Final</debezium.version>
        <tikv.version>3.2.0</tikv.version>
        <geometry.version>2.2.0</geometry.version>
        <!-- OracleE2eITCase will report "container cannot be accessed" error when running in Azure Pipeline with 1.16.1 testconainters.
          This might be a conflicts with "wnameless/oracle-xe-11g-r2" and 1.16 testcontainers.
          We may need to upgrade our Oracle base image to "gvenzl/oracle-xe" which is the default image of 1.16 testcontainers.
          See more https://github.com/testcontainers/testcontainers-java/issues/4297. -->
        <testcontainers.version>1.15.3</testcontainers.version>
        <java.version>1.8</java.version>
        <scala.binary.version>2.12</scala.binary.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
        <hamcrest.version>1.3</hamcrest.version>
        <version.awaitility>4.0.1</version.awaitility>
        <slf4j.version>1.7.15</slf4j.version>
        <log4j.version>2.17.1</log4j.version>
        <spotless.version>2.4.2</spotless.version>
        <oblogclient.version>1.0.6</oblogclient.version>
        <!-- Enforce single fork execution due to heavy mini cluster use in the tests -->
        <flink.forkCount>1</flink.forkCount>
        <flink.reuseForks>true</flink.reuseForks>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- Apache Flink dependencies -->
        <!-- These dependencies are provided, because they should not be packaged into the JAR file. -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-common</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-base</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <!--  Use fixed version 30.1.1-jre-16.0 of flink shaded guava  -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-shaded-guava</artifactId>
            <version>30.1.1-jre-16.0</version>
        </dependency>

        <!-- test dependencies -->
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-all</artifactId>
            <version>${hamcrest.version}</version>
            <type>jar</type>
            <scope>test</scope>
        </dependency>

        <!-- tests will have log4j as the default logging framework available -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>${log4j.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>${log4j.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <!-- API bridge between log4j 1 and 2 -->
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-1.2-api</artifactId>
            <version>${log4j.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.awaitility</groupId>
            <artifactId>awaitility</artifactId>
            <type>jar</type>
            <version>${version.awaitility}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <!-- Inherited from Apache parent, but not actually used. Disable to reduce noise. -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-site-plugin</artifactId>
                <executions>
                    <execution>
                        <id>attach-descriptor</id>
                        <phase>none</phase>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.rat</groupId>
                <artifactId>apache-rat-plugin</artifactId>
                <version>0.12</version><!--$NO-MVN-MAN-VER$-->
                <inherited>false</inherited>
                <executions>
                    <execution>
                        <phase>verify</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <excludeSubProjects>false</excludeSubProjects>
                    <numUnapprovedLicenses>0</numUnapprovedLicenses>
                    <licenses>
                        <!-- Enforce this license:
                            Copyright 2022 Ververica Inc.
                            Licensed under the Apache License, Version 2.0 (the "License");
                            you may not use this file except in compliance with the License.
                            You may obtain a copy of the License at
                              http://www.apache.org/licenses/LICENSE-2.0
                            Unless required by applicable law or agreed to in writing,
                            software distributed under the License is distributed on an
                            "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
                            KIND, either express or implied.  See the License for the
                            specific language governing permissions and limitations
                            under the License.
                        -->
                        <license
                                implementation="org.apache.rat.analysis.license.SimplePatternBasedLicense">
                            <licenseFamilyCategory>AL2</licenseFamilyCategory>
                            <licenseFamilyName>Apache License 2.0</licenseFamilyName>
                            <notes/>
                            <patterns>
                                <pattern>Copyright 2022 Ververica Inc.
                                </pattern>
                                <pattern>Licensed under the Apache License, Version 2.0
                                </pattern>
                            </patterns>
                        </license>
                    </licenses>
                    <licenseFamilies>
                        <licenseFamily implementation="org.apache.rat.license.SimpleLicenseFamily">
                            <familyName>Apache License 2.0</familyName>
                        </licenseFamily>
                    </licenseFamilies>
                    <excludes>
                        <!-- Additional files like .gitignore etc.-->
                        <exclude>**/.*/**</exclude>
                        <!-- Generated content -->
                        <exclude>**/target/**</exclude>
                        <!-- Build files -->
                        <exclude>**/*.iml</exclude>
                        <!-- Docs -->
                        <exclude>**/*.md</exclude>
                        <exclude>docs/Makefile</exclude>
                        <exclude>docs/Dockerfile</exclude>
                        <exclude>docs/make.bat</exclude>
                        <exclude>docs/_templates/version.html</exclude>
                        <exclude>docs/_static/button.js</exclude>
                        <exclude>docs/_build/**</exclude>
                        <!-- Site -->
                        <exclude>docs/site/**</exclude>
                        <!-- Tests -->
                        <exclude>**/*.txt</exclude>
                        <exclude>flink-connector-mysql-cdc/src/test/resources/file/*.json</exclude>
                        <exclude>flink-connector-db2-cdc/src/test/resources/db2_server/Dockerfile</exclude>
                    </excludes>
                </configuration>
            </plugin>

            <!-- Due to the Flink build setup, "mvn spotless:apply" and "mvn spotless:check"
				don't work. You have to use the fully qualified name, i.e.
				"mvn com.diffplug.spotless:spotless-maven-plugin:apply" -->
            <plugin>
                <groupId>com.diffplug.spotless</groupId>
                <artifactId>spotless-maven-plugin</artifactId>
                <version>${spotless.version}</version>
                <configuration>
                    <java>
                        <googleJavaFormat>
                            <version>1.7</version>
                            <style>AOSP</style>
                        </googleJavaFormat>

                        <!-- \# refers to the static imports -->
                        <importOrder>
                            <order>org.apache.flink,org.apache.flink.shaded,,javax,java,scala,\#
                            </order>
                        </importOrder>

                        <removeUnusedImports/>
                    </java>
                </configuration>
                <executions>
                    <execution>
                        <id>spotless-check</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-checkstyle-plugin</artifactId>
                <version>2.17</version>
                <dependencies>
                    <dependency>
                        <groupId>com.puppycrawl.tools</groupId>
                        <artifactId>checkstyle</artifactId>
                        <version>8.14</version>
                    </dependency>
                </dependencies>
                <executions>
                    <execution>
                        <id>validate</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <suppressionsLocation>/tools/maven/suppressions.xml</suppressionsLocation>
                    <includeTestSourceDirectory>true</includeTestSourceDirectory>
                    <configLocation>/tools/maven/checkstyle.xml</configLocation>
                    <logViolationsToConsole>true</logViolationsToConsole>
                    <failOnViolation>true</failOnViolation>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.0</version>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit47</artifactId>
                        <version>2.22.0</version>
                    </dependency>
                </dependencies>
                <executions>
                    <execution>
                        <id>default-test</id>
                        <phase>test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <includes>
                                <include>**/*Test.java</include>
                                <include>**/*Test.scala</include>
                            </includes>
                        </configuration>
                    </execution>
                    <execution>
                        <id>integration-tests</id>
                        <phase>integration-test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <includes>
                                <include>**/*ITCase.java</include>
                                <include>**/*ITCase.scala</include>
                            </includes>
                            <reuseForks>false</reuseForks>
                        </configuration>
                    </execution>
                </executions>
                <configuration>
                    <forkCount>${flink.forkCount}</forkCount>
                    <reuseForks>${flink.reuseForks}</reuseForks>
                    <systemPropertyVariables>
                        <forkNumber>0${surefire.forkNumber}</forkNumber>
                    </systemPropertyVariables>
                    <!-- Prevent ORA-01882: timezone region not found errors due to the Oracle DB
                         does not understanding some time zone which used in Ubuntu OS -->
                    <argLine>-Xms256m -Xmx2048m -Dmvn.forkNumber=${surefire.forkNumber}
                        -XX:+UseG1GC -Doracle.jdbc.timezoneAsRegion=false
                    </argLine>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <id>shade-flink</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <!-- Shading test jar have bug in some previous version, so close this configuration here,
                            see https://issues.apache.org/jira/browse/MSHADE-284 -->
                            <shadeTestJar>false</shadeTestJar>
                            <shadedArtifactAttached>false</shadedArtifactAttached>
                            <createDependencyReducedPom>true</createDependencyReducedPom>
                            <dependencyReducedPomLocation>
                                ${project.basedir}/target/dependency-reduced-pom.xml
                            </dependencyReducedPomLocation>
                            <filters combine.children="append">
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>module-info.class</exclude>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <artifactSet>
                                <includes>
                                    <!-- include nothing -->
                                    <include></include>
                                </includes>
                            </artifactSet>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>3.0.0-M1</version>
                <executions>
                    <execution>
                        <id>dependency-convergence</id>
                        <!-- disabled by default as it interacts badly with shade-plugin -->
                        <phase>none</phase>
                        <goals>
                            <goal>enforce</goal>
                        </goals>
                        <configuration>
                            <rules>
                                <dependencyConvergence/>
                            </rules>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <executions>
                    <execution>
                        <id>test-jar</id>
                        <goals>
                            <goal>test-jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <configuration>
                    <doclint>none</doclint>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <profiles>
        <profile>
            <id>release</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-gpg-plugin</artifactId>
                        <version>1.6</version>
                        <executions>
                            <execution>
                                <id>sign-artifacts</id>
                                <phase>verify</phase>
                                <goals>
                                    <goal>sign</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-enforcer-plugin</artifactId>
                        <executions>
                            <execution>
                                <id>enforce-java</id>
                                <goals>
                                    <goal>enforce</goal>
                                </goals>
                                <configuration>
                                    <rules>
                                        <requireJavaVersion>
                                            <version>[1.8,)</version>
                                        </requireJavaVersion>
                                    </rules>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-source-plugin</artifactId>
                        <version>2.2.1</version><!--$NO-MVN-MAN-VER$-->
                        <executions>
                            <execution>
                                <id>attach-sources</id>
                                <goals>
                                    <goal>jar</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                    <plugin>
                        <groupId>net.ju-n.maven.plugins</groupId>
                        <artifactId>checksum-maven-plugin</artifactId>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>artifacts</goal>
                                </goals>
                            </execution>
                        </executions>
                        <configuration>
                            <algorithms>
                                <algorithm>SHA-256</algorithm>
                            </algorithms>
                            <failOnError>false</failOnError>
                        </configuration>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-javadoc-plugin</artifactId>
                        <version>2.9.1</version><!--$NO-MVN-MAN-VER$-->
                        <configuration>
                            <quiet>true</quiet>
                            <detectOfflineLinks>false</detectOfflineLinks>
                            <additionalJOptions combine.children="append">
                                <additionalJOption>-Xdoclint:none</additionalJOption>
                            </additionalJOptions>
                        </configuration>
                        <executions>
                            <execution>
                                <id>attach-javadocs</id>
                                <goals>
                                    <goal>jar</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
                <pluginManagement>
                    <plugins>
                        <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-release-plugin</artifactId>
                            <version>2.1</version>
                            <configuration>
                                <mavenExecutorId>forked-path</mavenExecutorId>
                                <useReleaseProfile>false</useReleaseProfile>
                                <arguments>${arguments} -Psonatype-oss-release</arguments>
                            </configuration>
                        </plugin>
                    </plugins>
                </pluginManagement>
            </build>
        </profile>
    </profiles>
</project>

2. flink-cdc-connectors/flink-cdc-base/pom.xml

  1. 修改flink-table-runtime依赖适配

  2. 修改flink-test-utils依赖适配

  3. 修改flink-streaming-java依赖适配

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2022 Ververica Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>flink-cdc-connectors</artifactId>
        <groupId>com.ververica</groupId>
        <version>2.4-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>flink-cdc-base</artifactId>

    <dependencies>
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-debezium</artifactId>
            <version>${project.version}</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <artifactId>kafka-log4j-appender</artifactId>
                    <groupId>org.apache.kafka</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.zaxxer</groupId>
            <artifactId>HikariCP</artifactId>
            <version>4.0.3</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-runtime_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-test-utils</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-common</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-tests</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>


        <!-- Debezium test dependencies -->

        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-connector-mysql</artifactId>
            <version>${debezium.version}</version>
            <scope>test</scope>
        </dependency>


        <!-- test dependencies on TestContainers -->

        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>mysql</artifactId>
            <version>${testcontainers.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

3. flink-cdc-connectors/flink-connector-debezium/pom.xml

  1. 修改flink-test-utils依赖适配

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2022 Ververica Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>flink-cdc-connectors</artifactId>
        <groupId>com.ververica</groupId>
        <version>2.4-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>flink-connector-debezium</artifactId>
    <name>flink-connector-debezium</name>

    <dependencies>

        <!-- Debezium dependencies -->
        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-api</artifactId>
            <version>${debezium.version}</version>
        </dependency>
        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-embedded</artifactId>
            <version>${debezium.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>kafka-log4j-appender</artifactId>
                    <groupId>org.apache.kafka</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.glassfish.jersey.core</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

4. flink-cdc-connectors/flink-connector-test-util/pom.xml

  1. 修改flink-test-utils依赖适配

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2022 Ververica Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>flink-cdc-connectors</artifactId>
        <groupId>com.ververica</groupId>
        <version>2.4-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>flink-connector-test-util</artifactId>
    <name>flink-connector-test-util</name>
    <packaging>jar</packaging>

    <dependencies>
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-debezium</artifactId>
            <version>${project.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>kafka-log4j-appender</artifactId>
                    <groupId>org.apache.kafka</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
    </dependencies>

</project>

5. flink-cdc-connectors/flink-connector-mysql-cdc/pom.xml

  1. 修改flink-table-runtime依赖适配

  2. 修改flink-test-utils依赖适配

  3. 修改flink-streaming-java依赖适配

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2022 Ververica Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>flink-cdc-connectors</artifactId>
        <groupId>com.ververica</groupId>
        <version>2.4-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>flink-connector-mysql-cdc</artifactId>
    <name>flink-connector-mysql-cdc</name>
    <packaging>jar</packaging>

    <dependencies>

        <!-- Debezium dependencies -->
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-debezium</artifactId>
            <version>${project.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>kafka-log4j-appender</artifactId>
                    <groupId>org.apache.kafka</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-connector-mysql</artifactId>
            <version>${debezium.version}</version>
        </dependency>

        <!-- geometry dependencies -->
        <dependency>
            <groupId>com.esri.geometry</groupId>
            <artifactId>esri-geometry-api</artifactId>
            <version>${geometry.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.zaxxer</groupId>
            <artifactId>HikariCP</artifactId>
            <version>4.0.3</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- test dependencies on Debezium -->

        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-test-util</artifactId>
            <version>${project.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-core</artifactId>
            <version>${debezium.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.78</version>
            <scope>test</scope>
        </dependency>

        <!-- test dependencies on Flink -->

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-runtime_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-test-utils</artifactId>
            <version>${flink.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-common</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-tests</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>

        <!-- test dependencies on TestContainers -->

        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>mysql</artifactId>
            <version>${testcontainers.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.jayway.jsonpath</groupId>
            <artifactId>json-path</artifactId>
            <version>2.4.0</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

</project>

四、编译打包

  1. 执行spotless，统一格式化代码

mvn spotless:apply

  1. 执行编译构建

mvn clean install -DskipTests

xuen-zhang
关注
flink-connector-mysql-cdc：01 mysql-cdc础配置代码演示
QYmufeng的博客
12-04 564
flink-connector-mysql-cdc 代码测试以及常见问题处理，扩展应用，生产及代码分析
flink-cdc实时增量同步mysql数据到hbase
大数据技术派
09-16 1916
CDC是（Change Data Capture 变更数据获取）的简称。核心思想是，监测并捕获数据库的变动（包括数据 或 数据表的插入INSERT、更新UPDATE、删除DELETE等），将这些变更按发生的顺序完整记录下来，写入到消息中间件中以供其他服务进行订阅及消费。
flink-sql-connector-mysql-cdc-1.2.0.jar
03-05
flink cdc mysql jar包，2021年最新版
flink-connector-mysql-cdc编译——flink-1.15.2版本
qq_42418398的博客
08-29 6472
目前flink cdc的版本不支持flink1.15.*版本，可以自己编译cdc的源码，本文主要介绍编译过程中遇到的问题和需要注意的地方
Flink MySQL CDC
qq_32323239的博客
12-07 4129
Flink sqlclient使用方式下载flink-sql-connector-mysql-cdc-2.3-SNAPSHOT.jar放到.注意： flink-sql-connector-mysql-cdc-XXX-SNAPSHOT 版本为开发分支对应的代码。用户需要下载源代码并编译相应的jar。用户应该使用发布版本，例如flink-sql-connector-mysql-cdc-XXX.jar，发布版本将在 Maven 中央仓库中可用。为每个reader设置设置不同的每一个读取 binlog 的 MySQ
flink-connector-mysql-cdc-1.1.1.jar
03-12
flink-connector-mysql-cdc-1.1.1.jar
flink-connector-mysql-cdc
weixin_37536020的博客
09-19 1200
FlinkCDC是一个实现CDC(Change Data Capture)思想的数据同步工具。借助于Flink CDC Connector ，它监听数据库的二进制日志文件，来获取数据库变更的数据，以此实现数据同步。
flink-connector-mysql-cdc：03 mysql-cdc常见问题汇总
最新发布
QYmufeng的博客
12-04 793
flink-cdc版本：3.2.0flink版本：flink-1.15.2mysql版本：8.0.26java版本：1.8maven版本：3.8.4。
flink-1.13.6_cdc
07-07
1. "flink-connector-mysql-cdc-2.0.1.jar"：这是Flink MySQL CDC连接器的JAR文件，版本为2.0.1。这个连接器使得Flink可以监听MySQL数据库的变化，将DML操作（如INSERT, UPDATE, DELETE）转换为事件流，然后传递给...
flink-connector-mysql-cdc 和 mysql-connector-java 依赖冲突
05-31
具体来说，如果你在项目中引入了不同版本的 mysql-connector-java 和 flink-connector-mysql-cdc，可能会导致编译错误或者运行时异常。解决这个问题的一种方法是将 mysql-connector-java 库排除掉，让 flink-...
Flink CDC系列之：深入理解MySQL CDC Connector的原理和实际应用MySQL CDC实现流式数据处理后同步到Elasticsearch数据库、StarRocks数据库
zhengzaifeidelushang的博客
08-13 622
Flink CDC系列之：深入理解MySQL CDC Connector的原理和实际应用MySQL CDC实现流式数据处理后同步到Elasticsearch数据库、StarRocks数据库
flink-connector-mysql-cdc-2.1-SNAPSHOT.jar
09-28
flink-connector-mysql-cdc-2.1的包
flink-connector-mysql-cdc-2.2
03-25
Flink CDC Connectors 2.2源码修改，适配Flink 1.14.2（MySQL）
flink-connector-mysql-cdc-2.3-SNAPSHOT.jar
06-21
mysql connector 2.3 带依赖包
flink-connector-base-1.13.2-API文档-中文版.zip
06-05
赠送jar包：flink-connector-base-1.13.2.jar； 赠送原API文档：flink-connector-base-1.13.2-javadoc.jar； 赠送源代码：flink-connector-base-1.13.2-sources.jar； 赠送Maven依赖信息文件：flink-connector-base-1.13.2.pom； 包含翻译后的API文档：flink-connector-base-1.13.2-javadoc-API文档-中文(简体)版.zip； Maven坐标：org.apache.flink:flink-connector-base:1.13.2； 标签：apache、flink、connector、base、中文文档、jar包、java； 使用方法：解压翻译后的API文档，用浏览器打开“index.html”文件，即可纵览文档内容。 人性化翻译，文档中的代码和结构保持不变，注释和说明精准翻译，请放心使用。
MySQL-CDC-Connector（Flink SQL CDC Connectors）
icefox_zhaoyt的博客
03-13 2929
Flink SQL CDC Connectors 参见文章(Flink SQL CDC 实践以及一致性分析) Flink SQL CDCFlink SQL CDC Connectors一、安装flink二、配置数据库三、flink客户端操纵1.启动flink客户端2.查看Mysql CDC 同步四、通过flink-connector-mysql-cdc（JAVA代码）监听数据变化 一、安装flink 将flink-sql-connector-mysql-cdc-1.2.0.jar 拷贝至flink l
flink-connector-mysql-cdc_2.0.2
ykqidev的博客
03-09 4509
官方参考文档：MySQL CDC Connector — Flink CDC documentationhttps://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html 1、测试步骤： 1）开启MySQL Binlog并重启MySQL 2）启动HDFS集群 [hadoop@linux100 flink-1.13.5]$ start-dfs.sh 3）启动Flink集群 [hado
【flink实战】flink-connector-mysql-cdc导致mysql连接器报类型转换错误
hiliang521的博客
06-16 2399
【flink实战】flink-connector-mysql-cdc导致mysql连接器报类型转换错误
Flink实战之 MySQL CDC
老码农的博客
03-13 2071
Flink CDC主要关注于从源数据库（如MySQL、PostgreSQL等）捕获数据变更，并将这些变更实时地提供给Flink作业进行处理。Flink CDC的核心优势在于其实时性和一致性。通过捕获数据库的增量变动记录，Flink CDC能够实时地将这些变更数据同步到Flink流处理作业中，从而实现低延迟的数据处理和分析。同时，Flink CDC还保证了数据的一致性，确保在数据处理过程中数据的准确性和完整性。为了实现这一功能，Flink社区开发了flink-cdc-connectors组件。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值