Flink MySQL CDC connector 使用注意事项

注意事项

• 表要有主键

• 库名和表名不能有点号

是个 BUG，估计后续会修复。

• 表名不能有大写

也是个 BUG，估计后续会修复。

如果表名含有大写的字母，查询时日志可看到如下信息：

java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File does not exist: hdfs://hadoop/user/test/warehouse/test.db/ods_test
	at org.sparkproject.guava.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
	at org.sparkproject.guava.util.concurrent.AbstractFuture.get(AbstractFuture.java:513)
	at org.sparkproject.guava.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
	at org.sparkproject.guava.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:199)
	at org.sparkproject.guava.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2312)
	at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
	at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
	at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2044)
	at org.sparkproject.guava.cache.LocalCache.get(LocalCache.java:3952)
	at org.sparkproject.guava.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getCachedPlan(SessionCatalog.scala:158)

而底层 HDFS 路径保持了大写：

2023-05-11 18:18:27,943 INFO  [47] [org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:335)]  - Table [hdfs://hadoop/user/test/warehouse/test.db/ODS_test/ODS_test] already exists, no need to initialize the table
2023-05-11 18:18:27,943 INFO  [47] [org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:337)]  - Table update under base path hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:27,944 INFO  [47] [org.apache.hudi.common.table.HoodieTableMetaClient.updateTableAndGetMetaClient(HoodieTableMetaClient.java:507)]  - Update hoodie table with basePath hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:27,949 INFO  [47] [org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:125)]  - Loading HoodieTableMetaClient from hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:27,950 INFO  [47] [org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:366)]  - Hadoop UGI authentication : TAUTH
2023-05-11 18:18:27,993 INFO  [47] [org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:295)]  - Loading table properties from hdfs://hadoop/user/test/warehouse/test.db/ODS_test/.hoodie/hoodie.properties
2023-05-11 18:18:28,212 INFO  [47] [org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:144)]  - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from hdfs://hadoop/user/test/warehouse/test.db/ODS_test
2023-05-11 18:18:29,730 INFO  [47] [org.apache.hudi.common.table.HoodieTableMetaClient.updateTableAndGetMetaClient(HoodieTableMetaClient.java:512)]  - Finished update Table of type MERGE_ON_READ from hdfs://hadoop/user/test/warehouse/test.db/ODS_test

Flink写	Spark读
hdfs://hadoop/user/test/warehouse/test.db/ODS_test	hdfs://hadoop/user/test/warehouse/test.db/ods_test