FlinkCDC从mysql -kafka-es同步数据时遇到的问题

最新推荐文章于 2024-05-26 17:28:25 发布
最新推荐文章于 2024-05-26 17:28:25 发布
阅读量1.6k 收藏
点赞数
分类专栏： Flink 文章标签： flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/sghuu/article/details/112511741
版权 
2021-01-12 01:12:15,551 INFO  org.apache.kafka.clients.consumer.internals.SubscriptionState [] - [Consumer clientId=consumer-10, groupId=null] Resetting offset for partition mysql2es_t_bank_question-1 to offset 0.
2021-01-12 01:12:15,924 ERROR org.apache.flink.streaming.connectors.elasticsearch.util.NoOpFailureHandler [] - Failed Elasticsearch item request: Elasticsearch exception [type=version_conflict_engine_exception, reason=[article][appeoDKPQVu1459-qs_5ffbe1d4581a9_R2uAHCvJ]: version conflict, required seqNo [1797790], primary term [1]. current document has seqNo [1797850] and primary term [1]]
org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=version_conflict_engine_exception, reason=[article][appeoDKPQVu1459-qs_5ffbe1d4581a9_R2uAHCvJ]: version conflict, required seqNo [1797790], primary term [1]. current document has seqNo [1797850] and primary term [1]]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:510) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:421) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:135) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:198) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$3(RestHighLevelClient.java:549) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:580) [blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:621) [blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:375) [blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) [blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) [blob_p-e3fd8ced1f52c7574af952e2e6da0df8df08eb82-42012784b37d9afbb4aa86747f0bee29:4.4.6]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) [blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2021-01-12 01:12:15,930 ERROR org.apache.flink.streaming.runtime.tasks.StreamTask          [] - Error during disposal of stream operator.
java.lang.RuntimeException: An error occurred in ElasticsearchSink.
	at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.checkErrorAndRethrow(ElasticsearchSinkBase.java:380) ~[blob_p-92dd38faf9d380ca7874d869811112ae8a909875-3d17eda2919dfe5743b86aeefa0b3479:1.12.0]
	at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.close(ElasticsearchSinkBase.java:342) ~[blob_p-92dd38faf9d380ca7874d869811112ae8a909875-3d17eda2919dfe5743b86aeefa0b3479:1.12.0]
	at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:117) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:729) [flink-dist_2.11-1.11.2.jar:1.11.2]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:645) [flink-dist_2.11-1.11.2.jar:1.11.2]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:549) [flink-dist_2.11-1.11.2.jar:1.11.2]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) [flink-dist_2.11-1.11.2.jar:1.11.2]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) [flink-dist_2.11-1.11.2.jar:1.11.2]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=version_conflict_engine_exception, reason=[article][appeoDKPQVu1459-qs_5ffbe1d4581a9_R2uAHCvJ]: version conflict, required seqNo [1797790], primary term [1]. current document has seqNo [1797850] and primary term [1]]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:510) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:421) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:135) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:198) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$3(RestHighLevelClient.java:549) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:580) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:621) ~[blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:375) ~[blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) ~[blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[blob_p-e3fd8ced1f52c7574af952e2e6da0df8df08eb82-42012784b37d9afbb4aa86747f0bee29:4.4.6]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	... 1 more
2021-01-12 01:12:15,931 ERROR org.apache.flink.streaming.connectors.elasticsearch.util.NoOpFailureHandler [] - Failed Elasticsearch item request: Elasticsearch exception [type=version_conflict_engine_exception, reason=[article][appCqCo13HI5200-qs_5f9f9500c4912_cy7bT0cH0]: version conflict, required seqNo [1797803], primary term [1]. current document has seqNo [1797805] and primary term [1]]
org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=version_conflict_engine_exception, reason=[article][appCqCo13HI5200-qs_5f9f9500c4912_cy7bT0cH0]: version conflict, required seqNo [1797803], primary term [1]. current document has seqNo [1797805] and primary term [1]]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:510) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:421) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:135) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:198) ~[blob_p-21f0d8527b99290da28a61b1488a989434dbefef-99cc4e80db37d6b4bc1f923477e436d0:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$3(RestHighLevelClient.java:549) ~[blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:580) [blob_p-bc54ce7e13c1299ef100ba41e2b2254e5609270e-f1a544d31c47c4e06bbe9e8ca8a6f6d2:6.3.1]
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:621) [blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:375) [blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) [blob_p-99de036a2cd99dbecec1cc84f5d0e19032e74fa7-84c9ba8cf52dbddab768066dcef86f8d:6.3.1]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) [blob_p-e3fd8ced1f52c7574af952e2e6da0df8df08eb82-42012784b37d9afbb4aa86747f0bee29:4.4.6]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) [blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [blob_p-95aa3e6fb520191a0970a73cf09f62948ee614be-76eeb883350993edca8cd9a6eb494d0b:4.1.2]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [blob_p-f4be009e7505f6ceddf21e7960c759f413f15056-d173e56a892e118bf8d2e2d49d6d8f3a:4.4.5]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2021-01-12 01:12:15,932 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: TableSourceScan(table=[[hive, rtime_db_ods, ods_t_bank_question]], fields=[app_id, id, summary, title, audio_urls, question_type, score, correct_answer, analysis, disorder_match, state, question_library_id, question_from, weight, is_material_question, created_at, updated_at]) -> Sink: Sink(table=[hive.rtime_db_ads.ads_t_bank_question], fields=[app_id, id, summary, title, audio_urls, question_type, score, correct_answer, analysis, disorder_match, state, question_library_id, question_from, weight, is_material_question, created_at, updated_at]) (3/3) (1e8fc64c2e908821f95d8938c8ecf5fc) switched from RUNNING to FAILED.

从kafka的分区数为3，从kafka数据导入到es时，设置了并行度为3，es的表指定了主键采用update的方式写入es，启动程序后报该错误。

flink sql cdc做计算后， elastic search connector 多并发sink 会有问题，然后将并行度设置为1解决此异常，暂时没想到其他方法。

sghuu
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Flink CDC 同步mysql数据
congge_study的博客
03-24 1万+
Flink CDC 同步mysql数据
Kafka 之 offset 自动提交
weixin_49063354的博客
08-28 1930
Kafka 之 offset 自动提交 1）编写代码 需要用到的类： KafkaConsumer：需要创建一个消费者对象，用来消费数据 ConsumerConfig：获取所需的一系列配置参数 ConsuemrRecord：每条数据都要封装成一个ConsumerRecord对象 ​ 为了使我们能够专注于自己的业务逻辑，Kafka提供了自动提交offset的功能。 ​ 自动提交offset的相关参数： enable.auto.commit：是否开启自动提交offset功能 auto.commit.interva
flink cdc mysql整理与总结
最新发布
努力工作学习的程序员
05-26 1101
1、FlinkCDC 提供了对 Debezium 连接器的封装和集成，简化了配置和使用的过程，并提供了更高级的 API 和功能，例如数据格式转换、事件时间处理等。Flink CDC 使用 Debezium 连接器作为底层的实现，将其与 Flink 的数据处理能力结合起来。通过配置和使用 Flink CDC，您可以轻松地将数据库中的变化数据流转化为 Flink 的 DataStream 或 Table，并进行实时的数据处理、转换和分析。2、Flink的DataStream和SQL比较成熟和易用。
Flink cdc同步mysql
wudonglianga的专栏
09-25 1023
1.场景 两张mysql 的表通过cdc 进行同步 2. 准备条件 Flink 1.12.4 Mysql 5.7 CDC 1.2 3 依赖 [root@basenode lib]# ll 总用量 350476 -rw-r--r-- 1 root root 661717 9月 25 21:01 fastjson-1.2.78.jar -rw-r--r-- 1 root root 194725 9月 25 20:18 flink-connector-jdbc_2.11-1.12.4.jar
Kafka之Fetch offset xxx is out of range for partition xxx,resetting offset情况总结
热门推荐
敲代码的小小酥的博客
08-10 1万+
Kafka消费者出现这种日志，说明kafka消费者拉取的偏移量，在Kafka服务端没有，就会进行偏移量的重置。 为什么会没有偏移量呢？可能是因为该偏移量的数据删除了。也可能是因为程序里调用了seek方法，手动指定了偏移量，而这个偏移量在Kafka中还没有存储到数据，也会报out of range。 博主就是因为在负载均衡监听器中，获取consumer在kafka服务端的偏移量时，调用了consumer.commited()方法。该方法返回的就是下一个要拉取的偏移量，而在这里我又加了1。所以造成了out of
SpringBoot利用cancl监听mysql的binlog日志通过kafja实时同步数据到es
weixin_38071106的博客
03-04 7667
canal官网：https://github.com/alibaba/canal/wiki gitlab：https://github.com/alibaba/canal/wiki/QuickStart 下载地址：https://github.com/alibaba/canal/releases 阿里巴巴开源的 Canal 会将自己伪装成 MySQL 从节点（Slave），并从主节点（M...
flink mysql cdc到kafka
踏云追月
01-13 7602
思路： 1、创建flink mysql cdc表 2、将order join products的结果写入到kafka表中。 这样就相当于完成了，DWD中的事实表构建。写入到kafka中，等待消费构建DWS或者ADS。 主要参考https://ververica.github.io/flink-cdc-connectors/master/content/快速上手/index.html 安装flink1.3.5 启动单机版 flink ./bin/start-cluster.sh 启动flink
kafka-connect-elasticsearch-source:Kafka Connect Elasticsearch源
05-07
Kafka-connect-elasticsearch-source Kafka Connect Elasticsearch来源：从elastic-search获取数据并将其发送到kafka。 连接器仅使用严格的增量/时间字段（例如时间戳或增量ID）来获取新数据。 它支持动态模式和...
mysql-kafka-applier:用于kafka的mysql realtime-binlog
05-01
MySQL Kafka应用程序 用于kafka的mysql realtime-binlog 要求 MySQL Binlog事件1.0.0 librdkafka MySQL 5.7.X（二进制和源代码） 安装 跑步
kafka数据同步工具kafka2x-elasticsearch-master.zip
04-18
在实际部署中，kafka2x-elasticsearch-master包含了项目源码，你可以根据自己的需求进行定制和配置。主要的配置文件包括`config.properties`，这里你需要设置Kafka的bootstrap servers、Elasticsearch的节点地址、...
confluent-kafka-python:Confluent的Kafka Python客户端
04-12
Confluent的适用于Apache Kafka TM的Python... 对于我们来说，将客户端功能与核心Apache Kafka和组件保持同步是当务之急。 有关更多信息，请参见。 用法 以下是一些典型用法示例。 有关更多示例，请参阅目录或Conf
graphql-kafka-subscriptions：通过Kafka协议的Apollo graphql订阅
02-04
graphql-kafka订阅 它通过graphql-subscriptions包实现了PubSubEngine接口，并且还实现了新的AsyncIterator接口。 它使您可以将订阅管理器连接到用作发布/订阅通信渠道的单个Kafka主题。 安装 npm install graphql-...
java中SpringBoot项目定时将MySql数据同步到ES中
m0_61620621的博客
11-26 3152
项目所用依赖 <modelVersion>4.0.0</modelVersion> <artifactId>tm-shop-model</artifactId> <dependencies> <!--es--> <dependency> <groupId>org.springframework.boot</groupId>
Flink cdc 读取 kafka json 数据入PG数据库
lg4546的专栏
06-25 1140
在日常的开发中,最常用的数据格式是 JSON ,并且有的时候 JSON 的格式是非常复杂的(嵌套的格式),那在 Flink SQL 中进行解析的时候也会相当麻烦一点,下面将会演示如何在 DDL 里面定义 Map、Array、Row 类型的数据,以及在 SQL 里面如何获里面的值 数据格式如下: 以下数据完全是自己造的,没有任何实际含义 { "funcName":"test", "data":{ "snapshots":[{ "content_type":"application/...
Flink CDC 和 kafka 进行多源合并和下游同步方案
野狼e族 的博客
01-31 9612
一、前言 本文主要是针对 Flink SQL 使用 Flink CDC 无法实现多库多表的多源合并问题，以及多源合并后如何对下游 Kafka 同步更新的问题，因为目前 Flink SQL 也只能进行单表 Flink CDC 的作业操作，这会导致数据库 CDC 的连接数过多。 但是 Flink CDC 的 DataStream API 是可以进行多库多表的同步操作的，本文希望利用 Flink CDC 的 DataStream API 进行多源合并后导入一个总线 Kafka，下游只需连接总线 kafka 就可以
【FLINK】Kafka数据源通过Flink-cdc进行实时数据同步
weixin_37788200的博客
06-11 2487
通过flink-cdc实现kafka数据源的数据实时同步到目标数据库，并对目标库进行增删改操作
flink cdc 做的kafka实时数仓 如何支持聚合操作
yy的博客
08-05 937
// kafka connector怎么支持聚合操作? 他不是只支持append sink吗? 那他是怎么更新聚合结果的? // 聚合后数据存储到kafka就是累死 {"key1":"number1"}这种k,v的统计结果吗? // 基于cdc的kafka实时数仓每层的表再kafka中的数据是怎么样的.如何存储,存储格式. // 基于flinkcdc的kafka实时数仓 // kafka实时数仓每层数据展示解读 // kafka连接器 upsert-kafka连接器 连接器支持的source si
FlinkCDC写入kafka计算后写入写出hbase-工作实例
someInNeed的博客
03-25 2970
POM文件 <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you.
CDC 数据入湖方案：Flink CDC ＞ Kafka ＞ Hudi
Laurence的技术博客
02-20 724
本方案的技术链路为：使用 Flink CDC 将 MySQL 的 CDC 数据 （Json 格式）接入到 Kafka ，然后通过 Flink Hudi Connector 将摄取的 CDC 数据写入到 Hudi 表中。文本是本博客的 CDC 数据入湖系列方案中最为基础的一套堆栈，架构上也比较简单，适合作为 POC 快速搭建 CDC 实时处理链路。如果寻求更加适用于生产环境的解决方案，请参考
flinkCDC 从mysql同步到kafka 的流程
06-11
Flink CDC 从 MySQL 同步到 Kafka 的流程如下： 1. 配置 MySQL 数据源：在 Flink CDC 中，使用 JDBC Connector 连接 MySQL 数据库，并配置相应的参数，例如数据库连接 URL、用户名、密码等。 2. 配置 Kafka 数据接收器：使用 Kafka Connector 连接 Kafka，配置相应的参数，例如 Kafka 主题、Kafka Broker 地址等。 3. 创建 Flink CDC 任务：使用 Flink SQL 或 Flink Table API 创建 Flink CDC 任务，并配置相应的数据源和数据接收器。 4. 启动 Flink CDC 任务：使用 Flink 自带的命令行工具或 Web UI 启动 Flink CDC 任务，开始同步 MySQL 数据到 Kafka 中。 具体的步骤如下： 1. 下载并安装 Flink：从 Flink 官网下载并安装 Flink。 2. 配置 MySQL 数据源：在 Flink 的 conf 目录下创建一个新的文件，例如 mysql.properties，配置 MySQL 数据源相关的参数，例如： ``` connector.class = jdbc connector.url = jdbc:mysql://localhost:3306/test?useSSL=false connector.table = my_table connector.username = root connector.password = password ``` 3. 配置 Kafka 数据接收器：在 Flink 的 conf 目录下创建一个新的文件，例如 kafka.properties，配置 Kafka 数据接收器相关的参数，例如： ``` connector.class = kafka connector.topic = my_topic connector.properties.bootstrap.servers = localhost:9092 ``` 4. 创建 Flink CDC 任务：使用 Flink SQL 或 Flink Table API 创建 Flink CDC 任务，例如： ``` CREATE TABLE my_table ( id INT, name STRING, age INT ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://localhost:3306/test?useSSL=false', 'table-name' = 'my_table', 'username' = 'root', 'password' = 'password' ); CREATE TABLE my_topic ( id INT, name STRING, age INT ) WITH ( 'connector' = 'kafka', 'topic' = 'my_topic', 'properties.bootstrap.servers' = 'localhost:9092' ); INSERT INTO my_topic SELECT * FROM my_table; ``` 5. 启动 Flink CDC 任务：使用 Flink 自带的命令行工具或 Web UI 启动 Flink CDC 任务，例如： ``` ./bin/flink run -c com.example.MyCDCJob /path/to/my/cdc/job.jar ``` 通过以上步骤，就可以实现从 MySQL 同步数据到 Kafka 中的流程。需要注意的是，Flink CDC 可以根据实际的需求进行调整，例如任务并行度、缓冲区大小等参数。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值