学习FLink SQL,跑了一个例子,本地运行是OK的但是无法在Flink服务器上跑。
报错详情:
Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in
the classpath.
Reason: Required context properties mismatch.
The matching candidates:
org.apache.flink.table.sources.CsvAppendTableSourceFactory
Mismatched properties:
'connector.type' expects 'filesystem', but is 'kafka'
'format.type' expects 'csv', but is 'json'
代码内容是从Kafka消费数据,分窗后插入到mysql中。
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package myflink;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.*;
import org.apache.flink.table.api.java.StreamTableEnvironment;
/**
* Skeleton for a Flink Streaming Job.
* <p>
* <p>For a tutorial how to write a Flink streaming application, check the
* tutorials and examples on the <a href="https://flink.apache.org/docs/stable/">Flink Website</a>.
* <p>
* <p>To package your application into a JAR file for execution, run
* 'mvn clean package' on the command line.
* <p>
* <p>If you change the name of the main class (with the public static void main(String[] args))
* method, change the respective entry in the POM.xml file (simply search for 'mainClass').
*/
public class StreamingJob {
public static void main(String[] args) throws Exception {
EnvironmentSettings settings = EnvironmentSettings.newInstance()
.useBlinkPlanner()
.inStreamingMode()
.build();
// TableEnvironment tEnv = TableEnvironment.create(settings);
StreamExecutionEnvironment bsEnv = StreamExecutionEnvironment.getExecutionEnvironment();
bsEnv.setParallelism(1);
bsEnv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(bsEnv, settings);
String ddl = "CREATE TABLE user_log (\n" +
" userId BIGINT,\n" +
" itemId BIGINT,\n" +
" categoryId BIGINT,\n" +
" behavior STRING,\n" +
" ts TIMESTAMP(3),\n" +
" t as TO_TIMESTAMP(FROM_UNIXTIME(itemId / 1000,'yyyy-MM-dd HH:mm:ss'))," +
" proctime as PROCTIME(),\n" +
" WATERMARK FOR t as t - INTERVAL '0.001' SECOND \n" +
") WITH (\n" +
" 'connector.type' = 'kafka',\n" +
" 'connector.version' = 'universal',\n" +
" 'connector.topic' = 'myTest1',\n" +
" 'connector.startup-mode' = 'latest-offset',\n" +
" 'connector.properties.0.key' = 'zookeeper.connect',\n" +
" 'connector.properties.0.value' = 'localhost:2181',\n" +
" 'connector.properties.1.key' = 'bootstrap.servers',\n" +
" 'connector.properties.1.value' = 'localhost:9092',\n" +
" 'update-mode' = 'append',\n" +
" 'format.type' = 'json'\n" +
// " 'format.derive-schema' = 'true'\n" +
")";
tableEnv.sqlUpdate(ddl);
String ddlMysql = "CREATE TABLE pvuv_sink (\n" +
" dt TIMESTAMP(3),\n" +
" pv BIGINT,\n" +
" uv BIGINT\n" +
") WITH (\n" +
" 'connector.type' = 'jdbc',\n" +
" 'connector.url' = 'jdbc:mysql://localhost:3306/dota2',\n" +
" 'connector.table' = 'pvuv_sink',\n" +
" 'connector.username' = 'root',\n" +
" 'connector.password' = '',\n" +
" 'connector.write.flush.max-rows' = '1'\n" +
")";
tableEnv.sqlUpdate(ddlMysql);
String dml = "INSERT INTO pvuv_sink \n" +
"SELECT\n" +
" TUMBLE_START(t, INTERVAL '1' MINUTE) AS dt,\n" +
" count(categoryId) AS pv,\n" +
" userId AS uv\n" +
"FROM user_log GROUP BY TUMBLE(t, INTERVAL '1' MINUTE), userId";
tableEnv.sqlUpdate(dml);
tableEnv.execute("SQL Job");
}
}
最后一顿查,发现是在打完的JAR包中缺少东西
应该是和SPI的有关系。
https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/dev/table/connect.html
治标方案:
在这个文件中加入了两个factory
治本方案:
2020-05-07更新
经过码哥XiangYida的提示
https://stackoverflow.com/questions/52500048/flink-could-not-find-a-suitable-table-factory-for-org-apache-flink-table-facto
在这个stackoverflow里有解决方案。
在pom.xml里增加
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
原因正如之前所说的跟SPI有关:
再去看打包好的jar里面org.apache.flink.table.factories.TableFactory已经是加好内容了
完美解决,收~