1、ListSource:
public class Demo01ListSource {
public static void main(String[] args)throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
ArrayList<String> list = new ArrayList<>();
list.add("java");
list.add("java");
list.add("java");
list.add("java");
list.add("java");
list.add("java");
/**
* 基于集合的source --- 有界流
*/
DataStream<String> listDS = env.fromCollection(list);
listDS.print();
env.execute();
}
}
2、FileSource:
(1)老版本:
DataStream<String> linesDS = env.readTextFile("flink/src/data/students.csv");
linesDS.print();
env.execute();
(2)新版本:
env.fileSource(数据源,水位线,数据名)
相比于老版本更加灵活可以指定文件的格式编码,且可以进行无界流读取,实现流批统一
有界流:
//首先构建一个fileSource
FileSource<String> fileSource = FileSource
.forRecordStreamFormat(
//指定文件的格式编码
new TextLineInputFormat("UTF-8"),
//指定读取数据的路径
new Path("flink/src/data/students.csv")
).build();
//使用fileSource
DataStream<String> linesDS = env.fromSource(fileSource, WatermarkStrategy.noWatermarks(),"fileSource");
linesDS.print();
env.execute();
无界流:
monitorContinuously()方法
//构建一个fileSource
FileSource<String> fileSource = FileSource
.forRecordStreamFormat(
//指定文件的格式编码
new TextLineInputFormat("UTF-8"),
//指定读取数据的路径
new Path("flink/src/data/stu")
)
//每隔一段时间读取目录下新的文件,构建无界流
.monitorContinuously(Duration.ofSeconds(5))
.build();
//使用fileSource
DataStream<String> linesDS = env.fromSource(fileSource, WatermarkStrategy.noWatermarks(),"fileSource");
linesDS.print();
env.execute();
3、自定义Source
(1)教学
使用方法:
addSource()方法,括号内new一个自定义Source类(该类实现SourceFunction接口)
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Integer> MyDS = env.addSource(new MySource());
MyDS.print();
env.execute();
}
自定义Source类:
类的泛型与读取数据类型保持一致
/**
* 自定义source
* 实现SourceFunction接口,实现接口中的run方法
*/
class MySource implements SourceFunction<Integer>{
/**
* flink启动的时候会执行一次,再run方法中读取外部的数据,将数据发送到下游
*/
@Override
public void run(SourceContext<Integer> ctx) throws Exception {
//将数据发送到下游
ctx.collect(100);
}
//cancel方法再任务取消的时候执行,一般用于回收资源
@Override
public void cancel() {
}
}
(2)自定义Source读取mysql
因为sql中每个字段类型不同,我们需要创建一个bean类去接收
class SC{
private String Sid;
private String Cid;
private int score;
public String getSid() {
return Sid;
}
public void setSid(String sid) {
Sid = sid;
}
public String getCid() {
return Cid;
}
public void setCid(String cid) {
Cid = cid;
}
public int getScore() {
return score;
}
public void setScore(int score) {
this.score = score;
}
public SC(String sid, String cid, int score) {
Sid = sid;
Cid = cid;
this.score = score;
}
@Override
public String toString() {
return "SC{" +
"Sid='" + Sid + '\'' +
", Cid='" + Cid + '\'' +
", score=" + score +
'}';
}
}
MySQLSource:
class MySQLSource implements SourceFunction<SC>{
/**
* run方法再任务启动的时候执行一次
*/
@Override
public void run(SourceContext ctx) throws Exception {
//加载驱动
Class.forName("com.mysql.jdbc.Driver");
//创建数据库链接
Connection con = DriverManager.getConnection("jdbc:mysql://master:3306/bigdata29?useSSL=false", "root", "123456");
//编写sql查询语句
PreparedStatement stat = con.prepareStatement("select * from SC");
//执行查询语句
ResultSet resultSet = stat.executeQuery();
//解析数据
while (resultSet.next()){
String Sid = resultSet.getString("Sid");
String Cid = resultSet.getString("Cid");
int score = resultSet.getInt("score");
//将数据发送到下游
ctx.collect(new SC(Sid,Cid,score));
}
//4、关闭数据库连接
stat.close();
con.close();
}
@Override
public void cancel() {
}
}
主类:
public class Demo04MySQLSource {
public static void main(String[] args) throws Exception{
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//使用自定义source
DataStream<SC> scDS = env.addSource(new MySQLSource());
scDS.print();
env.execute();
}
}
4、之前出现的报错问题
原因:mysql数据连接少了参数useSLL=false