一角残叶的博客

1 Flink类型系统1.1 基础数据类型1.2 Values1.3 复合数据类型1.3 泛型和其他普通的Class类型1.4 Hadoop Writables类型1.5 创建TypeInformationTypeInformation使用场景2 自定义SourceFunction案例2.1 DataSource In Batch Model2.2 基于DataStream API批流一体3 基于DataStream API实现PV，UV

1 Window多流合并1.1 应用场景每个用户的点击 Join 这个用户最近10分钟内的浏览1.2 Window Join1.3 Tumbling Window Join1.4 Sliding Window Join1.5 Session Window Join1.6 Interval Join目前都是基于EventTime1.7 Window Join的使用1.7.1 Sliding Window Join实例1.7.2 Tumbling Window Jo

1 窗口计算1.1 window抽象概念1.2 Window AssignerFlink支持的窗口类型1.3 预置的 Keyed Windows1.4 触发器一次触发

1 分布式流处理模型2 DataStream API2.1 DataStream基本数据源2.2 DataStream的转换操作2.4 KeyedStream2.5 物理分组2.6 案例3 watermark3.1 基于eventTime处理事件时间 - 最大时延 = watermark9 meiyou

1 完成两个流关联数据的清洗package test.flink.scala.scalaprojectimport java.text.SimpleDateFormatimport java.utilimport java.util.{Date, Properties}import org.apache.flink.api.common.functions.RuntimeContextimport org.apache.flink.api.common.serialization.Simpl

1 用户 id 和域名映射create table user_domain_config(id int unsigned auto_increment,user_id varchar(50) not null,domain varchar(50) not null,primary key (id))insert into user_domain_config(user_id,domain) values('8000000','v1.go2yd.com');insert into use

1 修改索引curl -XDELETE 'http://master:9200/cdn'curl -XPUT 'http://master:9200/cdn'curl -H "Content-Type: application/json" -XPOST 'http://master:9200/cdn/traffic/_mapping?pretty' -d '{"traffic":{ "properties":{ "domain":{"type":"keyword"}, "traffic

1 es 部署nosql，存储数据https://www.elastic.co/cn/downloads/elasticsearch注意要使用普通用户后台启动2 Kibana 部署启动3 flink 数据 sink 到 Elasticsearchhttps://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/connectors/elasticsearch.html <de

1 为数据添加时间水印解决数据无序问题https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/event_timestamps_watermarks.htmlpackage test.flink.scala.scalaprojectimport java.text.SimpleDateFormatimport java.util.Propertiesimport org.apache.flink.api.comm

1 使用 flink 消费 kafka 的数据1.1 初步接收数据package test.flink.scala.scalaprojectimport java.util.Propertiesimport org.apache.flink.api.common.serialization.SimpleStringSchemaimport org.apache.flink.api.scala.createTypeInformationimport org.apache.flink.strea

1 日志统计分析接入的数据类型就是日志离线：Flume==>HDFS实时：Kafka==>流处理引擎==>ES==>Kibana项目功能1）统计一分钟内每个域名访问产生的流量Flink接收Kafka的进行处理2）统计一分钟内每个用户产生的流量域名和用户是有对应关系的Flink接收Kafka的进行 + Flink读取域名和用户的配置数据 进行处理aliyun CN A E [17/Jul/2018:17:07:50 +0800] 2 223.104.18.

1 HistoryServer

1 flink scala shell./bin/start-scala-shell.sh local

1 flink 源码编译根据 hadoop 和 scala 的版本安装 JDK1.8 和 mavenmvn clean install -DskipTests -Dfast -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.14.22 flink onYARN 的2种方式https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/yarn_setup.ht

1 checkpointsink 和 source 都可以使用

1 Connectorshttps://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/connectors/1.1 hadoop<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-filesystem_2.11</artifactId> <version

1 Timehttps://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/event_time.html2 windowhttps://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html2.1 案例滚动窗口package com.imooc.flink.java.windowtest;import o

1 Table API 和 SQLhttps://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table_2.11</artifactId> <version>1.6.1</version>&lt

1 mappackage com.imooc.flink.java.demo5;import org.apache.flink.api.common.functions.FilterFunction;import org.apache.flink.api.common.functions.MapFunction;import org.apache.flink.streaming.api.datastream.DataStreamSource;import org.apache.flink.str

1 DataStream API1.1 自定义数据源无并行度的数据源package com.imooc.flink.java.demo5;import org.apache.flink.streaming.api.functions.source.SourceFunction;import java.time.Period;/** * @Description TODO * @Author tzb * @Date 2020/9/29 17:03 * @Version 1.0

1 计数器并行度为1package test.flink.scalaimport org.apache.flink.api.common.functions.RichMapFunctionimport org.apache.flink.api.scala.{ExecutionEnvironment, createTypeInformation}object CounterApp { def main(args: Array[String]): Unit = {

1 join public static void joinFunction(ExecutionEnvironment env) throws Exception { List<Tuple2<Integer, String>> info1 = new ArrayList<>(); info1.add(new Tuple2(1, "Hadoop")); info1.add(new Tuple2(2, "Flink"));

1 DataSet 和 DataStream不可变所有 flink 程序都是延迟执行

1 流处理案例package com.imooc.flink.java.demo2;import org.apache.flink.api.common.functions.FlatMapFunction;import org.apache.flink.api.java.tuple.Tuple2;import org.apache.flink.streaming.api.datastream.DataStreamSource;import org.apache.flink.streaming.a

1