weixin_43847900的博客

读取kafka的偏移量的工具类// An highlighted blockimport java.util.Propertiesimport org.apache.kafka.clients.consumer.ConsumerRecordimport org.apache.kafka.common.TopicPartitionimport org.apache.kafka.common.serialization.StringDeserializerimport org.apache.sp

Maxwell 的使用1.Maxwell介绍Maxwell介绍Maxwell是由美国zendesk开源，用java编写的Mysql实时抓取软件，其抓取的原理也是基于binlog。2.Maxwell 和canal工具对比➢Maxwell没有canal那种server+client模式，只有一个server把数据发送到消息队列或redis。如果需要多个实例，通过指定不同配置文件启动多个进程。➢Maxwell有一个亮点功能，就是canal只能抓取最新数据，对已存在的历史数据没有办法处理。而Maxwell

下面展示一些 内联代码片。package io.github.interestinglab.waterdrop.flink.testimport java.util.Propertiesimport org.apache.flink.api.common.serialization.SimpleStringSchemaimport org.apache.flink.streaming.api.scala._import org.apache.flink.streaming.connectors.

下面展示一些 内联代码片。package io.github.flink.test/** * Flink 没有类似于 spark 中 foreach 方法，让用户进行迭代的操作。虽有对外的 * 输出操作都要利用 Sink 完成。最后通过类似如下方式完成整个任务最终输出操作。 * stream.addSink(new MySink(xxxx)) */import org.apache.flink.api.common.serialization.SimpleStringSchemai

下面展示一些 内联代码片。多线程爬取携程网酒店星级数据和经纬度import threadingfrom queue import Queueimport configimport urllibimport reimport pandas as pdnum_of_threads=10def write_fun(line): with open('酒店0502.csv','a',encoding='utf-8') as f: f.write(line)

// A code blockvar foo = 'bar';package io.github.flink.testimport java.util.Randomimport org.apache.flink.streaming.api.functions.sink.{RichSinkFunction, SinkFunction}import org.apache.flink.streaming.api.functions.source.SourceFunctionimport org.

maven 依赖 <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-elasticsearch6_2.11</artifactId> <version>1.9.2</version> </dependency>

KeyedProcessFunction 继承 AbstractRichFunction 函数Function为基础函数接口,为纯接口里面没有方法RichFunction 继承基础接口Function里面封装了到了运行环境函数,open和close 方法还有得到迭代运行时间内容public abstract class KeyedProcessFunction<K, I, O> extends AbstractRichFunction { private static fi.

BoundedOutOfOrdernessTimestampExtractor 实现了AssignerWithPeriodicWatermarks (注册周期生成Watermarks）源码如下第一次注册会生成当前最大时间戳currentMaxTimestamp =lastEmittedWatermark - 时间间隔(maxOutOfOrderness)public abstract class BoundedOutOfOrdernessTimestampExtractor<T>

// 创建index request，准备发送数据val indexRequest = Requests.indexRequest().index(“wtte”).type(“data”).source(json)flink 写入elasticsearch 报错 Caused by: java.lang.NoSuchMethodError: org.elasticsearch.action.DocWriteRequest.parent()Ljava/lang/解决 添加maven 依赖 &l

#查看集群健康Get /_cat/health/?vGet /_cat/health?v#查看节点情况GET /_cat/nodes/?v#查询各个索引状态GET /_cat/indices?v#创建一个索引PUT /movie_indexPUT /sensor#删除一个索引DELETE /movie_index#查看某一个索引的分片情况GET /_cat/shards/movie_index?vGET /_cat/shards/sensor?v#创建文.

flink 链接redis报错Caused by: redis.clients.jedis.exceptions.JedisDataException: NOAUTH Authentication required原因是没有权限(NOAUTH Authentication required),解决办法就是加上val conf = new FlinkJedisPoolConfig.Builder().setHost(“127.0.0.1”).setPort(6379).setPassword(’’

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/flink/api/common/typeinfo/TypeInformationat io.github.interestinglab.waterdrop.flink.test.RedisSinkTest.main(RedisSinkTest.scala)Caused by: java.lang.ClassNotFoundException: org.apache

// An highlighted blockpublic class ConfigKeyName { public final static String TIME_CHARACTERISTIC = "execution.time-characteristic"; public final static String BUFFER_TIMEOUT_MILLIS = "execution.buffer.timeout"; public final static String PAR

// An highlighted blockimport org.apache.flink.api.common.time.Time;import org.apache.flink.api.java.ExecutionEnvironment;import org.apache.flink.contrib.streaming.state.RocksDBStateBackend;import org.apache.flink.runtime.state.StateBackend;import o..

CREATE TABLE `poi` ( `poi_id` int NOT NULL, `name` varchar(128) DEFAULT NULL, `image` varchar(512) DEFAULT NULL, `link` varchar(512) DEFAULT NULL, `lat` float DEFAULT NULL, `lng` float DEFAULT NULL, `type` int DEFAULT NULL, `is_cnmain` int.

// An highlighted blockimport requests, re ,jsonfrom bs4 import BeautifulSoupimport randomimport jsonimport timeuser_agent_list = [ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Saf

下面展示一些 内联代码片。// An highlighted blockimport requests, re ,jsonfrom bs4 import BeautifulSoupimport randomimport jsonimport timeuser_agent_list = [ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.

// An highlighted block val otherPeopleDataset = ssc.createDataset( """{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}""" :: Nil) val otherPeople = ssc.read.json(otherPeopleDataset) otherPeople.show() val schema = ssc.re