2关于flink的时间字段-ProcessTime-重新debug-通知观察方接手进行处理

最新推荐文章于 2022-09-19 17:12:09 发布
最新推荐文章于 2022-09-19 17:12:09 发布
阅读量779 收藏
点赞数
文章标签： 大数据 java runtime
原文链接：https://my.oschina.net/qiangzigege/blog/2967003
版权

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

现在，观察方如何得知有消息进来？断点在

stop in  org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run
stop at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask:533

PS:这里面有waterMark的东西！！！
---

继续断点在stop at org.apache.flink.streaming.runtime.io.StreamInputProcessor:210

cafd9b2a8e5e0abecaa10cdf3bab981e2eb.jpg

---

e23660103334b0bc3d9d863635c1d7ff9cf.jpg

---重点是下面的这个地方

8e1a26516d6910a22653ad7fb007f1eefbf.jpg

相应的断点在stop at org.apache.flink.streaming.runtime.io.StreamInputProcessor:198

然后来到了下面这个地方

  [1] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:295)
  [2] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [3] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [4] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [5] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [6] java.lang.Thread.run (Thread.java:748)

当前的输入如下：

print element
 element = "Record @ (undef) : +ymm-sms-web172,java.net.ConnectException1543832264107264417004,ERROR,190.159.11.57,null,ymm-appmetric-dev-self1_11_12136847"

当获取这个StreamRecord的时候的时间戳时，走的是下面这个代码

/**
	 * Returns the timestamp associated with this stream value in milliseconds.
	 */
	public long getTimestamp() {
		if (hasTimestamp) {
			return timestamp;
		} else {
			return Long.MIN_VALUE;//执行这个
//			throw new IllegalStateException(
//					"Record has no timestamp. Is the time characteristic set to 'ProcessingTime', or " +
//							"did you forget to call 'DataStream.assignTimestampsAndWatermarks(...)'?");
		}
	}

然后，就到了

  [1] org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows.assignWindows (TumblingProcessingTimeWindows.java:62)
  [2] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:295)
  [3] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [4] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [5] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [6] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [7] java.lang.Thread.run (Thread.java:748)

ef90c05c627b6cd4e7054ce7bcca1ec036d.jpg

 

---

5fe0ceca9b7cec6658fab80a64ea428e4bb.jpg

如果是3秒的窗口，则0~2999,3000~5999,6000~8999分别在

[0,3000) [3000,6000),[6000,9000)窗口中

34eced6b61f1e8576296b564c5ae1bf9631.jpg

f260f2e751c9486bad1b8176204f79f0722.jpg

注意：添加的时候，会触发我们自己写的聚合函数

  [1] com.ymm.computation.udf.define.LastStringInGroupSkipNull.accumulate (LastStringInGroupSkipNull.java:19)
  [2] GroupingWindowAggregateHelper$65.accumulate (null)
  [3] org.apache.flink.table.runtime.aggregate.AggregateAggFunction.add (AggregateAggFunction.scala:49)
  [4] org.apache.flink.table.runtime.aggregate.AggregateAggFunction.add (AggregateAggFunction.scala:33)
  [5] org.apache.flink.runtime.state.heap.HeapAggregatingState$AggregateTransformation.apply (HeapAggregatingState.java:132)
  [6] org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.transform (CopyOnWriteStateTable.java:459)
  [7] org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.transform (CopyOnWriteStateTable.java:337)
  [8] org.apache.flink.runtime.state.heap.HeapAggregatingState.add (HeapAggregatingState.java:104)
  [9] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:391)
  [10] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [11] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [12] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [13] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [14] java.lang.Thread.run (Thread.java:748)

然后执行到了

  [1] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:28)
  [2] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onElement (WindowOperator.java:899)
  [3] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:396)
  [4] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [5] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [6] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [7] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [8] java.lang.Thread.run (Thread.java:748)

fa6b129089cd70a226cf4b03ec596a37898.jpg

---

6bf54fd3eaec9cd01a5f01eb6957bddbf2e.jpg

我们看一下当前的线程栈 

 where
  [1] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.registerProcessingTimeTimer (WindowOperator.java:880)
  [2] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:36)
  [3] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:28)
  [4] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onElement (WindowOperator.java:899)
  [5] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:396)
  [6] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [7] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [8] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [9] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [10] java.lang.Thread.run (Thread.java:748)

到了

[1] org.apache.flink.streaming.api.operators.HeapInternalTimerService.registerProcessingTimeTimer (HeapInternalTimerService.java:202)
  [2] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.registerProcessingTimeTimer (WindowOperator.java:880)
  [3] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:36)
  [4] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:28)
  [5] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onElement (WindowOperator.java:899)
  [6] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:396)
  [7] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [8] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [9] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [10] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [11] java.lang.Thread.run (Thread.java:748)

直接看202行的源码

23cb9931a5956c20343e859fbf4976d80fd.jpg

14eeb98ff0bc432243ca3b0e27a4dfe0300.jpg

定时器什么时候会触发？观察到有1个函数org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime

我们打上断点

stop in org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime

我们看一下上下文

  [1] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:254)
  [2] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [3] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [4] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [5] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [6] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [7] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [8] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [9] java.lang.Thread.run (Thread.java:748)

07fd165122661337ff2effe75963daaae22.jpg

验证一把，打断点stop in org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService.registerTimer

  [1] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService.registerTimer (SystemProcessingTimeService.java:112)
  [2] org.apache.flink.streaming.api.operators.HeapInternalTimerService.registerProcessingTimeTimer (HeapInternalTimerService.java:218)
  [3] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.registerProcessingTimeTimer (WindowOperator.java:880)
  [4] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:36)
  [5] org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger.onElement (ProcessingTimeTrigger.java:28)
  [6] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onElement (WindowOperator.java:899)
  [7] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement (WindowOperator.java:396)
  [8] org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput (StreamInputProcessor.java:202)
  [9] org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run (OneInputStreamTask.java:103)
  [10] org.apache.flink.streaming.runtime.tasks.StreamTask.invoke (StreamTask.java:306)
  [11] org.apache.flink.runtime.taskmanager.Task.run (Task.java:703)
  [12] java.lang.Thread.run (Thread.java:748)

这样就跟我们之前注册定时器的地方对应上了！

关键的代码是

946a9f40493ba7ca8524fbc69f7fab1c884.jpg

d89ee90202c4c22aa70ae5df925f6dd4736.jpg

回到正题，我们看定时器触发时做了什么操作吧！

stop in org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run

会执行

  [1] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:254)
  [2] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [3] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [4] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [5] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [6] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [7] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [8] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [9] java.lang.Thread.run (Thread.java:748)

最终执行到

  [1] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents (WindowOperator.java:548)
  [2] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime (WindowOperator.java:505)
  [3] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:266)
  [4] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [5] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [6] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [7] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [8] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [9] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [10] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [11] java.lang.Thread.run (Thread.java:748)
Time Trigger for groupBy: (pro, throwable, level, ip), window: (TumblingGroupWindow('w$, 'SPT, 3000.millis)), select: (pro, throwable, level, ip, COUNT(*) AS count, lastStrInGroupSkipNull($f5) AS id, start('w$) AS w$start, end('w$) AS w$end, proctime('w$) AS w$proctime) -> where: (=(1, uniqueWithin100MS(pro, throwable, _UTF-16LE'ERROR', ip, w$end))), select: (pro, throwable, level, ip, count, id, w$end AS time) -> to: Row -> Sink: Kafka010JsonTableSink(pro, throwable, level, ip, count, id, time) (1/1)[1] where
  [1] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents (WindowOperator.java:548)
  [2] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime (WindowOperator.java:505)
  [3] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:266)
  [4] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [5] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [6] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [7] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [8] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [9] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [10] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [11] java.lang.Thread.run (Thread.java:748)

89c8b278ef5d579e63fc4643a2848320e81.jpg

最下面的out是print out
 out = "org.apache.flink.table.runtime.aggregate.DataStreamTimeWindowPropertyCollector@e789426"

继续

就来到了

[1] org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect (TimeWindowPropertyCollector.scala:47)
  [2] org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply (IncrementalAggregateWindowFunction.scala:74)
  [3] org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply (IncrementalAggregateTimeWindowFunction.scala:72)
  [4] org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply (IncrementalAggregateTimeWindowFunction.scala:39)
  [5] org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process (InternalSingleValueWindowFunction.java:46)
  [6] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents (WindowOperator.java:550)
  [7] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime (WindowOperator.java:505)
  [8] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:266)
  [9] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [10] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [11] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [12] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [13] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [14] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [15] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [16] java.lang.Thread.run (Thread.java:748)

有一个问题，我们的处理时间还么有加上来，别急，进入下面的函数

[1] org.apache.flink.streaming.api.operators.TimestampedCollector.setTimestamp (TimestampedCollector.java:55)
  [2] org.apache.flink.streaming.api.operators.ProcessOperator.processElement (ProcessOperator.java:64)
  [3] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator (OperatorChain.java:560)
  [4] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:535)
  [5] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:515)
  [6] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:679)
  [7] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:657)
  [8] org.apache.flink.streaming.api.operators.TimestampedCollector.collect (TimestampedCollector.java:51)
  [9] org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect (TimeWindowPropertyCollector.scala:65)
  [10] org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply (IncrementalAggregateWindowFunction.scala:74)
  [11] org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply (IncrementalAggregateTimeWindowFunction.scala:72)
  [12] org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply (IncrementalAggregateTimeWindowFunction.scala:39)
  [13] org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process (InternalSingleValueWindowFunction.java:46)
  [14] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents (WindowOperator.java:550)
  [15] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime (WindowOperator.java:505)
  [16] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:266)
  [17] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [18] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [19] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [20] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [21] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [22] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [23] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [24] java.lang.Thread.run (Thread.java:748)

再往后，，就比较简单了

  [1] org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer010.invoke (FlinkKafkaProducer010.java:355)
  [2] org.apache.flink.streaming.api.operators.StreamSink.processElement (StreamSink.java:56)
  [3] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator (OperatorChain.java:560)
  [4] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:535)
  [5] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:515)
  [6] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:679)
  [7] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:657)
  [8] org.apache.flink.streaming.api.operators.StreamMap.processElement (StreamMap.java:41)
  [9] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator (OperatorChain.java:560)
  [10] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:535)
  [11] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:515)
  [12] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:679)
  [13] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:657)
  [14] org.apache.flink.streaming.api.operators.TimestampedCollector.collect (TimestampedCollector.java:51)
  [15] org.apache.flink.table.runtime.CRowWrappingCollector.collect (CRowWrappingCollector.scala:37)
  [16] org.apache.flink.table.runtime.CRowWrappingCollector.collect (CRowWrappingCollector.scala:28)
  [17] DataStreamCalcRule$88.processElement (null)
  [18] org.apache.flink.table.runtime.CRowProcessRunner.processElement (CRowProcessRunner.scala:66)
  [19] org.apache.flink.table.runtime.CRowProcessRunner.processElement (CRowProcessRunner.scala:35)
  [20] org.apache.flink.streaming.api.operators.ProcessOperator.processElement (ProcessOperator.java:66)
  [21] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator (OperatorChain.java:560)
  [22] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:535)
  [23] org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect (OperatorChain.java:515)
  [24] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:679)
  [25] org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect (AbstractStreamOperator.java:657)
  [26] org.apache.flink.streaming.api.operators.TimestampedCollector.collect (TimestampedCollector.java:51)
  [27] org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect (TimeWindowPropertyCollector.scala:65)
  [28] org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply (IncrementalAggregateWindowFunction.scala:74)
  [29] org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply (IncrementalAggregateTimeWindowFunction.scala:72)
  [30] org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply (IncrementalAggregateTimeWindowFunction.scala:39)
  [31] org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process (InternalSingleValueWindowFunction.java:46)
  [32] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents (WindowOperator.java:550)
  [33] org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime (WindowOperator.java:505)
  [34] org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime (HeapInternalTimerService.java:266)
  [35] org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run (SystemProcessingTimeService.java:281)
  [36] java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
  [37] java.util.concurrent.FutureTask.run (FutureTask.java:266)
  [38] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201 (ScheduledThreadPoolExecutor.java:180)
  [39] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:293)
  [40] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142)
  [41] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
  [42] java.lang.Thread.run (Thread.java:748)

就是写到kafka里了，源码如下：

@Override
	public void invoke(T value, Context context) throws Exception {

		checkErroneous();

		byte[] serializedKey = schema.serializeKey(value);
		byte[] serializedValue = schema.serializeValue(value);
		String targetTopic = schema.getTargetTopic(value);
		if (targetTopic == null) {
			targetTopic = defaultTopicId;
		}

		Long timestamp = null;
		if (this.writeTimestampToKafka) {
			timestamp = context.timestamp();
		}

		ProducerRecord<byte[], byte[]> record;
		int[] partitions = topicPartitionsMap.get(targetTopic);
		if (null == partitions) {
			partitions = getPartitionsByTopic(targetTopic, producer);
			topicPartitionsMap.put(targetTopic, partitions);
		}
		if (flinkKafkaPartitioner == null) {
			record = new ProducerRecord<>(targetTopic, null, timestamp, serializedKey, serializedValue);
		} else {
			record = new ProducerRecord<>(targetTopic, flinkKafkaPartitioner.partition(value, serializedKey, serializedValue, targetTopic, partitions), timestamp, serializedKey, serializedValue);
		}
		if (flushOnCheckpoint) {
			synchronized (pendingRecordsLock) {
				pendingRecords++;
			}
		}
		producer.send(record, callback);
	}

581ac967bb8b85fc37bffb7462e20cc20ac.jpg

 

关于ProcessingTime的debug全部结束！回家吃饭！

转载于:https://my.oschina.net/qiangzigege/blog/2967003

weixin_33979363
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Flink源码解析系列--ProcessingTimeService定时器
打酱油的葫芦娃
03-24 7117
本文的Flink源码版本为: 1.15-SNAPSHOT，读者可自行从Github clone. @PublicEvolving public interface ProcessingTimeService { // 返回当前的 processing time long getCurrentProcessingTime(); /** * Registers a task to be executed when (processing) time is {@code .
Flink-时间特性 | 定义处理时间的三种方式 | 定义事件时间的三种方式
SmallScorpion
06-01 2539
时间特性（Time Attributes） 基于时间的操作（比如 Table API 和 SQL 中窗口操作），需要定义相关的时间语义和时间数据来源的信息 Table 可以提供一个逻辑上的时间字段，用于在表处理程序中，指示时间和访问相应的时间戳 时间属性，可以是每个表schema的一部分。一旦定义了时间属性，它就可以作为一个字段引用，并且可以在基于时间的操作中使用 时间属性的行为类似于常规时间戳，可以访问，并且进行计算 定义处理时间（Processing Time） 处理时间语义下，允许表处理程序根据机
Flink流计算中double类型NullPointException
learnworm的专栏
05-22 771
Flink计算中Double类型数据如何Filter过滤，使用窗口算子函数报NullPointException空指针异常，'NaN'值过滤问题
Flink的Process Function
huahuaxiaoshao的博客
05-14 899
      之前熟悉的流处理API中的转换算子是无法访问事件的时间戳信息和水位线信息的。例如：MapFunction这样的map转换算子就无法访问时间戳或者当前事件的时间。然而，在一些场景下，又需要访问这些信息。基于此，DataStream API提供了一系列的Low-Level转换算子。这些算子支持访问时间戳、watermark以及注册定时事件。还可以输出特定的一些事件，例如超时事件等。Process Function用来构建事件驱动的应用以及实
Flink 出现异常 Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.state.State
pp的博客
09-19 961
java.lang.ClassNotFoundException: org.apache.flink.runtime.state.State
【随记】Flink 时间窗口的起始时间
qq_32495629的博客
09-09 1979
Flink 时间窗口的起始时间
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar.zip
05-27
Apache Flink 是一个流行的开源大数据处理框架，而 `flink-shaded-hadoop-2-uber-2.7.5-10.0.jar.zip` 文件是针对 Flink 优化的一个特殊版本的 Hadoop 库。这个压缩包中的 `flink-shaded-hadoop-2-uber-2.7.5-10.0....
flink-table-planner-2.12-1.14.3-API文档-中文版.zip
04-07
赠送jar包：flink-table-planner_2.12-1.14.3.jar 赠送原API文档：flink-table-planner_2.12-1.14.3-javadoc.jar 赠送源代码：flink-table-planner_2.12-1.14.3-sources.jar 包含翻译后的API文档：flink-table-...
flink-1.14.0-bin-scala_2.11.tgz
07-18
2. `conf` 目录：配置文件所在，包括 `flink-conf.yaml`，用户可以在这里定制 Flink 集群的配置参数。 3. `lib` 目录：包含所有依赖的 JAR 包，用于运行 Flink 作业。 4. `docs` 目录：包含 Flink 的官方文档，帮助...
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar
03-03
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar
flink-connector-jdbc-2.12-1.14.3-API文档-中文版.zip
04-07
赠送jar包：flink-connector-jdbc_2.12-1.14.3.jar 赠送原API文档：flink-connector-jdbc_2.12-1.14.3-javadoc.jar 赠送源代码：flink-connector-jdbc_2.12-1.14.3-sources.jar 包含翻译后的API文档：flink-...
Flink 解决乱序问题之WaterMarker
胖虎儿的博客
05-26 8380
事件乱序了怎么办？用WaterMaker啦！
Flink时间窗口代码分析
追赶者的博客
08-16 2151
本代码分析以TumblingEventTimeWindows 为例，若有疏忽，请包涵。 第一步 在TumblingEventTimeWindows 类中调用assignWindows（）方法。 public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerConte...
Flink学习之窗口函数
Svenran的博客
12-01 360
Flink学习之窗口函数 window function定义了要对窗口中收集的数据做一个计算操作，主要可以分为两类： 增量聚合函数（Incremental aggregate function）： 窗口不维护原始数据，只维护中间结果，每次基于中间结果合增量数据进行聚合。 每条数据到来就计算，保持一个简单的状态。 本文主要介绍AggregateFunction。 全窗口函数（Full window function） 窗口需要维护全部原始数据，窗口触发时进行全量聚合。 先把窗口的
Flink SQL之ProcessTime与EventTime的使用
蔡政洁的博客
04-08 7608
目录（1）时间属性（2）ProcessTime（2.1）在创建表的 DDL 中定义（2.2）在 DataStream 到 Table 转换时定义（3）EventTime（3.1）在创建表的 DDL 中定义（3.2）在 DataStream 到 Table 转换时定义 （1）时间属性 处理时间 指的是执行具体操作时的机器时间 事件时间 指的是数据本身携带的时间。这个时间是在事件产生时的时间。 时间属性可以是每个表模式的一部分。当通过CREATETABLE DDL或创建表格时定义。一旦定义了时间属性，就可以
《从0到1学习Flink》—— Flink 中几种 Time 详解
http://www.54tianzhisheng.cn/
01-10 439
Github 代码仓库 https://github.com/zhisheng17/flink-learning/ 以后这个项目的所有代码都将放在这个仓库里，包含了自己学习 flink 的一些 demo 和博客 相关文章 1、《从0到1学习Flink》—— Apache Flink 介绍 2、《从0到1学习Flink》—— Mac 上搭建 Flink 1.6.0 环境并构建运行简单程序入门 3、《...
使用scala.sys.process包和系统交互
fuliang
07-19 393
在Java中我们可以使用Runtime.getRuntime().exec();来和系统交互。这个API过于底层，exec返回一个代表进程的对象，然后从中读取执行结果或者错误信息。并且如果不关心进程的输出，而没有把输入结果重定向到/dev/null，超过输出结果的buffer就会阻塞。 scala.sys.process提供了shell的和系统交互的DSL，包括执行命令、逻辑操作、重定向、管道等...
1关于flink的时间字段-ProcessTime-重新debug-读kafka->通知观察方(含正确脚本)
weixin_33910434的博客
11-30 563
2019独角兽企业重金招聘Python工程师标准>>> ...
Flink sql 中指定处理时间以及事件时间
qq_44665283的博客
10-27 3597
Flink sql 中如何指定处理时间? Flink sql 中如何指定事件时间?
flink-shaded-hadoop-2-uber-3.0.0-cdh6.3.2-13.6
最新发布
02-05
flink-shaded-hadoop-2-uber-3.0.0-cdh6.3.2-13.6 是一个 Apache Flink 的库，用于在 Flink 软件框架中使用 Hadoop 的功能。这个库被设计成 Uber JAR，意味着它包含了所有依赖的库和代码，使得其可以独立的执行，...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值