Tez-之log日志内容解析

Tez-之log日志内容解析

参考网站：

https://stackoverflow.com/questions/39115712/map-reduce-logs-on-hive-tez/39144600#39144600
https://docs.qubole.com/en/latest/user-guide/engines/hive/using-hive-on-tez/hive-tez-tuning.html#understanding-log-pane

tez的日志中的内容代表什么含义呢？
The number of Tasks for each of the Mapper or Reducer Vertices is displayed in the Logs pane. The information is displayed as A (+B, -C) ÷ D where:

A implies the number of completed tasks
B implies the number of running tasks
C implies the number of failed tasks
D implies the total number of tasks

INFO  : Session is already open
INFO  : Dag name: SELECT a.Model...)
INFO  : Tez session was closed. Reopening...
INFO  : Session re-established.
INFO  :
INFO  : Status: Running (Executing on YARN cluster with App id application_14708112341234_1234)

INFO  : Map 1: -/-  Map 3: -/-  Map 4: -/-  Map 7: -/-  Reducer 2: 0/15     Reducer 5: 0/26 Reducer 6: 0/13
INFO  : Map 1: -/-  Map 3: 0/118    Map 4: 0/118    Map 7: 0/1  Reducer 2: 0/15 Reducer 5: 0/26  Reducer 6: 0/13
INFO  : Map 1: 0/118    Map 3: 0/118    Map 4: 0/118    Map 7: 0/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO  : Map 1: 0/118    Map 3: 0/118    Map 4: 0(+5)/118    Map 7: 0/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO  : Map 1: 0/118    Map 3: 0(+5)/118    Map 4: 0(+7)/118    Map 7: 0(+1)/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO  : Map 1: 0/118    Map 3: 0(+15)/118   Map 4: 0(+18)/118   Map 7: 0(+1)/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13

来解析一下吧！~~

–1、这个日志记录了在Yarn上开启了tez的会话，启动tez-appmaster
–2、这个DAG包含Map 1,Map 3,Map 4,Map 7 mappers 和Reducer 2,Reducer 5,Reducer 6 reducers
–3、每个所有的mapper和reducer都是一个vertice
–4、-/- >代表这个dag-vertice过程还没初始化
–5、0/118 >代表这个dag-vertice已经初始化完毕，总共118个map task（mapper），且执行完成的有0个
–6、0(+5)/118 >代表这个dag-vertice已经开始执行，目前完成了0个mapper，+5表示5个在进行中，如果是负数，代表失败或被kill的mapper个数
–7、Reducer 2: 0/15 >代表这个reduce阶段有15个reducer