有向无环图DAG(代码及图解)

最新推荐文章于 2023-08-07 20:04:30 发布

路途…

最新推荐文章于 2023-08-07 20:04:30 发布

阅读量7.5k

点赞数 2

分类专栏：系统层--框架spark

系统层--框架spark 专栏收录该内容

4 篇文章 1 订阅

订阅专栏

———————————————————————————————————

1、有向无环图

在图论中，如果一个有向图无法从某个顶点出发经过若干条边回到该点，则这个图是一个有向无环图（DAG图）。
因为有向图中一个点经过两种路线到达另一个点未必形成环，因此有向无环图未必能转化成树，但任何有向树均为有向无环图。
性质：有向无环图的生成树个数等于入度非零的节点的入度积。
这里写图片描述

2、代码结构

这里写图片描述

3、代码学习步鄹及方法

1、本文中涉及到了Spark的Dag和设计模式中的命令
2、Dag学习步鄹：task –> Node –> DAG –> DAGExecutor
3、设计模式命令 http://blog.csdn.net/silentwolfyh/article/details/52839495
4、图解Dag类的学习步鄹
这里写图片描述

4、重点代码讲解

下面这段代码是核心也是最难的，如何找到父节点

<span style="color:#000000"><code>  //判断Node的task节点的父节点运行状态（flase ，true）
  private <span style="color:#000088">def</span> <span style="color:#009900">getPending</span>: Option[T] = {
    _pending.find { name =>
      val parents = _nodes(name)
      !parents.exists(name => !_success.contains(name))
    }
  }</code></span>

1、nodes没有父节点时，!parents.exists() 为true
2、parents.exists() 为flase时，!parents.exists() 为true
这里写图片描述

5、代码展现

DAG.scala

<span style="color:#000000"><code><span style="color:#000088">package</span> com.yh.dag

<span style="color:#000088">import</span> java.time.{Duration, LocalDate}
<span style="color:#000088">import</span> com.yh.nodeexecutor._
<span style="color:#000088">import</span> org.slf4j.LoggerFactory
<span style="color:#000088">import</span> scala.collection.immutable.{ListMap, ListSet}

<span style="color:#880000">/**
  * Created by yuhui on 2016/8/25.
  * task --> Node --> DAG --> DAGExecutor
  */</span>

<span style="color:#000088">case</span> <span style="color:#000088">class</span> <span style="color:#4f4f4f">Node</span>[<span style="color:#4f4f4f">T</span>]<span style="color:#4f4f4f">(task: T, parent: T*)</span> {
  <span style="color:#000088">override</span> <span style="color:#000088">def</span> toString: String = {
    s<span style="color:#009900">"$task(${parent.mkString("</span>,<span style="color:#009900">")})"</span>
  }
}

<span style="color:#000088">case</span> <span style="color:#000088">class</span> <span style="color:#4f4f4f">DAG</span>[<span style="color:#4f4f4f">T</span>]<span style="color:#4f4f4f">(nodes: Node[T]*)</span>

<span style="color:#000088">case</span> <span style="color:#000088">class</span> <span style="color:#4f4f4f">DAGExecutor</span>[<span style="color:#4f4f4f">T</span>]<span style="color:#4f4f4f">(dag: DAG[T])</span> {
  <span style="color:#000088">private</span> <span style="color:#000088">val</span> LOG = LoggerFactory.getLogger(<span style="color:#000088">this</span>.getClass)
  <span style="color:#000088">private</span> <span style="color:#000088">val</span> _nodes: Map[T, Seq[T]] = dag.nodes.map(node => (node.task, node.parent.filter(_ != <span style="color:#000088">null</span>))).toMap
  <span style="color:#000088">private</span> <span style="color:#000088">var</span> _pending: Set[T] = ListSet()
  <span style="color:#000088">private</span> <span style="color:#000088">var</span> _fails = ListMap[T, String]()
  <span style="color:#000088">private</span> <span style="color:#000088">var</span> _success = Seq[T]()

  <span style="color:#880000">//判断Node的task节点的父节点运行状态（flase ，true）</span>
  <span style="color:#000088">private</span> <span style="color:#000088">def</span> getPending: Option[T] = {
    _pending.find { name =>
      <span style="color:#000088">val</span> parents = _nodes(name)
      !parents.exists(name => !_success.contains(name))
    }
  }

  <span style="color:#000088">private</span> <span style="color:#000088">def</span> fail(name: T, message: String): Unit = {
    _pending -= name
    _fails += name -> message
    <span style="color:#000088">for</span> (child <- _pending.filter(child => _nodes(child).contains(name))) {
      fail(child, s<span style="color:#009900">"依赖的任务无法执行: $name"</span>)
    }
  }

  <span style="color:#000088">private</span> <span style="color:#000088">def</span> success(name: T): Unit = {
    _pending -= name
    _success = _success :+ name
  }

  <span style="color:#000088">def</span> execute(func: T => Unit): Unit = {
    _pending = _nodes.keySet
    _fails = ListMap()
    _success = Seq()
    <span style="color:#000088">var</span> running = <span style="color:#000088">true</span>

    <span style="color:#000088">while</span> (running) {
      <span style="color:#000088">val</span> taskOpt = getPending
      <span style="color:#000088">if</span> (taskOpt.nonEmpty) {
        <span style="color:#000088">val</span> task = taskOpt.get
        <span style="color:#000088">val</span> startMills = System.currentTimeMillis()
        LOG.info(<span style="color:#009900">"start task {}"</span>, task)
        <span style="color:#000088">try</span> {
          println(<span style="color:#009900">"============="</span>)
          func(task)  <span style="color:#880000">//执行executor方法</span>
          println(<span style="color:#009900">"+++++++++++++"</span>)
          <span style="color:#000088">val</span> time = Duration.ofMillis(System.currentTimeMillis() - startMills)
          LOG.info(s<span style="color:#009900">"end task $task time=$time"</span>)
          success(task)
        } <span style="color:#000088">catch</span> {
          <span style="color:#000088">case</span> e: Throwable => fail(task, e.getMessage)
            LOG.error(e.getMessage, e)
            LOG.info(s<span style="color:#009900">"fail task $task"</span>)
        }
      } <span style="color:#000088">else</span> {
        running = <span style="color:#000088">false</span>
      }
    }

    <span style="color:#000088">for</span> (name <- _success) {
      LOG.info(s<span style="color:#009900">"success task: $name"</span>)
    }
    <span style="color:#000088">for</span> (name <- _fails) {
      LOG.info(s<span style="color:#009900">"fail task: ${name._1} - ${name._2}"</span>)
    }
  }
}

<span style="color:#000088">object</span> <span style="color:#4f4f4f">DAG</span> {
  <span style="color:#000088">val</span> allSDKDAG = <span style="color:#000088">new</span> DAG[Task](
    Node(UserDetailsExecutor, WebSdkparseExecutor),
    Node(UserTagExecutor, WebSdkparseExecutor,WebSdkparseExecutor),
    Node(WebSdkparseExecutor),
    Node(UserOverviewExecutor, WebSdkparseExecutor)
  )

  <span style="color:#000088">def</span> main(args: Array[String]): Unit = {
       DAGExecutor(allSDKDAG).execute { task =>task.executor(<span style="color:#009900">"appkey"</span>: String, LocalDate.now(), LocalDate.now())}
  }
}</code></span>

Task.scala

<span style="color:#000000"><code><span style="color:#000088">package</span> com.yh.dag

<span style="color:#000088">import</span> java.time.LocalDate
<span style="color:#000088">import</span> org.apache.spark.sql.SQLContext
<span style="color:#000088">import</span> org.slf4j.LoggerFactory

<span style="color:#880000">/**
  * Created by yuhui on 2016/12/27.
  */</span>
<span style="color:#000088">abstract</span> <span style="color:#000088">class</span> <span style="color:#4f4f4f">Task</span> {

  <span style="color:#000088">protected</span> <span style="color:#000088">val</span> LOG = LoggerFactory.getLogger(<span style="color:#000088">this</span>.getClass)

  <span style="color:#000088">def</span> executor(appkey: String, startDay: LocalDate, endDay: LocalDate): Unit

  <span style="color:#000088">def</span> run(appkey: String, startDay: LocalDate, endDay: LocalDate): Unit = {
    executor(appkey, startDay, endDay)
  }

}

<span style="color:#000088">abstract</span> <span style="color:#000088">class</span> <span style="color:#4f4f4f">Executor</span> <span style="color:#000088">extends</span> <span style="color:#4f4f4f">Task</span> <span style="color:#000088">with</span> <span style="color:#4f4f4f">SQLContextAware</span> {

  <span style="color:#000088">override</span> <span style="color:#000088">def</span> run(appkey: String, startDay: LocalDate, endDay: LocalDate)={}

}

<span style="color:#000088">trait</span> <span style="color:#4f4f4f">SQLContextAware</span> {
  implicit <span style="color:#000088">var</span> ctx: SQLContext = _
}
</code></span>

UserDetailsExecutor.scala

<span style="color:#000000"><code><span style="color:#000088">package</span> com.yh.nodeexecutor

<span style="color:#000088">import</span> java.time.LocalDate
<span style="color:#000088">import</span> com.yh.dag.Executor

<span style="color:#000088">object</span> <span style="color:#4f4f4f">UserDetailsExecutor</span> <span style="color:#000088">extends</span> <span style="color:#4f4f4f">Executor</span>{

  <span style="color:#000088">override</span> <span style="color:#000088">def</span> executor(appkey: String, startDay: LocalDate, endDay: LocalDate): Unit = {

    println(<span style="color:#009900">"++++我的UserDetailsProcessor的执行过程++++"</span>)

  }

}
</code></span>

UserOverviewExecutor.scala

<span style="color:#000000"><code><span style="color:#000088">package</span> com.yh.nodeexecutor

<span style="color:#000088">import</span> java.time.LocalDate
<span style="color:#000088">import</span> com.yh.dag.Executor

<span style="color:#880000">/**
  * Created by yuhui on 2016/12/27.
  */</span>
<span style="color:#000088">object</span> <span style="color:#4f4f4f">UserOverviewExecutor</span> <span style="color:#000088">extends</span> <span style="color:#4f4f4f">Executor</span>{

  <span style="color:#000088">override</span> <span style="color:#000088">def</span> executor(appkey: String, startDay: LocalDate, endDay: LocalDate): Unit = {
    println(<span style="color:#009900">"++++我的UserOverviewProcessor的执行过程++++"</span>)
  }

}</code></span>

UserTagExecutor.scala

<span style="color:#000000"><code><span style="color:#000088">package</span> com.yh.nodeexecutor

<span style="color:#000088">import</span> java.time.LocalDate
<span style="color:#000088">import</span> com.yh.dag.Executor

<span style="color:#880000">/**
  * Created by yuhui on 2016/12/27.
  */</span>
<span style="color:#000088">object</span> <span style="color:#4f4f4f">UserTagExecutor</span> <span style="color:#000088">extends</span> <span style="color:#4f4f4f">Executor</span>{

  <span style="color:#000088">override</span> <span style="color:#000088">def</span> executor(appkey: String, startDay: LocalDate, endDay: LocalDate): Unit = {
    println(<span style="color:#009900">"++++我的UserTagProcessor的执行过程++++"</span>)
  }

}</code></span>

WebSdkparseExecutor.scala

<span style="color:#000000"><code><span style="color:#000088">package</span> com.yh.nodeexecutor

<span style="color:#000088">import</span> java.time.LocalDate
<span style="color:#000088">import</span> com.yh.dag.Executor

<span style="color:#880000">/**
  * Created by yuhui on 2016/12/27.
  */</span>
<span style="color:#000088">object</span> <span style="color:#4f4f4f">WebSdkparseExecutor</span> <span style="color:#000088">extends</span> <span style="color:#4f4f4f">Executor</span>{

  <span style="color:#000088">override</span> <span style="color:#000088">def</span> executor(appkey: String, startDay: LocalDate, endDay: LocalDate): Unit = {
    println(<span style="color:#009900">"++++我的WebSdkparseProcessor的执行过程++++"</span>)
  }

}</code></span>

6、运行结果

<span style="color:#000000"><code>=============
++++我的WebSdkparseProcessor的执行过程++++
+++++++++++++
=============
<span style="color:#009900">++++我的UserDetailsProcessor的执行过程++++
+++++++++++++</span>
=============
++++我的UserTagProcessor的执行过程++++
+++++++++++++
=============
<span style="color:#009900">++++我的UserOverviewProcessor的执行过程++++
+++++++++++++</span>

Process finished with exit code 0</code></span>

转载于：https://blog.csdn.net/silentwolfyh/article/details/53996845

路途…

关注

2
点赞
踩
15

收藏

觉得还不错? 一键收藏
0
评论
有向无环图DAG(代码及图解)

目录：1、有向无环图2、代码结构3、代码学习步鄹及方法4、重点代码讲解5、代码展现6、运行结果———————————————————————————————————1、有向无环图在图论中，如果一个有向图无法从某个顶点出发经过若干条边回到该点，则这个图是一个有向无环图（DAG图）。因为有向图中一个点经过两种路线到达另一个点未必形成环，因此有向无环图未必能转化成...
复制链接

扫一扫