stackstorm 24. 源码分析之----stackstorm的rulesengine服务分析---4

最新推荐文章于 2023-12-23 23:48:13 发布

天地一扁舟

最新推荐文章于 2023-12-23 23:48:13 发布

阅读量263

点赞数

本文链接：https://blog.csdn.net/qingyuanluofeng/article/details/105375370

版权

本文详细分析了StackStorm的规则引擎服务（st2rulesengine）的工作原理，包括St2Timer和TriggerInstanceDispatcher的角色。St2Timer作为生产者，触发定时任务并发送消息到st2.trigger_instances_dispatch exchange，而TriggerInstanceDispatcher作为消费者，监听并处理这些消息，执行匹配规则并生成执行记录。整个过程涉及触发器实例、规则匹配、实时操作和消息传递等多个环节。

摘要由CSDN通过智能技术生成

目标:
弄清楚st2rulesengine服务原理

1和2的分析参见:
https://blog.csdn.net/qingyuanluofeng/article/details/105374863

3的分析参见:
https://blog.csdn.net/qingyuanluofeng/article/details/105374978

4的分析参见:
https://blog.csdn.net/qingyuanluofeng/article/details/105375331

5 st2actionrunner中生产者和消费者分析
st2/st2common/st2common/transport/reactor.py
# Exchange for Trigger CUD events
TRIGGER_CUD_XCHG = Exchange('st2.trigger', type='topic')

# Exchange for TriggerInstance events
TRIGGER_INSTANCE_XCHG = Exchange('st2.trigger_instances_dispatch', type='topic')
搜索:
TRIGGER_INSTANCE_XCHG
发现
def get_trigger_instances_queue(name, routing_key):
return Queue(name, TRIGGER_INSTANCE_XCHG, routing_key=routing_key)

# Used by the rules engine service
RULESENGINE_WORK_QUEUE = reactor.get_trigger_instances_queue(
name='st2.trigger_instances_dispatch.rules_engine', routing_key='#')

消费者:
def get_worker():
with Connection(transport_utils.get_messaging_urls()) as conn:
return TriggerInstanceDispatcher(conn, [RULESENGINE_WORK_QUEUE])

生产者:
class TriggerInstancePublisher(object):
def __init__(self, urls):
self._publisher = publishers.PoolPublisher(urls=urls)

def publish_trigger(self, payload=None, routing_key=None):
# TODO: We should use trigger reference as a routing key
self._publisher.publish(payload, TRIGGER_INSTANCE_XCHG, routing_key)

发送消息时:
exchange时st2.trigger_instances_dispatch，routing_key是'trigger_instance'

class TriggerDispatcher(object):
"""
This trigger dispatcher dispatches trigger instances to a message queue (RabbitMQ).
"""

def __init__(self, logger=LOG):
self._publisher = TriggerInstancePublisher(urls=transport_utils.get_messaging_urls())
self._logger = logger

class St2Timer(object):
"""
A timer interface that uses APScheduler 3.0.
"""
def __init__(self, local_timezone=None):
self._timezone = local_timezone
self._scheduler = BlockingScheduler(timezone=self._timezone)
self._jobs = {}
self._trigger_types = TIMER_TRIGGER_TYPES.keys()
self._trigger_watcher = TriggerWatcher(create_handler=self._handle_create_trigger,
update_handler=self._handle_update_trigger,
delete_handler=self._handle_delete_trigger,
trigger_types=self._trigger_types,
queue_suffix=self.__class__.__name__,
exclusive=True)
self._trigger_dispatcher = TriggerDispatcher(LOG)

分析:
TriggerInstanceDispatcher监听RULESENGINE_WORK_QUEUE队列，该队列绑定关系是:
st2.trigger_instances_dispatch--->#--->st2.trigger_instances_dispatch.rules_engine
注意#表示匹配0个或一个词组，这里实际就是fanout,也就是发送给st2.trigger_instances_dispatch这个exchange消息都会发送给
st2.trigger_instances_dispatch.rules_engine队列，然后被TriggerInstanceDispatcher类处理
而发送者是:
St2Timer聚合TriggerDispatcher，而TriggerDispatcher聚合TriggerInstancePublisher，然后
TriggerInstancePublisher中发送消息时指定:
exchange是: 'st2.trigger_instances_dispatch', routing_key是: 'trigger_instance', payload是trigger_instance(含有trace_context, trigger, payload)
综上:
在st2rulesengine服务中，St2Timer聚合的TriggerDispatcher聚合了TriggerInstancePublisher发送消息到名为'st2.trigger_instances_dispatch'的exchange,routing_key是
'trigger_instance'，payload是trigger_instance(含有trace_context, trigger, payload)
随后st2rulesengine服务中的TriggerInstanceDispatcher监听如下队列，该队列绑定关系为:
st2.trigger_instances_dispatch--->#--->st2.trigger_instances_dispatch.rules_engine
来处理上述消息。
St2Timer是生产者，TriggerInstanceDispatcher是消费者，服务处理路线为:
st2rulesengine服务的St2Timer聚合TriggerInstancePublisher发送trigger_instance消息 ->
st2rulesengine服务的TriggerInstanceDispatcher监听对应队列st2.trigger_instances_dispatch.rules_engine来处理该trigger_instance消息(处理消息实际就是获取trigger_instance中符合的rules列表，对每个rule,生成对应的liveaction和execution记录
liveaction的publish_create发送消息，指定exchange类型是topic,名字是st2.liveaction，routing_key为'create',payload,payload是LiveActionDB对象
liveaction的publish_status发送消息，指定exchange类型是topic,名字是st2.liveaction.status，routing_key是'requested',payload是LiveActionDB对象
actionexecution的publish_create发送消息，指定exchange类型是topic,名字是st2.execution,routing_key是'create'，payload是ActionExecutionDB对象
最后更新enforcement_db的execution_id为execution对象的id，相当于做了关联，然后更新rule_enforcement
)

6 st2rulesengine服务总结

6.1 St2Timer
St2Timer中会创建trigger_type，并查询数据库中每种trigger类型下的trigger列表，对每个triggger借助apscheduler触发定时任务，每次被触发的trigger叫做trigger instance，
即trigger实例。trigger instance(例如:定时任务)经过层层处理后，最终会调用TriggerInstancePublisher.publish_trigger(self, payload=None, routing_key=None)方法发送消息给st2actionrunner处理。
发送消息，具体是指定exchange类型是topic,名称是: st2.trigger_instances_dispatch,routing_key是'trigger_instance'，payload样例如下(实际可以认为payload就是一个
trigger_instance):
{'trace_context': <st2common.models.api.trace.TraceContext object at 0x3be4f90>, 'trigger': {'uid': u'trigger:core:8c13d9a3-12a3-4cff-88d3-66c804a47590:5e0acee19fa878c4fa07d596f7a893d6', 'parameters': {u'unit': u'minutes', u'delta': 5}, 'ref': u'core.8c13d9a3-12a3-4cff-88d3-66c804a47590', 'name': u'8c13d9a3-12a3-4cff-88d3-66c804a47590', 'type': u'core.st2.IntervalTimer', 'id': '5e78a2f0618fd301a43e2284', 'pack': u'core'}, 'payload': {'executed_at': '2020-03-25 10:55:01.848503+00:00', 'schedule': None}}

St2Timer也聚合了TriggerWatcher类，TriggerWatcher类指定队列为st2.trigger.watch.St2Timer-<random_string>, exchange类型是topic,名称是st2.trigger，'routing_key'是 '#'
TriggerWatcher会最终调用apscheduler的BlockingScheduler对象来添加job，job执行_emit_trigger_instance方法并携带参数trigger,
_emit_trigger_instance方法发送消息到队列上，发送消息时使用TriggerDispatcher的dispatch方法。
该dispatch方法发送消息时: exchange是'st2.trigger_instances_dispatch'，routing_key是'trigger_instance'。

搜索:
TRIGGER_CUD_XCHG = Exchange('st2.trigger', type='topic')
在st2/st2common/st2common/transport/reactor.py
class TriggerCUDPublisher(publishers.CUDPublisher):
"""
Publisher responsible for publishing Trigger model CUD events.
"""

def __init__(self, urls):
super(TriggerCUDPublisher, self).__init__(urls, TRIGGER_CUD_XCHG)

发现:
st2/st2common/st2common/persistence/trigger.py
class Trigger(ContentPackResource):
impl = trigger_access
publisher = None

@classmethod
def _get_impl(cls):
return cls.impl

@classmethod
def _get_publisher(cls):
if not cls.publisher:
cls.publisher = transport.reactor.TriggerCUDPublisher(
urls=transport_utils.get_messaging_urls())
return cls.publisher
也就是说Trigger对象publish时是使用的st2.trigger这个exchange的

topic
主题交换器，工作方式类似于组播，Exchange会将消息转发和ROUTING_KEY匹配模式相同的所有队列，比如，ROUTING_KEY为user.stock的Message会转发给绑定匹配模式为 * .stock，user.stock， * . * 和#.user.stock.#的队列。（ * 表是匹配一个任意词组，#表示匹配0个或多个词组）

参考:
https://www.jianshu.com/p/19a94c1c729b

6.2 TriggerInstanceDispatcher
TriggerInstanceDispatcher作为消费者，聚合了规则引擎类RulesEngine。TriggerInstanceDispatcher总入口就是处理消息的process方法，
根据给定的trigger,payload等信息向trigger_d_b表中创建一条trigger_instance记录(包含: trigger,payload,status,ref等)并返回,然后调用RulesEngine类对象的handle_trigger_instance方法，找到该trigger_instance对应的rules列表，判断trigger_instance中的payload是否符合criteria要求，从而得到匹配criteria的rules列表。
对每个rule,生成对应的liveaction(包含: status，parameters,runner_info,result等信息)和execution记录(包含: action(介绍待执行的action参数，元数据文件，脚本文件等)，
runner(介绍执行该action的运行器，模块，所在包等)，liveaction(action名称，传入的action执行的参数，运行情况等),result(介绍是否成功/失败，输出结果等))
然后进行了如下的消息发送:
liveaction的publish_create发送消息，指定exchange类型是topic,名字是st2.liveaction，routing_key为'create',payload,payload是LiveActionDB对象
liveaction的publish_status发送消息，指定exchange类型是topic,名字是st2.liveaction.status，routing_key是'requested',payload是LiveActionDB对象
actionexecution的publish_create发送消息，指定exchange类型是topic,名字是st2.execution,routing_key是'create'，payload是ActionExecutionDB对象
最后更新enforcement_db的execution_id为execution对象的id，相当于做了关联，然后更新rule_enforcement

6.3 服务之间关系
在st2rulesengine服务中有两个服务St2Timer和TriggerInstanceDispatcher服务，
其中St2Timer既是消费者，也作为生产者发送消息给TriggerInstanceDispatcher处理。
TriggerInstanceDispatcher接收到消息根据trigger_instance获取匹配的rules列表，并对每个rule生成liveaction和execution记录，
liveaction的publish_create发送消息，指定exchange类型是topic,名字是st2.liveaction，routing_key为'create',payload,payload是LiveActionDB对象
liveaction的publish_status发送消息，指定exchange类型是topic,名字是st2.liveaction.status，routing_key是'requested',payload是LiveActionDB对象
actionexecution的publish_create发送消息，指定exchange类型是topic,名字是st2.execution,routing_key是'create'，payload是ActionExecutionDB对象
最后更新enforcement_db的execution_id为execution对象的id，相当于做了关联，然后更新rule_enforcement。

服务之间的处理路线如下:
st2rulesengine服务的St2Timer聚合TriggerInstancePublisher发送trigger_instance消息->
st2rulesengine服务的TriggerInstanceDispatcher监听st2.trigger_instances_dispatch.rules_engine队列来处理trigger_instance消息，发送liveaction状态修改的消息,当前状态为: 'requested' --->
st2actionrunner服务的ActionExecutionScheduler监听st2.actionrunner.req队列并处理该消息,状态变更: 'requested'变为'scheduled'，发送消息 --->
st2actionrunner服务的ActionExecutionDispatcher监听st2.actionrunner.work队列并处理该消息(处理消息实际就是执行action得到execution)

6.4 关于定时器
定时器的实现是通过St2Timer类聚合了apscheduler.schedulers.background.BlockingScheduler对象，用于添加定时任务，
其中当添加定时任务时调用 _add_job方法，该方法中指定定时任务执行_emit_trigger_instance方法并传入该方法参数trigger。
所以本质还是借助apscheduler实现定时器。

6.5 关于各个实体之间的关系
整体关系如下:
trigger_type
->实例化得到trigger_d_b
->trigger_d_b在参数固定的前提下，每次传入不同的payload得到不同的trigger_instance
->找到该trigger_instance对应的rules列表，rule中持有这个trigger_d_b的ref，通过trigger的ref可以查询到对应的rule
->根据rule + trigger_instance
生成liveaction(action名称，传入的action执行的参数，result, context包含trigger_instance_id等),
生成execution(包含action,runner,liveaction,result)

1) 关于trigger
trigger_type则只指定了payload_schema,parameters_schema等参数的定义。
trigger_d_b是每种类型触发器trigger_type的实例，trigger_d_b指定了参数最终的值，以及trigger_type。
trigger_instance是包含了payload,status,trigger的type。
关系是:
trigger_type ->实例化得到trigger_d_b -> trigger_d_b在参数固定的前提下，每次传入不同的payload得到不同的trigger_instance。

trigger_instance中会有trigger的ref信息，以及发生时间，传递过来的消息，当前状态等字段。
可以根据trigger的ref信息找到trigger_d_b表中对应记录。

2) trigger与rule的关系
rule中会持有这个trigger的ref，所以通过trigger的ref可以查询到对应的rule。
每一次的触发rule的信息都写入数据库trigger_instance_d_b中，样例如下:
> db.trigger_instance_d_b.findOne({'_id': ObjectId("5e79d4d58dc0dd00ea0f362f")});
{
   "_id" : ObjectId("5e79d4d58dc0dd00ea0f362f"),
   "trigger" : "core.8c13d9a3-12a3-4cff-88d3-66c804a47590",
   "payload" : {
       "executed_at" : "2020-03-24 09:17:33.887548+00:00",
       "schedule" : null
   },
   "occurrence_time" : ISODate("2020-03-24T09:31:27.463Z"),
   "status" : "processing"
}

3) RuleEnforcer与Rule,TriggerInstance的关系
RuleEnforcer=Rule + TriggerInstance
即一次Rule执行等同于这个Rule加上此次触发实例的信息
rule_enforcement_d_b中包含trigger_instance_id, execution_id, rule的ref等信息。

4) action与liveaction的关系
liveaction是一次action执行情况，含有此次action执行的具体参数，结果等信息。
当执行action的时候，会获取action实际参数，context(含trigger_instance, rule等)，action名称，
status等在live_action表中生成一条记录，样例如下:
> db.live_action_d_b.find({'_id': ObjectId('5e7ac84b8dc0dd0113f02627')}).pretty();
{
   "_id" : ObjectId("5e7ac84b8dc0dd0113f02627"),
   "status" : "requested",
   "start_timestamp" : NumberLong("1585104727783440"),
   "action" : "email.mistral-network-check",
   "action_is_workflow" : true,
   "parameters" : {
       "cmd" : "curl busybox:80/cmd/network/check.sh%20cn",
       "email_to" : [
           "receiver@example.org"
       ],
       "email_from" : "sender@example.org"
   },
   "result" : {
   },
   "context" : {
       "trigger_instance" : {
           "id" : "5e7abd778dc0dd0113f02625",
           "name" : null
       },
       "trace_context" : {
           "id_" : "5e7abd798dc0dd0113f02626",
           "trace_tag" : "st2.IntervalTimer-8c13d9a3-12a3-4cff-88d3-66c804a47590"
       },
       "rule" : {
           "id" : "5e660433c75b50001c06b822",
           "name" : "network-check"
       },
       "user" : "admin@example.org"
   },
   "callback" : {

   },
   "runner_info" : {

   }
}

5) liveaction与actionexecution的关系
真正的执行是用actionexecution，向action_execution_d_b数据库写入此次liveaction对应的执行记录。
得到execution的id后，会更新we_url，所以继续回写数据库。
> db.action_execution_d_b.find({"_id": ObjectId('5e7ad9048dc0dd0113f02628')}).pretty();
{
   "_id" : ObjectId("5e7ad9048dc0dd0113f02628"),
   "trigger" : {
       "uid" : "trigger:core:8c13d9a3-12a3-4cff-88d3-66c804a47590:5e0acee19fa878c4fa07d596f7a893d6",
       "parameters" : {
           "unit" : "minutes",
           "delta" : 5
       },
       "type" : "core.st2.IntervalTimer",
       "pack" : "core",
       "ref" : "core.8c13d9a3-12a3-4cff-88d3-66c804a47590",
       "id" : "5e78a2f0618fd301a43e2284",
       "name" : "8c13d9a3-12a3-4cff-88d3-66c804a47590"
   },
   "trigger_type" : {
       "description" : "Triggers on specified intervals. e.g. every 30s, 1week etc.",
       "tags" : [ ],
       "parameters_schema" : {
           "additionalProperties" : false,
           "type" : "object",
           "properties" : {
               "timezone" : {
                   "type" : "string"
               },
               "unit" : {
                   "enum" : [
                       "weeks",
                       "days",
                       "hours",
                       "minutes",
                       "seconds"
                   ],
                   "required" : true
               },
               "delta" : {
                   "required" : true,
                   "type" : "integer"
               }
           }
       },
       "name" : "st2.IntervalTimer",
       "payload_schema" : {
           "type" : "object",
           "properties" : {
               "executed_at" : {
                   "default" : "2014-07-30 05:04:24.578325",
                   "type" : "string",
                   "format" : "date-time"
               },
               "schedule" : {
                   "default" : {
                       "units" : "seconds",
                       "delta" : 30
                   },
                   "type" : "object"
               }
           }
       },
       "uid" : "trigger_type:core:st2.IntervalTimer",
       "ref" : "core.st2.IntervalTimer",
       "id" : "5e6603b9a7918a0001b8a3b3",
       "pack" : "core"
   },
   "trigger_instance" : {
       "status" : "processing",
       "occurrence_time" : "2020-03-25T02:09:58.990000Z",
       "trigger" : "core.8c13d9a3-12a3-4cff-88d3-66c804a47590",
       "id" : "5e7abd778dc0dd0113f02625",
       "payload" : {
           "executed_at" : "2020-03-25 02:07:30.775464+00:00",
           "schedule" : null
       }
   },
   "rule" : {
       "description" : "Check L3 vRouter & LB & DHCP",
       "tags" : [ ],
       "ref" : "email.network-check",
       "enabled" : true,
       "trigger" : {
           "ref" : "core.8c13d9a3-12a3-4cff-88d3-66c804a47590",
           "type" : "core.st2.IntervalTimer",
           "parameters" : {
               "unit" : "minutes",
               "delta" : 5
           }
       },
       "context" : {

       },
       "criteria" : {

       },
       "action" : {
           "ref" : "email.mistral-network-check",
           "parameters" : {
               "cmd" : "curl busybox:80/cmd/network/check.sh%20cn",
               "email_to" : [
                   "receiver@example.org"
               ],
               "email_from" : "sender@example.org"
           }
       },
       "uid" : "rule:email:network-check",
       "pack" : "email",
       "type" : {
           "ref" : "standard",
           "parameters" : {

           }
       },
       "id" : "5e660433c75b50001c06b822",
       "name" : "network-check"
   },
   "action" : {
       "runner_type" : "mistral-v2",
       "name" : "mistral-network-check",
       "parameters" : {
           "email_account" : {
               "default" : "esdozer",
               "type" : "string"
           },
           "cmd" : {
               "required" : true,
               "type" : "string"
           },
           "email_to" : {
               "required" : true,
               "type" : "array"
           },
           "email_from" : {
               "type" : "string"
           },
           "timeout" : {
               "default" : 1800,
               "type" : "integer"
           }
       },
       "tags" : [ ],
       "enabled" : true,
       "entry_point" : "workflows/mistral-network-check.yaml",
       "notify" : {

       },
       "uid" : "action:email:mistral-network-check",
       "pack" : "email",
       "ref" : "email.mistral-network-check",
       "id" : "5e660431c75b50001c06b807",
       "description" : "Run network check script and output the result then send email."
   },
   "runner" : {
       "runner_module" : "mistral_v2",
       "uid" : "runner_type:mistral-v2",
       "name" : "mistral-v2",
       "enabled" : true,
       "query_module" : "mistral_v2",
       "runner_parameters" : {
           "skip_notify" : {
               "default" : [ ],
               "type" : "array",
               "description" : "List of tasks to skip notifications for."
           },
           "task" : {
               "type" : "string",
               "description" : "The name of the task to run for reverse workflow."
           },
           "context" : {
               "default" : {

               },
               "type" : "object",
               "description" : "Additional workflow inputs."
           },
           "workflow" : {
               "type" : "string",
               "description" : "The name of the workflow to run if the entry_point is a workbook of many workflows. The name should be in the format \"<pack_name>.<action_name>.<workflow_name>\". If entry point is a workflow or a workbook with a single workflow, the runner will identify the workflow automatically."
           }
       },
       "id" : "5e6603dcc75b50001c06b2ec",
       "description" : "A runner for executing mistral v2 workflow."
   },
   "liveaction" : {
       "runner_info" : {

       },
       "parameters" : {
           "cmd" : "curl busybox:80/cmd/network/check.sh%20cn",
           "email_to" : [
               "receiver@example.org"
           ],
           "email_from" : "sender@example.org"
       },
       "action_is_workflow" : true,
       "callback" : {

       },
       "action" : "email.mistral-network-check",
       "id" : "5e7ac84b8dc0dd0113f02627"
   },
   "status" : "requested",
   "start_timestamp" : NumberLong("1585104727783440"),
   "parameters" : {
       "cmd" : "curl busybox:80/cmd/network/check.sh%20cn",
       "email_to" : [
           "receiver@example.org"
       ],
       "email_from" : "sender@example.org"
   },
   "result" : {

   },
   "context" : {
       "trigger_instance" : {
           "id" : "5e7abd778dc0dd0113f02625",
           "name" : null
       },
       "trace_context" : {
           "id_" : "5e7abd798dc0dd0113f02626",
           "trace_tag" : "st2.IntervalTimer-8c13d9a3-12a3-4cff-88d3-66c804a47590"
       },
       "rule" : {
           "id" : "5e660433c75b50001c06b822",
           "name" : "network-check"
       },
       "user" : "admin@example.org"
   },
   "children" : [ ],
   "log" : [
       {
           "status" : "requested",
           "timestamp" : ISODate("2020-03-25T04:04:53.094Z")
       }
   ],
   "web_url" : "https://dozer-st2rulesengine-dfddf8d47-7c8bw/#/history/5e7ad9048dc0dd0113f02628/general"
}

6) 后续处理
liveaction中status从requested变成scheduled的时候，actionrunner服务会调用st2/st2actions/st2actions/worker.py中
消费者处理类ActionExecutionDispatcher(MessageHandler)的_run_action方法，在_run_action方法中，
从liveaction下一步到action_execution
具体是根据liveaction中的id，去action_execution中查询liveaction__id为指定id的结果。
db.action_execution_d_b.find({'liveaction.id': '5e7bca79b4a73700014aaa49'}).pretty();
将ActionExecution实际就是action的execution发布出去的时候，指定
routing_key为'update'，指定exchange类型为topic，exchange名字为'st2.execution'，

参考:
stackstorm 2.6代码