【无标题】

摘要 我们介绍了VERFIAI，这是一个用于正式设计和分析包括人工智能（AI）和机器学习（ML）组件的系统的软件工具包。VERIFAI特别解决了将形式化方法应用于ML组件的挑战，例如基于深度神经网络的感知系统以及包含这些组件的系统，并在存在环境不确定性的情况下建模和分析系统行为。我们描述了VERIFAI的初始版本，它以基于仿真的验证和合成为中心，由正式模型和规范指导。我们给出了几个用例的示例，包括时序逻辑伪造、基于模型的系统模糊测试、参数合成、反例分析和数据集扩充。

The increasing use of artificial intelligence (AI) and machine learning (ML) in systems,
including safety-critical systems, has brought with it a pressing need for formal methods and tools for their design and verification.However, AI/ML-based systems, such as
autonomous vehicles, have certain characteristics that make the application of formal
methods very challenging. We mention three key challenges here; see Seshia et al. [23]
for an in-depth discussion.First, several uses of AI/ML are for perception, the use of
computational systems to mimic human perceptual tasks such as object recognition and
classification, conversing in natural language, etc. For such perception components,writing a formal specification is extremely difficult, if not impossible. Additionally, the
signals processed by such components can be very high-dimensional, such as streams
of images or LiDAR data.Second, machine learning being a dominant paradigm in
AI, formal tools must be compatible with the data-driven design flow for ML and also
be able to handle the complex, high-dimensional structures in ML components such as
deep neural networks.Third, the environments in which AI/ML-based systems operate can be very complex, with considerable uncertainty even about how many (which)
agents are in the environment (both human and robotic), let alone about their intentions
and behaviors.例如，考虑建立自动驾驶汽车必须运行的城市交通环境模型的困难。事实上，AI/ML经常被引入到这些系统中，正是为了处理这种复杂性和不确定性！From a formal methods perspective, this makes it very hard to create realistic environment models with respect to which one can perform verification or synthesis.

In this paper, we introduce the VERIFAI toolkit, our initial attempt to address
the three core challenges—perception, learning, and environments—that are outlined
above. VERIFAI takes the following approach:
Perception: 感知组件将具体的特征空间（例如像素）映射到输出，例如分类、预测或状态估计。为了解决感知组件缺乏规范的问题，VERFIAI使用系统级规范在闭环系统的上下文中对其进行分析。此外，为了扩展到复杂的高维特征空间，VERIAI在抽象特征空间（或语义特征空间）[10]上运行，该空间描述了感知环境的语义方面，而不是像素等原始特征。

Learning: VERIFAI aims to not only analyze the behavior of ML components but
also use formal methods for their (re-)design.为此，它提供了以下功能：（i）设计用于训练和测试的数据集[9]，（ii）分析反例以通过ML模型洞察错误，以及（iii）合成参数，包括训练算法的超参数和ML模型参数。

Environment Modeling: 由于对基于AI的系统的环境进行详尽的建模可能很困难（如果不是不可能的话），因此VERIFAI旨在提供一些方法来捕获设计者对环境的假设，包括ML组件所做的分布假设，并以直观、说明性的方式描述抽象特征空间。To this end, VERIFAI provides users with
SCENIC [12,13], a probabilistic programming language for modeling environments.
SCENIC, combined with a renderer or simulator for generating sensor data, can produce semantically-consistent input for perception components.

VERIFAI is currently focused on AI-based cyber-physical systems (CPS), although
its basic ideas can also be applied to other AI-based systems.作为一种务实的选择，我们侧重于基于仿真的验证，其中模拟器被视为一个黑匣子，以便广泛适用于工业中使用的各种模拟器。VERIFAI的输入是一个“闭环”CPS模型，包括一个由环境模型验证的基于AI的CPS系统组成，以及闭环模型上的一个属性。The AI-based CPS typically comprises a perception component (not necessarily based on ML), a planner/controller, and the plant (i.e., the system under control).鉴于这些，VERIFAI提供了以下用例：（1）时序逻辑伪造；（2） 基于模型的模糊测试；（3） 反例引导数据增强；（4） 反例（误差表）分析；（5） 超参数综合，（6）模型参数综合。VERIFAI的新颖之处在于，它是第一个以集成方式提供这套用例的工具，由抽象特征空间的通用表示统一，并在该特征空间上附带建模语言和搜索算法，所有这些都在模块化实现中提供。The algorithms and formalisms in VERIFAI are presented in papers published by the authors in other venues (e.g., [7–10,12,15,22]).The problem of temporal-logic falsification or simulation-based verification of CPS models is well studied and several tools exist (e.g. [3,11]); our work was the first to extend these techniques to CPS models with ML components [7,8]. Work on verification of ML components, especially neural networks (e.g., [14,26]), is complementary to the system-level analysis performed by VERIFAI. Fuzz testing based on formal models is common in software engineering (e.g. [16]) but our work is unique in the CPS context.Similarly, property-directed parameter synthesis has also been studied in the formal methods/CPS community, but our work is the first to apply these ideas to the synthesis of hyper-parameters for ML training and ML model parameters. Finally, to our knowledge, our work on augmenting training/test data sets [9], implemented in VERIFAI, is the first use of formal techniques for this purpose.在Sect2，我们描述了该工具的结构，以提供上述功能。Sect3通过自动驾驶领域的示例说明了用例。

2 VERIFAI Structure and Operation