AutoGround学习总结：Fuzz Testing（模糊测试）

I What Is Fuzz Testing

Fuzz testing is a novel way to discover security vulnerabilities or bugs in software applications. Unlike traditional software testing methodologies – SAST, DAST or IAST – fuzz testing essentially “pings” code with random (or semi-random) inputs in an effort to crash it and thus identify “faults” that would otherwise not be apparent. (GitLab Definition)

 1、Fuzzing技术首先是一种自动化技术，即软件自动执行相对随机的测试用例。因为是依靠计算机软件自动执行，所以测试效率相对人来讲远远高出几个数量级。比如，一个优秀的测试人员，一天能执行的测试用例数量最多也就是几十个，很难达到100个。而Fuzzing工具可能几分钟就可以轻松执行上百个测试用例。

2、Fuzzing技术本质是依赖随机函数生成随机测试用例，随机性意味着不重复、不可预测，可能有意想不到的输入和结果。

3、根据概率论里面的“大数定律”，只要我们重复的次数够多、随机性够强，那些概率极低的偶然事件就必然会出现。Fuzzing技术就是大数定律的典范应用，足够多的测试用例和随机性，就可以让那些隐藏的很深很难出现的Bug成为必然现象。

（Fuzzing技术总结）

II Benefits of Fuzz Testing

Because of the random nature of fuzz testing, experts say it’s the methodology most likely to find bugs missed by other tests. It’s also seen as an incredibly low-effort testing methodology, or what some like to call “set it and forget it.” Once the test harness is created fuzz testing is fully automated and will run indefinitely. It can be scaled easily by spinning up more machines and is a good choice for regression testing.

Fuzz testing is also ideal to work alongside a manual testing team as both sets of inputs will educate the other.

1. 更容易发现Bug

2. 开销少

3. 自动化

4. 可以与人工测试协作

III How to Do Fuzz Testing

The steps for fuzzy testing include the basic testing steps:

Step 1) Identify the target system

Step 2) Identify inputs

Step 3) Generate Fuzzed data

Step 4) Execute the test using fuzzy data

Step 5) Monitor system behavior

Step 6) Log defects

(According to here)

1. Examples of Fuzzers

• Mutation Based Fuzzers: 改变现有测试数据，创造新数据
• Generation Based Fuzzers: 根据某个模型从头开始定义新数据
• Protocol Based Fuzzers: 根据某个固定的协议/规范来生成测试数据

2. Fuzz-Testing Infrastructures

• AFL (American Fuzz Loop)
• libFuzzer

3. Fuzz-Testing Approaches

进行场景变异的主要目标是生成有代表性、Ego容易翻车的测试场景，有如下算法待选：

1. 遗传算法

IV Current Prospect

1. 随机场景生成

• 需要借助Scenic的Python解释器来构建AST（抽象语法树）
• 需要利用Scenic的Range，Uniform等特性

研究结果：

scenic.syntax.scenarioFromStream调用scenic.syntax.compileStream。compileStream分为七步：

1. Tokenize the input using the Python tokenizer.
2. Partition the tokens into blocks separated by import statements.
   This is done by the `partitionByImports` function.
3. Translate Scenic constructions into valid Python syntax.
   This is done by the `TokenTranslator`.
4. Parse the resulting Python code into an AST using the Python parser.
5. Modify the AST to achieve the desired semantics for Scenic.
   This is done by the `translateParseTree` function.
6. Compile and execute the modified AST.
7. After executing all blocks, extract the global state (e.g. objects).
   This is done by the `storeScenarioStateIn` function.

意味着，.scenic文件首先被tokenize，然后分块，再由TokenTranslator直接转为Python代码，转为AST，修改AST使得其成为valid Python代码，最后执行。

因此，我们可以不直接对Scenic代码进行变异，而是变异由TokenTranslator转换成的Python代码的AST。只不过这样的AST无法转换为Scenic代码。

另一种方式是，对解析过的Tokens构建AST（保证此过程可逆），然后对AST进行变异。此方法的难度较大。