Golang 开启黑盒测试
黑盒测试必要性
单元测试具有局限性,因为单元测试用例的输入是开发人员增加的。黑盒测试的好处是可以识别并增加边缘性的测试case。
测试Demo
开发代码
package main
import "fmt"
func main() {
input := "The quick brown fox jumped over the lazy dog"
rev := Reverse(input)
doubleRev := Reverse(rev)
fmt.Printf("original: %q\n", input)
fmt.Printf("reversed: %q\n", rev)
fmt.Printf("reversed again: %q\n", doubleRev)
}
// Reverse 字符串逆序函数
func Reverse(s string) string {
b := []byte(s)
for i, j := 0, len(b)-1; i < len(b)/2; i, j = i+1, j-1 {
b[i], b[j] = b[j], b[i]
}
return string(b)
}
单元测试
在同级目录下新增 reverse_test.go 文件 代码如下
package main
import (
"testing"
)
func TestReverse(t *testing.T) {
testcases := []struct {
in, want string
}{
{"Hello, world", "dlrow ,olleH"},
{" ", " "},
{"!12345", "54321!"},
}
for _, tc := range testcases {
rev := Reverse(tc.in)
if rev != tc.want {
t.Errorf("Reverse: %q, want %q", rev, tc.want)
}
}
}
进行单元测试 输出如下
$ go test
PASS
ok example/fuzz 0.013s
黑盒测试
在reverse_test.go文件里面增加以下函数,golang中的黑盒测试同样也存在部分限制:
- 单元测试中需要对输出结果做出判断 并验证是否跟预期一致, 如 Hello 的输出结果为 olleH
- 黑盒测试不能判断期望的输出结果 因为无法控制输入参数
无论如何,针对Reverse函数也有一些基本的属性可以进行验证,如下两点:
- 反转2次后需要跟原始的数据一致
- UTF-8编码的字符串 需要正常执行
需要注意黑盒测试的语法跟单元测试也有一点不同:
- 函数名称从 TestXxx(单元测试) 变成 FuzzXxx(黑盒测试),参数由 *testing.T 变成 *testing.F
- 在期望执行 t.Run 的地方,将会有 f.Fuzz 执行目标函数。由 f.Add 函数将输入放入到待测试的数据集中
func FuzzReverse(f *testing.F) {
testcases := []string{"Hello, world", " ", "!12345"}
for _, tc := range testcases {
f.Add(tc) // Use f.Add to provide a seed corpus
}
f.Fuzz(func(t *testing.T, orig string) {
rev := Reverse(orig)
doubleRev := Reverse(rev)
if orig != doubleRev {
t.Errorf("Before: %q, after: %q", orig, doubleRev)
}
if utf8.ValidString(orig) && !utf8.ValidString(rev) {
t.Errorf("Reverse produced invalid UTF-8 string %q", rev)
}
})
}
执行黑盒测试, 发现并没有执行成功,中途报错。
PS C:\Users\User36\GolandProjects\awesomeProject\fuzz\day01> go test -fuzz=Fuzz
fuzz: elapsed: 0s, gathering baseline coverage: 0/3 completed
fuzz: elapsed: 0s, gathering baseline coverage: 3/3 completed, now fuzzing with 8 workers
fuzz: minimizing 34-byte failing input file
fuzz: elapsed: 0s, minimizing
--- FAIL: FuzzReverse (0.09s)
--- FAIL: FuzzReverse (0.00s)
reverse_test.go:36: Reverse produced invalid UTF-8 string "\xb6\xd5"
Failing input written to testdata\fuzz\FuzzReverse\8fd8b02db1ce4b6cd20b03c7447f9b21a278f5931241b61e0437b14b7e5a6b0e
To re-run:
go test -run=FuzzReverse/8fd8b02db1ce4b6cd20b03c7447f9b21a278f5931241b61e0437b14b7e5a6b0e
FAIL
exit status 1
FAIL awesomeProject/fuzz/day01 0.143s
在当前目录下 testdata/fuzz/FuzzReverse/8fd8b02db1ce4b6cd20b03c7447f9b21a278f5931241b61e0437b14b7e5a6b0e文件,内容:
go test fuzz v1
string("泃")
再次执行单元测试命令,发现也是失败的,golang的内部机制决定了 只要测试失败 应该Debug问题
PS C:\Users\User36\GolandProjects\awesomeProject\fuzz\day01> go test
--- FAIL: FuzzReverse (0.00s)
--- FAIL: FuzzReverse/8fd8b02db1ce4b6cd20b03c7447f9b21a278f5931241b61e0437b14b7e5a6b0e (0.00s)
reverse_test.go:36: Reverse produced invalid UTF-8 string "\xb6\xd5"
FAIL
exit status 1
FAIL awesomeProject/fuzz/day01 0.031s
问题跟踪
考虑到字符串反转函数的输入有可能是中文,因此需要做一些调整,代码如下
func Reverse(s string) string {
fmt.Printf("input: %q\n", s)
r := []rune(s)
fmt.Printf("runes: %q\n", r)
for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
r[i], r[j] = r[j], r[i]
}
return string(r)
}
利用之前的测试用例 ,再次执行 发现已经解决了,完美
PS C:\Users\User36\GolandProjects\awesomeProject\fuzz\day01> go test -run=FuzzReverse/8fd8b02db1ce4b6cd20b03c7447f9b21a278f5931241b61e0437b14b7e5a6b0e
input: "ն"
runes: ['ն']
input: "ն"
runes: ['ն']
PASS
ok awesomeProject/fuzz/day01 0.032s
等等 不要高兴的太早了,再次执行黑盒测试脚本,特么的 问题又出现了, 日志显示还是一些UTF8编码的字符串搞得鬼
PS C:\Users\User36\GolandProjects\awesomeProject\fuzz\day01> go test -fuzz=Fuzz
input: "Hello, world"
runes: ['H' 'e' 'l' 'l' 'o' ',' ' ' 'w' 'o' 'r' 'l' 'd']
input: " "
runes: [' ']
input: "!12345"
runes: ['!' '1' '2' '3' '4' '5']
fuzz: elapsed: 0s, gathering baseline coverage: 0/10 completed
fuzz: minimizing 46-byte failing input file
failure while testing seed corpus entry: FuzzReverse/0df90bdec6befb47c1c0a8064e9af09e83c4f50f61c948026f3a46152302792d
fuzz: elapsed: 0s, gathering baseline coverage: 4/10 completed
--- FAIL: FuzzReverse (0.05s)
--- FAIL: FuzzReverse (0.00s)
reverse_test.go:33: Before: "\xbc", after: "�"
FAIL
exit status 1
FAIL awesomeProject/fuzz/day01 0.079s
再次修改 Reverse 函数,判断为UTF8的字符串直接返回 注意修改后的函数返回了两个值了
func Reverse(s string) (string, error) {
if !utf8.ValidString(s) {
return s, errors.New("input is not valid UTF-8")
}
r := []rune(s)
for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
r[i], r[j] = r[j], r[i]
}
return string(r), nil
}
再次执行黑盒测试脚本,发现会一直不停的执行下去,知道按住ctrl-C
中断执行
PS C:\Users\User36\GolandProjects\awesomeProject\fuzz\day01> go test -fuzz=Fuzz
fuzz: elapsed: 0s, gathering baseline coverage: 0/10 completed
fuzz: elapsed: 0s, gathering baseline coverage: 10/10 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 391042 (130032/sec), new interesting: 25 (total: 35)
fuzz: elapsed: 6s, execs: 1068358 (226007/sec), new interesting: 27 (total: 37)
fuzz: elapsed: 9s, execs: 1574658 (167653/sec), new interesting: 27 (total: 37)
fuzz: elapsed: 12s, execs: 2242408 (223348/sec), new interesting: 28 (total: 38)
也可以 执行黑盒测试时指定测试时间
PS C:\Users\User36\GolandProjects\awesomeProject\fuzz\day01> go test -fuzz=Fuzz -fuzztime 10s
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
fuzz: elapsed: 0s, gathering baseline coverage: 42/42 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 601251 (200011/sec), new interesting: 1 (total: 43)
fuzz: elapsed: 6s, execs: 1208050 (202093/sec), new interesting: 1 (total: 43)
fuzz: elapsed: 9s, execs: 1810927 (200932/sec), new interesting: 1 (total: 43)
fuzz: elapsed: 10s, execs: 2031601 (201306/sec), new interesting: 1 (total: 43)
PASS
ok awesomeProject/fuzz/day01 10.165s
最终代码
main.go
package main
import (
"errors"
"fmt"
"unicode/utf8"
)
func main() {
input := "The quick brown fox jumped over the lazy dog"
rev, _ := Reverse(input)
doubleRev, _ := Reverse(rev)
fmt.Printf("original: %q\n", input)
fmt.Printf("reversed: %q\n", rev)
fmt.Printf("reversed again: %q\n", doubleRev)
}
// Reverse 字符串逆序函数
func Reverse(s string) (string, error) {
if !utf8.ValidString(s) {
return s, errors.New("input is not valid UTF-8")
}
r := []rune(s)
for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
r[i], r[j] = r[j], r[i]
}
return string(r), nil
}
reverse_test.go
package main
import (
"testing"
"unicode/utf8"
)
func TestReverse(t *testing.T) {
testcases := []struct {
in, want string
}{
{"Hello, world", "dlrow ,olleH"},
{" ", " "},
{"!12345", "54321!"},
}
for _, tc := range testcases {
rev, _ := Reverse(tc.in)
if rev != tc.want {
t.Errorf("Reverse: %q, want %q", rev, tc.want)
}
}
}
func FuzzReverse(f *testing.F) {
testcases := []string{"Hello, world", " ", "!12345"}
for _, tc := range testcases {
f.Add(tc) // Use f.Add to provide a seed corpus
}
f.Fuzz(func(t *testing.T, orig string) {
rev, _ := Reverse(orig)
doubleRev, _ := Reverse(rev)
if orig != doubleRev {
t.Errorf("Before: %q, after: %q", orig, doubleRev)
}
if utf8.ValidString(orig) && !utf8.ValidString(rev) {
t.Errorf("Reverse produced invalid UTF-8 string %q", rev)
}
})
}
总结一下,确实比单元测试香多了