背景
目前日志系统基本是走json log,一般公有云的log管理页面,都会有自动展开json log的功能。
在开发过程中,本地看json形式的日志有点麻烦,可读性不高。举个例子,有一行日志是这样的
{"level":"info","ts":1602161606.829082,"caller":"zap_test/main.go:44","msg":"embed","a":{"b":"bbb"},"json-str":"{\"Name\":\"a\",\"Age\":2}","stack":"/opt/go_work/src/mtest/misc/zap_test/main.go:43 (0x127d307)\n\tmain: stackOut,_ := utils.Stack(1)\n/opt/tools/go/src/runtime/proc.go:203 (0x1030585)\n\tmain: fn()\n/opt/tools/go/src/runtime/asm_amd64.s:1357 (0x105ae70)\n\tgoexit: BYTE\t$0x90\t// NOP\n"}
使用一些format工具,大概能处理成这样子:
{
"level": "info",
"ts": 1602161606.829082,
"caller": "zap_test/main.go:44",
"msg": "embed",
"a":{
"b":"bbb"
},
"json-str": "{\"Name\":\"a\",\"Age\":2}",
"stack": "/opt/go_work/src/mtest/misc/zap_test/main.go:43 (0x127d307)\n\tmain: stackOut,_ := utils.Stack(1)\n/opt/tools/go/src/runtime/proc.go:203 (0x1030585)\n\tmain: fn()\n/opt/tools/go/src/runtime/asm_amd64.s:1357 (0x105ae70)\n\tgoexit: BYTE\t$0x90\t// NOP\n"
}
但是可读性还是很差,主要有两种场景需要优化:
- 某个field是一个json字符串,标准json要求先转义,这样就很难读懂(哈哈,其实公有云也不一定有这个功能,不过我就想要)
- 某个field对应的字段,有换行符,如果可以换行显示最好
代码
参考gcp的log expand nested fields的显示样式,用python实现了类似的功能。上面提及的两种场景的解决思路如下:
- 对于json转义的字段,尝试json反序列化,如果成功了,就把这个字段变成字典,继续遍历,否则就显示字符串
- 对于有换行符的字符串,换行显示
贴个python 版本的 demo:
# 自动展开json
import json
import argparse
parser = argparse.ArgumentParser(description='manual to this script')
parser.add_argument('--path', type=str, default=None)
args = parser.parse_args()
with open(args.path) as f:
data = json.load(f)
# print(data)
indent_space = " "
# 尝试把数据展开
def expand_dict(data, layer):
brace_indent = ""
indent = ""
for i in range(layer):
indent += indent_space
for i in range(layer-1):
brace_indent += indent_space
# left brace
if layer == 1:
print(brace_indent+"{")
for k in data:
item = data[k]
if isinstance(item, dict):
print('''{}"{}": '''.format(indent, k)+"{")
expand_dict(item, layer+1)
continue
if not isinstance(item, str):
print('''{}"{}": {}'''.format(indent, k, item))
continue
if len(item) < 6:
print('''{}"{}": "{}"'''.format(indent, k, item.replace("\n", "\n"+indent+(len(k)+4)*" ", 10)))
continue
if item[0] == "{" and item[-1] == "}":
try:
new_data = json.loads(item)
except Exception as e:
print("[parse json of data:{} data err {}][ignore this line*******]".format(item, e))
print('''{}"{}": "{}"'''.format(indent, k, item.replace("\n", "\n"+indent+(len(k)+4)*" ", 10)))
continue
data[k] = new_data
print('''{}"{}": '''.format(indent, k)+"{")
expand_dict(new_data, layer+1)
else:
# normal string
print('''{}"{}": "{}"'''.format(indent, k, item.replace("\n", "\n"+indent+(len(k)+4)*" ", 10)))
# left brace
print(brace_indent+"}")
expand_dict(data, 1)
保存为expand.py, 把最上面那行log保存成一个文件,然后执行
python expand.py --path ./expand_input.json
输出
{
"level": "info"
"ts": 1602161606.829082
"caller": "zap_test/main.go:44"
"msg": "embed"
"a": {
"b": "bbb"
}
"json-str": {
"Name": "a"
"Age": 2
}
"stack": "/opt/go_work/src/mtest/misc/zap_test/main.go:43 (0x127d307)
main: stackOut,_ := utils.Stack(1)
/opt/tools/go/src/runtime/proc.go:203 (0x1030585)
main: fn()
/opt/tools/go/src/runtime/asm_amd64.s:1357 (0x105ae70)
goexit: BYTE $0x90 // NOP
"
}
这样子修改一下,基本达到预期,可读性高一些。不过需要保存log,脚本也要传参,感觉还是太不方便。准备移植为js代码,做成一个前端页面,这种小型工具在开发的时候挺方便。