10

json log自动展开嵌套字段(expand nested fields)

 3 years ago
source link: https://blog.csdn.net/oqqYuan1234567890/article/details/108969879
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

json log自动展开嵌套字段(expand nested fields)

皿小草 2020-10-08 23:02:50 30

目前日志系统基本是走json log，一般公有云的log管理页面，都会有自动展开json log的功能。
在开发过程中，本地看json形式的日志有点麻烦，可读性不高。举个例子，有一行日志是这样的

{"level":"info","ts":1602161606.829082,"caller":"zap_test/main.go:44","msg":"embed","a":{"b":"bbb"},"json-str":"{\"Name\":\"a\",\"Age\":2}","stack":"/opt/go_work/src/mtest/misc/zap_test/main.go:43 (0x127d307)\n\tmain: stackOut,_ := utils.Stack(1)\n/opt/tools/go/src/runtime/proc.go:203 (0x1030585)\n\tmain: fn()\n/opt/tools/go/src/runtime/asm_amd64.s:1357 (0x105ae70)\n\tgoexit: BYTE\t$0x90\t// NOP\n"}

使用一些format工具，大概能处理成这样子：

{
    "level": "info",
    "ts": 1602161606.829082,
    "caller": "zap_test/main.go:44",
    "msg": "embed",
    "a":{
        "b":"bbb"
    },
    "json-str": "{\"Name\":\"a\",\"Age\":2}",
    "stack": "/opt/go_work/src/mtest/misc/zap_test/main.go:43 (0x127d307)\n\tmain: stackOut,_ := utils.Stack(1)\n/opt/tools/go/src/runtime/proc.go:203 (0x1030585)\n\tmain: fn()\n/opt/tools/go/src/runtime/asm_amd64.s:1357 (0x105ae70)\n\tgoexit: BYTE\t$0x90\t// NOP\n"
}

但是可读性还是很差，主要有两种场景需要优化：

  1. 某个field是一个json字符串，标准json要求先转义，这样就很难读懂（哈哈，其实公有云也不一定有这个功能，不过我就想要）
  2. 某个field对应的字段，有换行符，如果可以换行显示最好

参考gcp的log expand nested fields的显示样式，用python实现了类似的功能。上面提及的两种场景的解决思路如下：

  1. 对于json转义的字段，尝试json反序列化，如果成功了，就把这个字段变成字典，继续遍历，否则就显示字符串
  2. 对于有换行符的字符串，换行显示

贴个demo：

# 自动展开json
import json

import argparse
parser = argparse.ArgumentParser(description='manual to this script')
parser.add_argument('--path', type=str, default=None)
args = parser.parse_args()


with open(args.path) as f:
    data = json.load(f)


# print(data)

indent_space = "    "

# 尝试把数据展开
def expand_dict(data, layer):
    brace_indent = ""

    indent = ""
    for i in range(layer):
        indent += indent_space
    for i in range(layer-1):
        brace_indent += indent_space
    # left brace
    if layer == 1:
        print(brace_indent+"{")
    for k in data:
        item = data[k]
        if isinstance(item, dict):
            print('''{}"{}": '''.format(indent, k)+"{")
            expand_dict(item, layer+1)
            continue
        if not isinstance(item, str):
            print('''{}"{}": {}'''.format(indent, k, item))
            continue
        if len(item) < 6:
            print('''{}"{}": "{}"'''.format(indent, k, item.replace("\n", "\n"+indent+(len(k)+4)*" ", 10)))
            continue

        if item[0] == "{" and item[-1] == "}":
            try:
                new_data = json.loads(item)
            except Exception as e:
                print("[parse json of data:{} data err {}][ignore this line*******]".format(item, e))
                print('''{}"{}": "{}"'''.format(indent, k, item.replace("\n", "\n"+indent+(len(k)+4)*" ", 10)))
                continue
            data[k] = new_data
            print('''{}"{}": '''.format(indent, k)+"{")
            expand_dict(new_data, layer+1)
        else:
            # normal string
            print('''{}"{}": "{}"'''.format(indent, k, item.replace("\n", "\n"+indent+(len(k)+4)*" ", 10)))
    # left brace
    print(brace_indent+"}")
expand_dict(data, 1)

保存为expand.py， 把最上面那行log保存成一个文件，然后执行
python expand.py --path ./expand_input.json
输出

{
    "level": "info"
    "ts": 1602161606.829082
    "caller": "zap_test/main.go:44"
    "msg": "embed"
    "a": {
        "b": "bbb"
    }
    "json-str": {
        "Name": "a"
        "Age": 2
    }
    "stack": "/opt/go_work/src/mtest/misc/zap_test/main.go:43 (0x127d307)
             	main: stackOut,_ := utils.Stack(1)
             /opt/tools/go/src/runtime/proc.go:203 (0x1030585)
             	main: fn()
             /opt/tools/go/src/runtime/asm_amd64.s:1357 (0x105ae70)
             	goexit: BYTE	$0x90	// NOP
             "
}

这样子修改一下，基本达到预期，可读性高一些。不过需要保存log，脚本也要传参，感觉还是太不方便。准备移植为js代码，做成一个前端页面，这种小型工具在开发的时候挺方便。


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK