This article introduces jsonvalue Library , this is the first multi-functional and comprehensive Go library I developed on Github. At present, it is mainly used in the development of Tencent's future community to replace map[string]interface {}.

Why develop this library?

Go is a cutting edge in background development. Go engineers will come into contact with the "encoding/json" library at an early stage: for JSON data of known format, the typical method of go is to define a struct to serialize and deserialize (marshal/unmarshal).

However, for unknown formats or cases where it is inconvenient to fix formats, the typical solution is to use map[string]interface {}. However, in practical application, this scheme has some shortcomings.

Shortcomings of map[string]interface {}

In some cases, we really need to use map[string]interface {} to parse and process JSON, which often occurs in the JSON logic that needs to process all or part of the unknown format in middleware, gateway, proxy server and so on.

It is inconvenient to judge the value type

Suppose I have a map after unmarshal: M: = map [string] interface {} {}. When I want to judge whether a key value pair (such as "aNum") is a number, I need to judge two situations respectively:

v, exist := m["aNum"]
if false == exist {
    return errors.New("aNum does not exist")
}

n, ok := v.(float64)
if false == ok {
    return fmt.Errorf("'%v' is not a number", v)
}

It is inconvenient to get deep fields

For example, Tencent cloud API has several layers of nested data return format, as shown below:

{
    "Response": {
        "Result": {
            "//"Here, I assume that I need to find the following field:",
            "AnArray": [
                {
                    "SomeString": "Hello, world!"
                }
            ]
        }
    }
}

When the interface fails, it will return:

{
    "Response": {
        "Error": {
            "Code": "error code",
            "Message": "error message"
        }
    }
}

Suppose that in normal logic, we must use map[string]interface {} to parse the data because of some factors. It's hard to judge the response Result. AnArray[0]. When the value of somestring is, because we can't trust the data of the opposite end 100% (the server may be hijacked, crashed, invaded, etc.), we need to check each field. Therefore, the complete code is as follows:

    m := map[string]interface{}{}
    // Some unmarshal actions 
    // ......
    //
    // First, judge whether the interface is wrong
    var response map[string]interface{}
    var ok bool
    //
    // First, get the Response information
    if v, exist := m["Response"]; !exist {
        return errors.New("missing Response")
    //
    // Then you need to judge whether the Response is an object type
    } else if response, ok = v.(map[string]interface{}); !ok {
        return errors.New("Response is not an object")
    //
    // Then you need to determine whether there is an Error field
    } else if e, exist = response["Error"]; exist {
        return fmt.Errorf("API returns error: %_+v", e)
    }
    //
    // Then judge the specific value
    // First, you need to judge whether there is a Result field
    if resultV, exist := response["Result"]; !exist {
        return errors.New("missing Response.Result")
    //
    // Then judge whether the Result field is object
    } else if result, ok := resultV.(map[string]interface{}); !ok {
        return errors.New("Response.Result is not an object")
    //
    // Then get the AnArray field
    } else if arrV, exist := resule["AnArray"]; !exist {
        return errors.New("missing Response.Result.AnArray")
    //
    // Then judge the type of AnArray
    } else if arr, ok := arrV.([]interface{}); !ok {
        return errors.New("Response.Result.AnArray is not an array")
    // Then judge the length of AnArray
    } else if len(arr) < 1 {
        return errors.New("Response.Result.AnArray is empty")
    //
    // Then get the first member of the array and judge whether it is an object
    } else if firstObj, ok := arr[0].(map[string]interface{}); !ok {
        return errors.New("Response.Result.AnArray[0] is not an object")
    //
    // Then get the SomeString field
    } else if v, exist := firstObj["SomeString"]; !exist {
        return errors.New("missing Response.Result.AnArray[0].SomeString")
    //
    // Then judge the type of SomeString
    } else if str, ok := v.(string); !ok {
        return errors.New("Response.Result.AnArray[0].SomeString is not a string")
    //
    // It is finally completed.
    } else {
        fmt.Printf("SomeString = '%s'\n", str)
        return nil
    }

I don't know how readers feel. Anyway, I'm going to lift the table

Marshall () is inefficient

In Unmarshal(), the deserialization efficiency of map[string]interface {} type is slightly lower than that of struct, but it is roughly the same. But in Marshall (), the difference between the two is very obvious. According to a later test scheme, map takes about five times as long as struct. A serialization / deserialization operation takes twice as long.

jsonvalue function introduction

Jsonvalue is a Go language library for dealing with JSON. The part of parsing JSON text is based on jsonparser realization. The parsing of specific content, CURD of JSON and serialization are implemented independently.

First, let's introduce the basic usage

Deserialization

Jsonvalue also provides the marshal/unmarshal interface of the response to serialize / deserialize JSON strings. Let's get the response from the front Result. AnArray[0]. The function of somestring is illustrated by an example. The code including complete error checking is as follows:

    // Deserialization
    j, err := jsonvalue.Unmarshal(plainText)
    if err != nil {
        return err
    }

    // Determine whether the interface returns an error
    if e, _ := jsonvalue.Get("Response", "Error"); e != nil {
        return fmt.Errorf("Got error from server: %v", e)
    }

    // Get the string we want
    str, err := j.GetString("Response", "Result", "AnArray", 0, "SomeString")
    if err != nil {
        return err
    }
    fmt.Printf("SomeString = '%s'\n", str)
    return nil

It's over. Isn't it simple? In j.GetString(...) In, the function completes the following functions:

1. It is allowed to pass in indefinite parameters and parse them down in sequence
2. When parsing to a layer, if the current parameter type is string, it will automatically judge whether the current level is Json object. If not, it will return error
3. When parsing a layer, if the current parameter type is an integer number, it will automatically judge whether the current level is Json array. If not, it will return error
4. When fetching from array, if the given array index exceeds the length of array, error will be returned
5. When fetching from object, if the specified key does not exist, error is returned
6. Finally, if the specified key value pair is obtained, it will judge whether the type is Json string. If yes, it will return string value, otherwise it will return error

In other words, in the previous problem, a long series of checks are automatically solved for you in this function.
In addition to the string type, jsonvalue also supports GetBool, GetNull, GetInt, GetUint, GetInt64, GetArray, GetObject and other types. As long as you think of the Json type, it is available.

JSON editing

In most cases, we need to edit a JSON object. Use J: = jsonvalue NewObject(). SetXXX () can be used later At() series of functions to set child members. Like the GetXxx series of functions mentioned above, jsonvalue also supports one-stop complex structure generation. Let's explain one by one:

Set child members of JSON object

For example, set a child member of string type under j: someString = 'Hello, world!'

j.SetString("Hello, world!").At("someString")   // Indicates "set the string type value 'Hello, world!' in the 'someString' key"

Similarly, we can set other types:

j.SetBool(true).At("someBool") // "someBool": true
j.SetArray().At("anArray")     // "anArray": []
j.SetInt(12345).At("anInt")    // "anInt": 12345

Set child members of JSON array

Adding child members to JSON arrays is also a necessary feature. Similarly, we first create an array: A: = jsonvalue NewArray(). The basic operations on arrays are as follows:

// Add an element at the beginning of the array
a.AppendString("Hello, world!").InTheBegging()

// Adds an element to the end of the array
a.AppendInt(5678).InTheEnd()

// Inserts an element before the specified position in the array
a.InsertFloat32(3.14159).Before(1)

// Inserts an element after the specified position in the array
a.InsertNull().After(2)

Quickly edit JSON deeper content

For editing scenarios, jsonvalue also provides the function of quickly creating hierarchies. For example, JSON mentioned earlier:

{
    "Response": {
        "Result": {
            "AnArray": [
                {
                    "SomeString": "Hello, world!"
                }
            ]
        }
    }
}

Using jsonvalue requires only two lines to generate a jsonvalue type object (* jsonvalue.V):

j := jsonvalue.NewObject()
j.SetString("Hello, world!").At("Response", "Result", "AnArray", 0, "SomeString")

In the At() function, jsonvalue will recursively check the JSON value of the current level, and automatically create the corresponding JSON value if necessary according to the requirements of the parameters. The details are as follows:

1. It is allowed to pass in indefinite parameters and parse them down in sequence
2. When parsing to a layer, if the parameter type of the next layer is string, it will automatically judge whether the current level is Json object. If not, it will return error
3. When parsing a layer, if the parameter type of the next layer is an integer number, it will automatically judge whether the current level is Json array. If not, it will return error
4. When resolving to a certain layer, if there are no subsequent parameters, then this is the final goal. The child member is created according to the child member type specified by SetXxxx above

As for the above example, the whole operation logic is as follows:

1. The SetString() function indicates that a child member of type string is to be set
2. The At() function indicates the start of addressing in the JSON object.
3. For the "Response" parameter, first check that this is not the last parameter, then first judge whether the current j is an object object, and if not, return error
4. If the "Response" object exists, take it out; If it does not exist, it is created, and then Response is called internally recursively SetString("Hello, world!"). At("Result", "AnArray", 0, "SomeString")
5. The same applies to "Result"
6. After getting the object of the "Result" layer, check the next parameter and find that it is an integer. Then the function judges that the target "AnArray" of the next layer is expected to be an array. Then, get the target first in the function. If it does not exist, create an array; If it exists, error will be returned if the target is not an array

7. After obtaining "AnArray", the current parameter is an integer. The logic here is complex:

   1. If this parameter is equal to - 1, it means that an element is added at the end of the current array
   2. If the value of this parameter is equal to the length of the current array, it also means that an element is added at the end of the current array
   3. If the value of this parameter is greater than or equal to zero and less than the length of the current array, it means that the specified position of the current array is replaced with a new specified element
8. The last parameter "SomeString" is a string type, which means that AnArray[0] should be an object. Then create a JSON object at AnArray[0] and set {"SomeString":"Hello, world!"}

In fact, you can see that the above process is not very intuitive for the target array type. Therefore, for the level where the target JSON is an array, the Append and Insert functions mentioned above also do not support quantitative parameters. For example, if we need the response mentioned above Result. If a true is added at the end of anarray array, it can be called as follows:

j.AppendBool(true).InTheEnd("Response", "Result", "AnArray")

serialize

Put a jsonvalue The way of V serialization is also very simple: B, _:= j. Marshal () can generate binary strings of type [] byte. As long as jsonvalue is used normally, there will be no error, so you can directly use B: = J. mustmarshal()

For the case where the serialization result of string type needs to be obtained directly, use s: = j.mustmarshalstring(), because bytes. Is used internally Buffer direct output can reduce the additional time-consuming caused by string(b) conversion.

jsonvalue performance test

I compared the three modes of jsonvalue, predefined struct and map[string]interface {}. I simply mixed and nested the types of integer, floating point, string, array and object set. The test results are as follows:

Unmarshal operation comparison

Marshal l operation comparison

It can be seen that the efficiency of jsonvalue in deserialization is slightly higher than that of struct and map; In terms of serialization, struct and jsonvalue are far behind the map scheme, and jsonvalue takes about 1 / 3 more time than struct. On the whole, jsonvalue takes about 5.5% more time to deserialize and serialize than struct. After all, jsonvalue deals with Json with uncertain format. In fact, this achievement is fairly good.

The test command described above is go test - bench =- Run = none - benchmem - benchtime = 10s, CPU is the 10th generation i5 2GHz.
Readers can see my benchmark file.

Other advanced parameters of Jsonvalue

In addition to the above basic operations, jsonvalue also supports some functions that cannot be realized by the map scheme during serialization. The author will record these contents in another article later. Readers can also refer to jsonvalue godoc , as detailed in the document.

This article adopts Knowledge sharing Attribution - non commercial use - sharing in the same way 4.0 international license agreement License.

Original author: amc , welcome to reprint, but please indicate the source.

Original title: still processing JSON with map[string]interface {}? Tell you a more efficient method - jsonvalue

Release date: August 10, 2020

Original published on Cloud + community Is also my blog