Official tutorial: Go fuzzing fuzzy test

preface

Go 1.18 introduces fuzzy testing in the go tool chain, which can help us find bugs in the go code or inputs that may cause program crash. The official go team has also released an introductory tutorial on fuzzing on the official website to help you get started quickly.


On the basis of translation, I have optimized the expression of the official Go course for the readers.

Note: fuzzing fuzzy testing is complementary to Go's existing unit testing and performance testing frameworks, and is not a substitute relationship.

Tutorial content

This tutorial introduces you to the basics of Go fuzzing. Fuzzing can construct random data to find bugs in the code or inputs that may cause the program to crash. Vulnerabilities that can be found through fuzzing include SQL injection, buffer overflow, denial of service attack and XSS (cross site scripting) attack.

In this tutorial, you will write a fuzzy test program for a function, then run the go command to find the problems in the code, and finally fix the problems through debugging.

For the technical terms involved in this article, please refer to Go Fuzzing glossary.

Next, it will be introduced according to the following chapters:

  1. Create a directory for your code
  2. Implement a function
  3. Add unit test
  4. Add blur test
  5. Fix 2 bug s
  6. summary

preparation

  • Install Go 1.18 Beta 1 or later. Installation instructions can be referred to The following introduction.
  • There is a code editing tool. Any text editor will do.
  • There is a command line terminal. Go can run on any command line terminal on Linux or Mac, or on PowerShell or cmd of Windows.
  • There is an environment that supports fuzzing. At present, Go fuzzing only supports AMD64 and ARM64 architectures.

Install and use the beta version

This tutorial requires the generic functionality of Go 1.18 Beta 1 or later. Use the following steps to install the beta version

  1. Install the beta version using the following command

    $ go install golang.org/dl/go1.18beta1@latest
  2. Run the following command to download the update

    $ go1.18beta1 download

    Note: if go1.18beta1 prompts command not found on MAC or Linux, you need to set the profile environment variable file corresponding to bash or zsh. Bash is set at ~/ bash_ In the profile file, the contents are:

    export GOROOT=/usr/local/opt/go/libexec
    export GOPATH=$HOME/go
    export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

    The values of GOROOT and GOPATH can be viewed through the go env command. After setting, execute source ~/ bash_ Profile makes the settings take effect. After go1.18beta1 is executed, no error will be reported.

  3. Use the go command of the beta version instead of the go command of the release version

    You can use the go1.18beta1 command directly or give go1.18beta1 a simple alias

    • Directly use the go1.18beta1 command

      $ go1.18beta1 version
    • Alias the go1.18beta1 command

      $ alias go=go1.18beta1
      $ go version

    The following tutorials assume that you have set the alias go for the go1.18beta1 command.

Create a directory for your code

First, create a directory to store the code you write.

  1. Open a command line terminal and switch to your home directory

    • Execute the following commands on Linux or Mac (you can enter the home directory by executing cd on Linux or MAC)

      cd
    • Execute the following command on Windows

      C:\> cd %HOMEPATH%
  2. At the command line terminal, create a directory named fuzz y and enter it

    $ mkdir fuzz
    $ cd fuzz
  3. Create a go module

    Run the go mod init command to set the module path for your project

    $ go mod init example/fuzz

    Note: for production code, you can specify the module path according to the actual situation of the project. If you want to know more, you can refer to Go Module dependency management.

Next, we use map to write some simple code to do string inversion, and then use fuzzing to do fuzzy testing.

Implement a function

In this chapter, you need to implement a function to reverse the string.

Writing code

  1. Open your text editor and create a main Go source file.
  2. In main Write the following code in go:

    // maing.go
    package main
    
    import "fmt"
    
    func Reverse(s string) string {
        b := []byte(s)
        for i, j := 0, len(b)-1; i < len(b)/2; i, j = i+1, j-1 {
            b[i], b[j] = b[j], b[i]
        }
        return string(b)
    }
    
    func main() {
        input := "The quick brown fox jumped over the lazy dog"
        rev := Reverse(input)
        doubleRev := Reverse(rev)
        fmt.Printf("original: %q\n", input)
        fmt.Printf("reversed: %q\n", rev)
        fmt.Printf("reversed again: %q\n", doubleRev)
    }

Run code

In main Execute the command go run in the directory where go is located To run the code, the results are as follows:

$ go run .
original: "The quick brown fox jumped over the lazy dog"
reversed: "god yzal eht revo depmuj xof nworb kciuq ehT"
reversed again: "The quick brown fox jumped over the lazy dog"

Add unit test

In this chapter, you will write unit test code for the Reverse function.

Writing unit tests

  1. Create a file reverse in the fuzz y directory_ test. go.
  2. In reverse_ test. Write the following code in go:

    package main
    
    import (
        "testing"
    )
    
    func TestReverse(t *testing.T) {
        testcases := []struct {
            in, want string
        }{
            {"Hello, world", "dlrow ,olleH"},
            {" ", " "},
            {"!12345", "54321!"},
        }
        for _, tc := range testcases {
            rev := Reverse(tc.in)
            if rev != tc.want {
                    t.Errorf("Reverse: %q, want %q", rev, tc.want)
            }
        }
    }

Run unit tests

Use the go test command to run unit tests

$ go test
PASS
ok      example/fuzz  0.013s

Next, we add fuzzy test code to the Reverse function.

Add blur test

Unit tests have limitations. Each test input must be specified by the developer and added to the test cases of unit tests.

One of the advantages of fuzzing is that it can automatically generate new random test data based on the test input specified in the developer's code as the basic data to find the boundary conditions that are not covered by the specified test input.

In this chapter, we will transform unit tests into fuzzy tests, which can easily generate more test inputs.

Note: you can put unit test, performance test and fuzzy test in the same*_ test.go file.

Writing fuzzy tests

Reverse in the text editor_ test. The unit test code TestReverse in go is replaced by the following fuzzy test code FuzzReverse.

func FuzzReverse(f *testing.F) {
    testcases := []string{"Hello, world", " ", "!12345"}
    for _, tc := range testcases {
        f.Add(tc)  // Use f.Add to provide a seed corpus
    }
    f.Fuzz(func(t *testing.T, orig string) {
        rev := Reverse(orig)
        doubleRev := Reverse(rev)
        if orig != doubleRev {
            t.Errorf("Before: %q, after: %q", orig, doubleRev)
        }
        if utf8.ValidString(orig) && !utf8.ValidString(rev) {
            t.Errorf("Reverse produced invalid UTF-8 string %q", rev)
        }
    })
}

Fuzzing also has some limitations.

In the unit test, because the test input is fixed, you can know what the Reverse string obtained from each input string after calling the Reverse function should be, and then judge whether the Reverse execution result is consistent with the expectation in the unit test code. For example, for the test case Reverse("Hello, world"), the expected result of the unit test is "dlrow, olleh".

However, when using fuzzing, we can't predict the output results, because the test inputs are randomly generated by fuzzing in addition to the use cases specified in our code. For randomly generated test inputs, we certainly can't know what the output results are in advance.

Nevertheless, the Reverse function in this article has several features that we can verify in fuzzy testing.

  1. Reverse a string twice, and the result is the same as the source string
  2. The inverted string is still a valid UTF-8 encoded string

Note: fuzzing fuzzy testing is complementary to Go's existing unit testing and performance testing frameworks, and is not a substitute relationship.

For example, if the Reverse function we implemented is a wrong version and directly return s the input string, it can completely pass the above fuzzy test, but it cannot pass the unit test we wrote earlier. Therefore, unit testing and fuzzy testing are complementary to each other, not substitutes.

Go fuzzy test and unit test have the following differences in Syntax:

  • Go fuzzy test function starts with FuzzXxx, and unit test function starts with TestXxx
  • Go fuzzy test function with *testing F as the input parameter, the unit test function uses *testing T as input parameter
  • The Go fuzzy test calls the f.Add function and the f.fuzzy function.

    • f. The add function takes the specified input as the seed corpus of the fuzzy test, and fuzzing generates random input based on the seed corpus.
    • f. The fuzzy function receives a fuzzy target function as an input parameter. The fuzzy target function has multiple parameters. The first parameter is *testing T. Other parameters are fuzzy types (Note: fuzzy types currently only support some built-in types, which are listed in the Go Fuzzing docs , more built-in types will be supported in the future).

The package utf8 is used in the FuzzReverse function above, so the reverse_ test. import the package starting with go. Refer to the following code:

package main

import (
    "testing"
    "unicode/utf8"
)

Run fuzzy test

  1. Execute the following command to run the fuzzy test.

    This method only uses the seed corpus, and does not generate random test data. This method can be used to verify whether the test data of the seed corpus can pass the test. (fuzz test without fuzzing)

    $ go test
    PASS
    ok      example/fuzz  0.013s

    If reverse_ test. There are other unit test functions or fuzzy test functions in the go file, but we only want to run the FuzzReverse fuzzy test function. We can execute the go test -run=FuzzReverse command.

    Note: by default, go test will execute all unit test functions beginning with TestXxx and fuzzy test functions beginning with FuzzXxx. By default, it does not run performance test functions beginning with BenchmarkXxx. If we want to run benchmark cases, we need to add the -benchmark parameter.

  2. If you want to generate random test data based on the seed corpus for fuzzy testing, you need to add the -fuzzy parameter to the go test command. (fuzz test with fuzzing)

    $ go test -fuzz=Fuzz
    fuzz: elapsed: 0s, gathering baseline coverage: 0/3 completed
    fuzz: elapsed: 0s, gathering baseline coverage: 3/3 completed, now fuzzing with 8 workers
    fuzz: minimizing 38-byte failing input file...
    --- FAIL: FuzzReverse (0.01s)
        --- FAIL: FuzzReverse (0.00s)
            reverse_test.go:20: Reverse produced invalid UTF-8 string "\x9c\xdd"
    
        Failing input written to testdata/fuzz/FuzzReverse/af69258a12129d6cbba438df5d5f25ba0ec050461c116f777e77ea7c9a0d217a
        To re-run:
        go test -run=FuzzReverse/af69258a12129d6cbba438df5d5f25ba0ec050461c116f777e77ea7c9a0d217a
    FAIL
    exit status 1
    FAIL    example/fuzz  0.030s

    The above fuzzing test result is FAIL, and the input data causing FAIL is written into a corpus file. The next time you run the go test command, even if there is no -fuzzy parameter, the test data in this corpus file will be used.

    You can use a text editor to open the file in the testdata/fuzzy/fuzzreverse directory to see what the test data that causes the Fuzzing test to fail looks like. The following is a sample file. The test data you get after running there may be different from this, but the content format in the file will be the same.

    go test fuzz v1
    string("Diamond")

    The first line in the corpus file identifies the encoding Version (that is, the version of the content format in the sub corpus file). Although there is only v1 at present, the Fuzzing designer adds the concept of encoding version in consideration of the possible introduction of new encoding versions in the future.

    Starting from line 2, each line of data corresponds to one of the parameters of each corpus entry, which is arranged in order of parameters.

    f.Fuzz(func(t *testing.T, orig string) {
            rev := Reverse(orig)
            doubleRev := Reverse(rev)
            if orig != doubleRev {
                t.Errorf("Before: %q, after: %q", orig, doubleRev)
            }
            if utf8.ValidString(orig) && !utf8.ValidString(rev) {
                t.Errorf("Reverse produced invalid UTF-8 string %q %q", orig, rev)
            }
    })

    The fuzzy target function func (t *test.T, Orig string) in the FuzzReverse in this article only has Orig as the real test input, that is, each test data actually has one input. Theme, in the testdata/fuzzy/fuzzreverse directory of the above example, there is only a line of string ("di")

    If each test data has N parameters, each test data found by fuzzing that leads to the failure of fuzzy test will have N lines in the file under the testdata directory, and the i line corresponds to the i parameter.

  3. Run the go test command again, this time without the -fuzz y parameter.

    We will find that although there is no -fuzzy parameter, the fuzzy test still uses the test data found in step 2 above.

    $ go test
    --- FAIL: FuzzReverse (0.00s)
        --- FAIL: FuzzReverse/af69258a12129d6cbba438df5d5f25ba0ec050461c116f777e77ea7c9a0d217a (0.00s)
            reverse_test.go:20: Reverse produced invalid string
    FAIL
    exit status 1
    FAIL    example/fuzz  0.016s

    Since the Go fuzzing test failed, we need to debug the code to find out the problem.

Fix 2 bug s

In this chapter, we will debug the program and fix the bugs detected by Go fuzzing.

You can take some time to think for yourself and try to solve the problem yourself.

Positioning problem

You can use different methods to debug the bugs found above.

If you use VS Code, you can set your Debug debugger To add breakpoints for debugging.

In this article, we will use the method of printing logs for debugging.

The error message when running the fuzzy test is: reverse_test.go:20: Reverse produced invalid UTF-8 string "\x9c\xdd"

Based on this error, let's take a look at the utf8.ValidString Description of.

ValidString reports whether s consists entirely of valid UTF-8-encoded runes.

The Reverse function we implemented is to Reverse the string according to the dimension of bytes, which is the problem.

For example, the Chinese character is actually composed of three bytes if you reverse the bytes, you will get an invalid string

Therefore, to ensure that a valid UTF-8 encoded string is still obtained after string inversion, we should perform string inversion according to run.

In order to better understand how many rune s there are in the Chinese character "run" dimension and what the result looks like after byte inversion, we have made some modifications to the code.

Writing code

Modify the code in FuzzReverse as follows.

f.Fuzz(func(t *testing.T, orig string) {
    rev := Reverse(orig)
    doubleRev := Reverse(rev)
    t.Logf("Number of runes: orig=%d, rev=%d, doubleRev=%d", utf8.RuneCountInString(orig), utf8.RuneCountInString(rev), utf8.RuneCountInString(doubleRev))
    if orig != doubleRev {
        t.Errorf("Before: %q, after: %q", orig, doubleRev)
    }
    if utf8.ValidString(orig) && !utf8.ValidString(rev) {
        t.Errorf("Reverse produced invalid UTF-8 string %q", rev)
    }
})

Run code

$ go test
--- FAIL: FuzzReverse (0.00s)
    --- FAIL: FuzzReverse/28f36ef487f23e6c7a81ebdaa9feffe2f2b02b4cddaa6252e87f69863046a5e0 (0.00s)
        reverse_test.go:16: Number of runes: orig=1, rev=3, doubleRev=1
        reverse_test.go:21: Reverse produced invalid UTF-8 string "\x83\xb3\xe6"
FAIL
exit status 1
FAIL    example/fuzz    0.598s

Each symbol in our seed corpus is a single byte. However, Chinese symbols such as "are composed of multiple bytes If you reverse them with bytes as the dimension, you will get invalid results.

Note: if you are interested in how Go handles strings, you can read this article in the official blog Strings, bytes, runes and characters in Go To deepen understanding.

Now that we have identified the problem, we can fix the bug.

Fix problem

String inversion with rune as the dimension.

Writing code

Modify the implementation of the Reverse function as follows:

func Reverse(s string) string {
    r := []rune(s)
    for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
        r[i], r[j] = r[j], r[i]
    }
    return string(r)
}

Run code

  1. Run command: go test

    $ go test
    PASS
    ok      example/fuzz  0.016s

    Test passed! (don't be happy too early, this just passed the seed corpus and before)

  2. Run go test -fuzz y again to see if we find any new bug s

    $ go test -fuzz=Fuzz
    fuzz: elapsed: 0s, gathering baseline coverage: 0/37 completed
    fuzz: minimizing 506-byte failing input file...
    fuzz: elapsed: 0s, gathering baseline coverage: 5/37 completed
    --- FAIL: FuzzReverse (0.02s)
        --- FAIL: FuzzReverse (0.00s)
            reverse_test.go:33: Before: "\x91", after: "�"
    
        Failing input written to testdata/fuzz/FuzzReverse/1ffc28f7538e29d79fce69fef20ce5ea72648529a9ca10bea392bcff28cd015c
        To re-run:
        go test -run=FuzzReverse/1ffc28f7538e29d79fce69fef20ce5ea72648529a9ca10bea392bcff28cd015c
    FAIL
    exit status 1
    FAIL    example/fuzz  0.032s

    Through the above error reports, we find that the result of reversing a string twice is different from the original string.

    The test input itself is illegal unicode, but why is the string obtained after two inversions still different?

    Let's continue debugging.

Fix the bug of twice string inversion

Positioning problem

For this problem, adding breakpoint debugging will be a good location. For the convenience of explanation, this article uses the method of adding logs for debugging.

We can carefully observe the results of the first inversion of the original string to locate the problem.

Writing code

  1. Modify the Reverse function.

    func Reverse(s string) string {
        fmt.Printf("input: %q\n", s)
        r := []rune(s)
        fmt.Printf("runes: %q\n", r)
        for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
            r[i], r[j] = r[j], r[i]
        }
        return string(r)
    }

    This helps us understand what happens when we turn the original string into a rune slice.

Run code

This time, we only run the test data that makes fuzzy test fail, and use the go test -run command.

Run the corpus test data specified in the FuzzXxx/testdata directory. You can specify the value {FuzzTestName}/{filename} for the -run parameter. This allows us to focus on the test data that makes fuzzy test fail.

$ go test -run=FuzzReverse/28f36ef487f23e6c7a81ebdaa9feffe2f2b02b4cddaa6252e87f69863046a5e0
input: "\x91"
runes: ['�']
input: "�"
runes: ['�']
--- FAIL: FuzzReverse (0.00s)
    --- FAIL: FuzzReverse/28f36ef487f23e6c7a81ebdaa9feffe2f2b02b4cddaa6252e87f69863046a5e0 (0.00s)
        reverse_test.go:16: Number of runes: orig=1, rev=1, doubleRev=1
        reverse_test.go:18: Before: "\x91", after: "�"
FAIL
exit status 1
FAIL    example/fuzz    0.145s

First of all, we need to understand that in Go, a string is a read only slice of bytes. Each byte in the byte slice may not be a valid UTF-8 encoded byte. For details, please refer to a string is a read only slice of bytes.

In the above example, the input string is a byte slice with only one byte, which is \x91.

When we convert the input string to []run, Go will encode the byte slice as UTF-8, so it will replace \x91 with ',', or 'after the restaurant', which will cause the original string \x91 to be 'after being reversed once.

Now the problem is clear, because the input data is illegal unicode. Then we can modify the implementation of the Reverse function.

Fix problem

The repair method is: check whether the input is a legal UTF-8 encoded string in Reverse. If it is illegal, return eror.

Writing code

  1. Modify the Reverse implementation as follows:

    func Reverse(s string) (string, error) {
        if !utf8.ValidString(s) {
            return s, errors.New("input is not valid UTF-8")
        }
        r := []rune(s)
        for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
            r[i], r[j] = r[j], r[i]
        }
        return string(r), nil
    }
  2. Since the Reverse function will now return error, modify main The corresponding code in go is modified as follows:

    func main() {
        input := "The quick brown fox jumped over the lazy dog"
        rev, revErr := Reverse(input)
        doubleRev, doubleRevErr := Reverse(rev)
        fmt.Printf("original: %q\n", input)
        fmt.Printf("reversed: %q, err: %v\n", rev, revErr)
        fmt.Printf("reversed again: %q, err: %v\n", doubleRev, doubleRevErr)
    }

    Since the main function contains valid UTF-8 encoded strings, the call to Reverse will return an error with a value of nil.

  3. Since the Reverse function uses two packages, errors and utf8, the main The two packages should be import ed at the beginning of go.

    import (
        "errors"
        "fmt"
        "unicode/utf8"
    )
  4. Again, we need to modify reverse_test.go file. For illegal string input, you can directly skip the test.

    func FuzzReverse(f *testing.F) {
        testcases := []string {"Hello, world", " ", "!12345"}
        for _, tc := range testcases {
            f.Add(tc)  // Use f.Add to provide a seed corpus
        }
        f.Fuzz(func(t *testing.T, orig string) {
            rev, err1 := Reverse(orig)
            if err1 != nil {
                return
            }
            doubleRev, err2 := Reverse(rev)
            if err2 != nil {
                 return
            }
            if orig != doubleRev {
                t.Errorf("Before: %q, after: %q", orig, doubleRev)
            }
            if utf8.ValidString(orig) && !utf8.ValidString(rev) {
                t.Errorf("Reverse produced invalid UTF-8 string %q", rev)
            }
        })
    }

    In addition to using return, you can also call t.Skip() to skip the current test input and continue the next round of test input.

Run code

  1. Run test code

    $ go test
    PASS
    ok      example/fuzz  0.019s
  2. Run the fuzzy testgo test -fuzzy=fuzzy. After a few seconds of execution, use ctrl-C to end the test.

    $ go test -fuzz=Fuzz
    fuzz: elapsed: 0s, gathering baseline coverage: 0/38 completed
    fuzz: elapsed: 0s, gathering baseline coverage: 38/38 completed, now fuzzing with 4 workers
    fuzz: elapsed: 3s, execs: 86342 (28778/sec), new interesting: 2 (total: 35)
    fuzz: elapsed: 6s, execs: 193490 (35714/sec), new interesting: 4 (total: 37)
    fuzz: elapsed: 9s, execs: 304390 (36961/sec), new interesting: 4 (total: 37)
    ...
    fuzz: elapsed: 3m45s, execs: 7246222 (32357/sec), new interesting: 8 (total: 41)
    ^Cfuzz: elapsed: 3m48s, execs: 7335316 (31648/sec), new interesting: 8 (total: 41)
    PASS
    ok      example/fuzz  228.000s

    If fuzzy test does not encounter errors, it will run continuously by default. You need to use ctrl-C to end the test.

    You can also pass the -fuzztime parameter to specify the test time, so you don't have to use ctrl-C.

  3. Specify the test time. Go test -fuzzy=fuzzy -fuzztime 30s if no error is encountered, it will automatically end after 30s.

    $ go test -fuzz=Fuzz -fuzztime 30s
    fuzz: elapsed: 0s, gathering baseline coverage: 0/5 completed
    fuzz: elapsed: 0s, gathering baseline coverage: 5/5 completed, now fuzzing with 4 workers
    fuzz: elapsed: 3s, execs: 80290 (26763/sec), new interesting: 12 (total: 12)
    fuzz: elapsed: 6s, execs: 210803 (43501/sec), new interesting: 14 (total: 14)
    fuzz: elapsed: 9s, execs: 292882 (27360/sec), new interesting: 14 (total: 14)
    fuzz: elapsed: 12s, execs: 371872 (26329/sec), new interesting: 14 (total: 14)
    fuzz: elapsed: 15s, execs: 517169 (48433/sec), new interesting: 15 (total: 15)
    fuzz: elapsed: 18s, execs: 663276 (48699/sec), new interesting: 15 (total: 15)
    fuzz: elapsed: 21s, execs: 771698 (36143/sec), new interesting: 15 (total: 15)
    fuzz: elapsed: 24s, execs: 924768 (50990/sec), new interesting: 16 (total: 16)
    fuzz: elapsed: 27s, execs: 1082025 (52427/sec), new interesting: 17 (total: 17)
    fuzz: elapsed: 30s, execs: 1172817 (30281/sec), new interesting: 17 (total: 17)
    fuzz: elapsed: 31s, execs: 1172817 (0/sec), new interesting: 17 (total: 17)
    PASS
    ok      example/fuzz  31.025s

    Fuzzing test passed!

    In addition to the -fuzz y parameter, several new parameters have also been introduced into the go test command. For details, see documentation.

summary

Now you have learned how to use Go fuzzing.

Next, you can try to use fuzzing to find bug s in your code.

If you do find a bug, please consider submitting the case to trophy case.

If you find any problems with Go fuzzing or want to mention feature s, you can give feedback here file an issue.

View document go.dev/doc/fuzz Learn more about Go Fuzzing.

Full code reference for this article Go Fuzzing sample code.

Open source address

Articles and sample code are open source in GitHub: Go language elementary, intermediate and advanced tutorials.

Official account: coding advanced. Follow the official account to get the latest Go interview questions and technology stack.

Personal website: Jincheng's Blog.

Know: Wuji.

References

Tags: Go Back-end Testing

Posted by spighita on Sat, 06 Aug 2022 21:24:47 +0300