unit testing framework

https://en.wikipedia.org/wiki/Unit_testing

https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

https://docs.python-guide.org/writing/tests/

https://docs.python.org/zh-cn/3/library/unittest.html#module-unittest

https://docs.python.org/zh-cn/3/using/cmdline.html#cmdoption-m

https://docs.python.org/3/library/main.html?highlight=main#module-main

https://docs.pytest.org/en/latest/

https://github.com/scrapy/scrapy/tree/master/tests

unit test definition

  • Comprehensive applications can be built for complex applications by writing tests for the smallest testable unit and then for the composite behavior between them without running the final application 's test.
  • The smallest testable unit can be a function, a class, an interface, a module, etc.

unit test meaning

  • Writing unit tests requires a certain 'cost', because the smallest unit with low quality cannot write good unit tests.
  • At the same time as paying the 'price', it brings advantages (mutual influence includes).
    • Improve quality, forcing to look like the master
      • Force compliance with the single responsibility principle to ensure a single function
      • Force to abide by the principle of low coupling, after all, the responsibility is single
      • Force to comply with stateless input and output, that is, the concept of functional programming, do not abuse global variables and other stunts
      • Force compliance with coding standards and framework standards (if you don’t want to build your own wheels, you must use an open source testing framework, and you must organize files according to routines and name them)
    • Improve maintainability and ensure refactoring quality
      • Handover of old and new, unit tests are living documents that help newcomers to familiarize themselves with functions and business
      • Refactoring process to ensure that previous test cases can pass
    • Increase automation
      • Unit testing focuses on smaller granularity and can be combined with interface testing, UI testing, and integration testing
      • Can penetrate into automation, continuous integration, continuous build, continuous delivery, static analysis (such as Jenkins code test coverage, pass rate)
    • Inheriting open source culture (sharing, mutual assistance, teamwork)
      • Unit testing and code review have become standardized processes in software engineering
      • Learn from each other's strengths in discussions and exchanges
      • Big Brother appeals, companies without code review and unit testing culture, please leave

unit testing framework

  • Almost all programming languages ​​have unit testing frameworks

  • Let's feel it first, Cmocka, a unit testing framework in c language

    /**
     * https://github.com/clibs/cmocka/blob/master/example/assert_module.h
     * head File
     */
    //Increments the value pointed to by value by one
    extern void increment_value(int * const value);
    //Decrements the value pointed to by value by one
    extern void decrement_value(int * const value);
    
    
    /**
     * https://github.com/clibs/cmocka/blob/master/example/assert_module.c
     * Source File
     */
    #include <assert.h>
    #include "assert_module.h"
    
    void increment_value(int * const value) {
        assert(value);
        (*value) ++;
    }
    
    void decrement_value(int * const value) {
        if (value) {
          (*value) --;
        }
    }
    
    
    /**
     * https://github.com/clibs/cmocka/blob/master/example/assert_module_test.c
     * test entry file
     */
    #include <stdarg.h>
    #include <stddef.h>
    #include <setjmp.h>
    #include <cmocka.h>
    #include "assert_module.h"
    
    /**
     *increment_value Execution process, assert(value) is false and an exception is thrown
     */
    static void increment_value_fail(void **state) {
        (void) state;
    
        increment_value(NULL);
    }
    
    /**
     *increment_value Execution process, assert(value) is false and an exception is thrown
     *expect_assert_failure Expect the exception to appear and the use case to pass
     */
    /* This test case succeeds since increment_value() asserts on the NULL
     * pointer. */
    static void increment_value_assert(void **state) {
        (void) state;
    
        expect_assert_failure(increment_value(NULL));
    }
    
    /**
     *decrement_value Execution process, if (value) is false, no running (*value) -- and no exception
     *expect_assert_failure The expected exception did not appear and the use case failed
     */
    static void decrement_value_fail(void **state) {
        (void) state;
    
        expect_assert_failure(decrement_value(NULL));
    }
    
    int main(void) {
        const struct CMUnitTest tests[] = {
            cmocka_unit_test(increment_value_fail),
            cmocka_unit_test(increment_value_assert),
            cmocka_unit_test(decrement_value_fail),
        };
        return cmocka_run_group_tests(tests, NULL, NULL);
    }
    
    /**
     * main.c
     * Application entry file
     */
    #include <stdio.h>
    #include "assert_module.h"
    
    int main(int argc, char **argv) {
        int x = 5;
        increment_value(&x);
        printf("5 After adding 1:%d\n", x);
        
        int y = 9;
        decrement_value(&y);
        printf("9 After subtracting 1:%d\n", y);
        
        return 0;
    }
    
    /*
     * Header file tool.h
     */
    #ifndef __TOOL_H__
    #define __TOOL_H__
    
    #include <stdbool.h>
    
    extern void readline(char *value);
    extern bool isValidNum(char *value);
    
    #endif
    
    /*
     * Source file tool.c
     */
    #include <stdio.h>
    #include <string.h>
    #include "tool.h"
    
    void readline(char *value) {
      int position = 0;
      char c;
      while ((c = getchar()) != '\n') {
        value[position++] = c;
      }
      value[position] = '\0';
    }
    
    bool isValidNum(char *value) {
      if (!value) {
        return false;
      }
    
      //Positive and negative signs are skipped in the first place
      if (*value == '-' || *value == '+') {
        value += 1;
      }
    
      int position = 0;
      while (*value) {
        char c = *value;
        //number
        if (c < '0' || c > '9') {
          return false;
        }
        position++;
        value += 1;
      }
    
      return position > 0;
    }
    
    /*
     * Unit test entry test.c
     */
    #include <setjmp.h>
    #include <stdarg.h>
    #include <stddef.h>
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <cmocka.h>
    #include "tool.h"
    
    static void test_isValidNum_1(void **state) {
      (void)state; /* unused */
      char value[] = "tst";
      bool want = false;
      bool target = isValidNum(value);
      assert_int_equal(want, target);
    }
    
    static void test_isValidNum_2(void **state) {
      (void)state; /* unused */
      char value[] = "123";
      bool want = true;
      bool target = isValidNum(value);
      assert_int_equal(want, target);
    }
    
    static void test_isValidNum_3(void **state) {
      (void)state; /* unused */
      char value[] = "+123";
      bool want = true;
      bool target = isValidNum(value);
      assert_int_equal(want, target);
    }
    
    static void isValidNum_4(void **state) {
      (void)state; /* unused */
      char value[] = "1-23";
      bool want = true;
      bool target = isValidNum(value);
      assert_int_equal(want, target);
    }
    
    int main(int argc, char **argv) {
      const struct CMUnitTest tests[] = {
          cmocka_unit_test(test_isValidNum_1), cmocka_unit_test(test_isValidNum_2),
          cmocka_unit_test(test_isValidNum_3), cmocka_unit_test(isValidNum_4)};
    
      return cmocka_run_group_tests(tests, NULL, NULL);
    }
    
    /*
     * Application entry main.c
     */
    #include <stdio.h>
    #include "tool.h"
    
    #include <stdio.h>
    #include "tool.h"
    
    int main(int argc, char **argv) {
      char tmp[255];
    
      while (true) {
        readline(tmp);
        if (isValidNum(tmp)) {
          printf("%s is number!\n", tmp);
        } else {
          printf("%s is not number!\n", tmp);
        }
      }
    
      return 0;
    }
    
  • Use case passed: expected value == running value; use case failed: expected value != running value.

  • Test assert_module_test.c and assert_module.h assert_module.c are isolated and do not affect each other.

  • Write a main.c file as the formal program entry; assert_module_test.c as the unit test entry.

  • The official compilation specifies main.c, and the unit test specifies assert_module_test.c.

  • Combined with the characteristics of the programming language itself, the test framework can hide the entry file, separate release code and test code through magic such as @Test, file Test_, etc., preconditions, abide by conventions (similar to Servlet specifications, etc.); specify the running entry (such as pytest).

unittest

  • unittest is a python standard module, similar in style to JUnit.

  • demo file running mode

    #launch.json
    {
      "version": "0.2.0",
      "configurations": [{
        "name": "mytest",
        "type": "python",
        "request": "launch",
        "program": "${file}",
        "console": "integratedTerminal",
        "args": ["-v"]
      }]
    }
    
    #widget.py
    class Widget():
    
      def __init__(self, name):
        self.name = name
        self.width = 50
        self.height = 50
    
      def size(self):
        return (self.width, self.height)
    
      def resize(self, width, height):
        self.width = width
        self.height = height
    
    #demo.py
    import unittest
    from widget import Widget
    
    
    # unittest.TestCase provides the basic framework for running test cases
    class Demo(unittest.TestCase):
    
      # Each test case must start with test, and the assert method compares the expected value with the running value
      def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')
    
      def testadd(self):
        self.assertTrue(2 + 3 == 6)
    
      def tstadd(self):
        self.assertTrue(2 + 3 == 5)
    
    
    class Tst(unittest.TestCase):
    
      def testin(self):
        self.assertNotIn('n', ['h', 'e', 'l', 'l', 'o'])
    
    
    class WidgetTest(unittest.TestCase):
    
      #Pre-operation, creating resources, etc.
      def setUp(self):
        self.widget = Widget('The widget')
    
      def test_default_widget_size(self):
        value = self.widget.size()
        self.assertEqual(value, (50, 50), 'incorrect default size')
    
      def test_widget_resize(self):
        self.widget.resize(100, 150)
        value = self.widget.size()
        self.assertEqual(value, (100, 150), 'wrong size after resize')
    
      #Post operations, clean up resources, etc.
      def tearDown(self):
        pass
    
    
    if __name__ == '__main__':
      #launch config -v shows more details
      unittest.main()
        
        
    #operation result
    '''
    test_upper (__main__.Demo) ... ok
    testadd (__main__.Demo) ... FAIL
    testin (__main__.Tst) ... ok
    test_default_widget_size (__main__.WidgetTest) ... ok
    test_widget_resize (__main__.WidgetTest) ... ok
    
    ======================================================================
    FAIL: testadd (__main__.Demo)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "d:\testdemo\demo.py", line 13, in testadd
        self.assertTrue(2 + 3 == 6)
    AssertionError: False is not true
    
    ----------------------------------------------------------------------
    Ran 5 tests in 0.006s
    
    FAILED (failures=1)
    '''
    
  • demo command-line running mode, let the framework find the test unit by itself, or specify the test unit

    python -m unittest -v demo.py            #Specify the demo.py file
    python -m unittest -v                    #Rename demo.py to test*.py
    python -m unittest -v demo.Demo          #Specify test class
    python -m unittest -v demo.Demo.tstadd   #Specify the test function
    
  • python -m parameter: Search sys.path for the module with the specified name and execute its content as the __main__ module, which runs __mian__.py under the module.

    #__main__.py under the unittest module
    from .main import main
    
    main(module=None) #unittest.main()
    
  • unittest.main() or TestProgram() is the entry point for the execution of the unittest framework, and TestLoader (default loader) identifies and loads test cases.

  • It is necessary to separate test code and source code into separate files.

pytest

  • Third-party modules, need to be installed

  • The framework is encapsulated more intelligently, and the use case discovery is simpler (test_ prefix function, Test prefix class), users only need to name according to the convention and import pytest

    #luanch.json
    {
      "version": "0.2.0",
      "configurations": [{
        "name": "mytest",
        "type": "python",
        "request": "launch",
        "program": "${file}",
        "console": "integratedTerminal",
        "module": "pytest",
        "cwd": "${fileDirname}"
      }]
    }
    
    #demo.py
    def inc(x):
      return x + 1
    
    #test_demo.py
    import pytest
    from demo import inc
    
    
    def test_answer():
      assert inc(3) == 5
    
    
    class TestDemo():
    
      def testinc(self):
        assert inc(8) == 9
     
    
    '''
    =========================================================== test session starts ============================================================
    platform win32 -- Python 3.7.5, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
    rootdir: D:\testdemo
    collected 2 items                                                                                                                            
    
    test_demo.py F.                                                                                                                       [100%] 
    
    ================================================================= FAILURES ================================================================= 
    _______________________________________________________________ test_answer ________________________________________________________________ 
    
        def test_answer():
    >     assert inc(3) == 5
    E     assert 4 == 5
    E      +  where 4 = inc(3)
    
    test_demo.py:6: AssertionError
    ======================================================= 1 failed, 1 passed in 0.14s ========================================================
    '''
    

Specify test cases

  • Run tests in a module pytest test_mod.py
  • Run tests in the directory pytest testing/
  • Run tests by node ID pytest test_mod.py::test_func or pytest test_mod.py::TestClass::test_method
  • Run tests from package pytest --pyargs pkg.testing

trace back print

  • –showlocals # show local variables in tracebacks
  • -l # show local variables (shortcut)
  • –tb=auto # (default) 'long' tracebacks for the first and last # entry, but 'short' style for the other entries
  • –tb=long # exhaustive, informative traceback formatting
  • –tb=short # shorter traceback format
  • –tb=line # only one line per failure
  • –tb=native # Python standard library formatting
  • –tb=no # no traceback at all

Detailed summary report -r

Default fE

  • f - fail

  • E - Error

  • s skip

  • x - fail

  • X -XPASS

  • p pass

  • P - Pass output

save results

  • JUnitXML format file pytest --junitxml=path
  • Result log format file pytest --resultlog=path

The code calls pytest

  • pytest.main()

fixtures

  • Initialize test functions that provide a fixed baseline for tests to perform reliably and produce consistent, repeatable results. Initialization can set up a service, state, or other operating environment. In fixture functions, each function's parameter is usually named fixture after test
  • pytest fixtures provide significant improvements over traditional xUnit-style setup/teardown functions
  • conftest.py shared initialization
  • @pytest.fixtures(params=), loopable sequence of params, request.param. No need to change test function code. let's run again
import pytest
from sqlalchemy import create_engine, exc, inspect, text


@pytest.fixture(
    params=[
        # request: (sql_url_id, sql_url_template)
        ('sqlite_memory', 'sqlite:///:memory:'),
        ('sqlite_file', 'sqlite:///{dbfile}'),
        # ('psql', 'postgresql://records:records@localhost/records')
    ],
    ids=lambda r: r[0]
)
def db(request, tmpdir):
    """Instance of `records.Database(dburl)`

    Ensure, it gets closed after being used in a test or fixture.

    Parametrized with (sql_url_id, sql_url_template) tuple.
    If `sql_url_template` contains `{dbfile}` it is replaced with path to a
    temporary file.

    Feel free to parametrize for other databases and experiment with them.
    """
    id, url = request.param
    # replace {dbfile} in url with temporary db file path
    url = url.format(dbfile=str(tmpdir / "db.sqlite"))
    print('request:', id, url)
    print('tmpdir:', tmpdir)
    _engine = create_engine(url)
    yield _engine  # providing fixture value for a test case
    # tear_down
    _engine.dispose()


@pytest.fixture
def foo_table(db):
    """Database with table `foo` created

    tear_down drops the table.

    Typically applied by `@pytest.mark.usefixtures('foo_table')`
    """
    db.connect().execute(text('CREATE TABLE foo (a integer)'))
    yield
    db.connect().execute(text('DROP TABLE foo'))


def test_aaa(db):
    db.connect().execute(text("CREATE table users (id text)"))
    db.connect().execute(text("SELECT * FROM users WHERE id = :user"), user="Te'ArnaLambert")


if __name__ == "__main__":
    pytest.main([__file__])

Summarize

  • Unit testing framework to get started quickly
  • Write more, practice more, read more
  • In asynchronous multi-threaded scenarios, it is difficult to write unit tests

Tags: unit testing

Posted by sirup_segar on Tue, 03 May 2022 16:33:06 +0300