Deep parsing of Python function objects (Part 2): how functions are called at the bottom

Wedge

In the last blog, we talked about the underlying implementation of Python functions, and also demonstrated how to customize a function. Although this doesn't make much sense in work, it can make us deeply understand the behavior of functions. In addition, we also introduced how to get the parameters of the function. This time, let's see how the function is called.

Function call

s = """
def foo():
    a, b = 1, 2
    return a + b

foo()
"""

if __name__ == '__main__':
    import dis
    dis.dis(compile(s, "call_function", "exec"))

Let's take a very simple function as an example to see its bytecode:

  2           0 LOAD_CONST               0 (<code object foo at 0x00000219BA3F1450, file "call_function", line 2>)
              2 LOAD_CONST               1 ('foo')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (foo)

  6           8 LOAD_NAME                0 (foo)
             10 CALL_FUNCTION            0
             12 POP_TOP
             14 LOAD_CONST               2 (None)
             16 RETURN_VALUE

Disassembly of <code object foo at 0x00000219BA3F1450, file "call_function", line 2>:
  3           0 LOAD_CONST               1 ((1, 2))
              2 UNPACK_SEQUENCE          2
              4 STORE_FAST               0 (a)
              6 STORE_FAST               1 (b)

  4           8 LOAD_FAST                0 (a)
             10 LOAD_FAST                1 (b)
             12 BINARY_ADD
             14 RETURN_VALUE

In other words, the module has a PyCodeObject object, and the function also has a PyCodeObject object, but the latter is in the constant pool of the former. Moreover, the dis module automatically helps us separate when displaying the bytecode. We stroke it from top to bottom.

  • 0 LOAD_CONST 0 (< code object...: if you encounter the def keyword, you know this is a function, so you will load its corresponding PyCodeObject object
  • 2 LOAD_CONST 1 ('foo '): load function name
  • 4 MAKE_FUNCTION 0: through make_ The function instruction constructs a function
  • 6 STORE_ Name 0 (foo): bind the symbol "foo" with the function obtained in the previous step and store it in the local space, which is obviously the local space of the module, that is, the global space
  • 8 LOAD_ Name 0 (foo): note that this step occurs when calling. Load the variable foo
  • 10 CALL_FUNCTION 0: through call_ The function instruction calls this function (which we will focus on later). The following 0 indicates the number of parameters
  • 12 POP_TOP: pop the return value of the previous function from the top of the runtime stack
  • 14 LOAD_ Const 2 (None): load return value None
  • 16 RETURN_VALUE: returns the return value

The bytecode corresponding to the module is like the above. Let's take a look at the function. In fact, it is very simple for you now.

  • 0 LOAD_ Const 1 ((1, 2)): load tuples from the constant pool. We say that for the list, the internal elements are loaded one by one, and then through BUILD_LIST builds a list, but for tuples, it can be loaded directly, because the address of the element pointing to the object in the tuple cannot be changed
  • 2 UNPACK_ Sequence 2: unpacking
  • 4 STORE_ Fast 0 (a): assign the first of the two constants obtained by unpacking to a
  • 6 STORE_ Fast 0 (b): assign the second of the two constants obtained by unpacking to B
  • 8 LOAD_ Local variable loading (a): 0
  • 10 LOAD_ Fast 1 (b): load local variable b
  • 12 BINARY_ADD: perform addition operation
  • 14 RETURN_VALUE: returns the return value

So from the current point of view, these bytecodes are no longer difficult, but we can see that the calling function has used CALL_FUNCTION instruction, what does this instruction do?

        case TARGET(CALL_FUNCTION): {
            PREDICTED(CALL_FUNCTION);
            //sp: runtime stack top pointer
            //res: the return value of the function, a PyObject*
            PyObject **sp, *res;
            //Points to the top of the runtime stack
            sp = stack_pointer;
            //Call the function and assign the return value to res. tstate represents the thread object, & SP is obviously a three-level pointer, and oparg represents the operand of the instruction
            res = call_function(tstate, &sp, oparg, NULL);
            stack_pointer = sp;
            PUSH(res);
            if (res == NULL) {
                goto error;
            }
            DISPATCH();
        }

Then the focus is call_ The function function, let's take a look, is also located in ceval C) medium.

#define PyCFunction_Check(op) (Py_TYPE(op) == &PyCFunction_Type)
#define PyFunction_Check(op) (Py_TYPE(op) == &PyFunction_Type)


Py_LOCAL_INLINE(PyObject *) _Py_HOT_FUNCTION
call_function(PyThreadState *tstate, PyObject ***pp_stack, Py_ssize_t oparg, PyObject *kwnames)
{	
    //Get PyFunctionObject object object because pp_stack is in call_ Stack top pointer passed in function instruction
    //The oparg passed in is 0, kwnames is NULL, and pfunc here is make_ PyFunctionObject object created in function
    PyObject **pfunc = (*pp_stack) - oparg - 1;
    //func here is the same as pfunc
    PyObject *func = *pfunc;
    PyObject *x, *w;
    //Processing parameters. For our current function, nkwags and nargs here are both 0    
    Py_ssize_t nkwargs = (kwnames == NULL) ? 0 : PyTuple_GET_SIZE(kwnames);
    Py_ssize_t nargs = oparg - nkwargs;
    //Move stack pointer
    PyObject **stack = (*pp_stack) - nargs - nkwargs;
	
    //Then there are two execution methods, which we will talk about later, but we see that the return value is assigned to x
    if (tstate->use_tracing) {
        x = trace_call_function(tstate, func, stack, nargs, kwnames);
    }
    else {
        x = _PyObject_Vectorcall(func, stack, nargs | PY_VECTORCALL_ARGUMENTS_OFFSET, kwnames);
    }

    assert((x != NULL) ^ (_PyErr_Occurred(tstate) != NULL));

    //Empty function stack
    while ((*pp_stack) > pfunc) {
        w = EXT_POP(*pp_stack);
        Py_DECREF(w);
    }

    return x;
}



static PyObject *
trace_call_function(PyThreadState *tstate,
                    PyObject *func,
                    PyObject **args, Py_ssize_t nargs,
                    PyObject *kwnames)
{
    PyObject *x; //Return value
    //Call_ PyObject_Vectorcall, set the return value to x
    if (PyCFunction_Check(func)) {
        C_TRACE(x, _PyObject_Vectorcall(func, args, nargs, kwnames));
        return x;
    }
    //Don't worry about it for the moment. This is to call a method. Obviously, it is related to the class. We will say when introducing the class
    else if (Py_TYPE(func) == &PyMethodDescr_Type && nargs > 0) {
        PyObject *self = args[0];
        func = Py_TYPE(func)->tp_descr_get(func, self, (PyObject*)Py_TYPE(self));
        if (func == NULL) {
            return NULL;
        }
        C_TRACE(x, _PyObject_Vectorcall(func,
                                        args+1, nargs-1,
                                        kwnames));
        Py_DECREF(func);
        return x;
    }
    return _PyObject_Vectorcall(func, args, nargs | PY_VECTORCALL_ARGUMENTS_OFFSET, kwnames);
}

Then it will call_ PyFunction_FastCallDict} function:

//Objects/call.c
PyObject *
_PyFunction_FastCallDict(PyObject *func, PyObject *const *args, Py_ssize_t nargs,
                         PyObject *kwargs)
{
    PyCodeObject *co = (PyCodeObject *)PyFunction_GET_CODE(func); //Get PyCodeObject object object
    PyObject *globals = PyFunction_GET_GLOBALS(func);//Get global namespace
    PyObject *argdefs = PyFunction_GET_DEFAULTS(func);//Get parameters
    PyObject *kwdefs, *closure, *name, *qualname; //Some other properties
    PyObject *kwtuple, **k;
    PyObject **d;
    Py_ssize_t nd, nk;
    PyObject *result;

    assert(func != NULL);
    assert(nargs >= 0);
    assert(nargs == 0 || args != NULL);
    assert(kwargs == NULL || PyDict_Check(kwargs));
	
    //Let's look at the return below
    //One is function_code_fastcall, one is the last_ PyEval_EvalCodeWithName
    //Function can be seen from the name_ code_ Fastcall is a fast branch, which applies to functions without parameters
    if (co->co_kwonlyargcount == 0 &&
        (kwargs == NULL || PyDict_GET_SIZE(kwargs) == 0) &&
        (co->co_flags & ~PyCF_MASK) == (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE))
    {
        /* Fast paths */
        if (argdefs == NULL && co->co_argcount == nargs) {
            //function_ code_ The logic in fastcall is very simple
            //Directly extract the information such as PyCodeObject and global namespace of function runtime in the current PyFunctionObject
            //Directly create a PyFrameObject object based on the PyCodeObject object object, and then PyEval_EvalFrameEx executes stack frames
            //That is, it really enters the function call and executes the code in the function
            return function_code_fastcall(co, args, nargs, globals);
        }
        else if (nargs == 0 && argdefs != NULL
                 && co->co_argcount == PyTuple_GET_SIZE(argdefs)) {
            /* function called with no arguments, but all parameters have
               a default value: use default values as arguments .*/
            args = _PyTuple_ITEMS(argdefs);
            return function_code_fastcall(co, args, PyTuple_GET_SIZE(argdefs),
                                          globals);
        }
    }
	
    //It is applicable to the case with parameters
    nk = (kwargs != NULL) ? PyDict_GET_SIZE(kwargs) : 0;
    if (nk != 0) {
        Py_ssize_t pos, i;
        kwtuple = PyTuple_New(2 * nk);
        if (kwtuple == NULL) {
            return NULL;
        }

        k = _PyTuple_ITEMS(kwtuple);
        pos = i = 0;
        while (PyDict_Next(kwargs, &pos, &k[i], &k[i+1])) {
            Py_INCREF(k[i]);
            Py_INCREF(k[i+1]);
            i += 2;
        }
        assert(i / 2 == nk);
    }
    else {
        kwtuple = NULL;
        k = NULL;
    }
	
    //Get relevant parameters
    kwdefs = PyFunction_GET_KW_DEFAULTS(func);
    closure = PyFunction_GET_CLOSURE(func);
    name = ((PyFunctionObject *)func) -> func_name;
    qualname = ((PyFunctionObject *)func) -> func_qualname;

    if (argdefs != NULL) {
        d = _PyTuple_ITEMS(argdefs);
        nd = PyTuple_GET_SIZE(argdefs);
    }
    else {
        d = NULL;
        nd = 0;
    }
	
    //If there are parameters, I will take this step now, and the logic will be more complex, but these are later words
    //But it's clear that pyeval will eventually pass_ Evalframeex, and then enter which big for loop
    result = _PyEval_EvalCodeWithName((PyObject*)co, globals, (PyObject *)NULL,
                                      args, nargs,
                                      k, k != NULL ? k + 1 : NULL, nk, 2,
                                      d, nd, kwdefs,
                                      closure, name, qualname);
    Py_XDECREF(kwtuple);
    return result;
}

Therefore, we can see that there are two paths in total, one for non participation and the other for participation, but in the end, they all come to PyEval_EvalFrameEx. Then, the virtual machine executes a new PyCodeObject in the new stack frame, and this PyCodeObject is the PyCodeObject corresponding to the function.

But I'm afraid some people have questions here. We said earlier that PyFrameObject is created based on PyCodeObject, and PyFunctionObject is also created based on PyCodeObject. What's the relationship between PyFrameObject and PyFunctionObject?

If PyCodeObject is compared to "sister", then PyFunctionObject is the "spare tire" of the sister, and PyFrameObject is the "sweetheart" of the sister. Actually, pyeval_ When evalframeex is executed in the stack frame, the influence of PyFunctionObject has disappeared. What really affects the stack frame are PyCodeObject object and global namespace in PyFunctionObject. In other words, in the end, PyFrameObject and PyCodeObject are like glue, which has nothing to do with PyFunctionObject. Therefore, PyFunctionObject works hard and actually makes wedding clothes for others. PyFunctionObject is mainly a packaging and transportation method for PyCodeObject and global namespace.

In addition, we mentioned the fast track here, so what is the function to judge whether it can enter the fast track? The answer is to determine whether you can enter the fast channel in the form of function parameters. Let's take a look at the implementation of parameters in the function.

Implementation of function parameters

The biggest feature of function is that it can pass in parameters, otherwise it can only be simply encapsulated, which is too boring. For Python, what parameters will be passed is unknown to the function. The function body only uses parameters to do some things, such as calling the get method of parameters, but whether you can call the get method depends on what value you pass to the parameters. Therefore, a parameter can be regarded as a placeholder. Let's assume that there is such a thing. We can directly operate it as an existing variable or constant, and then transfer a value to the corresponding parameter when calling, and then the parameter can go through the logic corresponding to the specific value passed in.

Parameter category

In Python, the parameters passed when calling functions can be divided into four categories according to different forms:

def foo(a, b):
    pass
  • Position parameter: foo(a, b), a and B are passed through position parameters
  • Keyword argument: foo (a = 1, b = 2). A and b pass the keyword argument
  • Excess position argument: foo(*args), args is passed through the extension position parameter
  • The excess keyword argument: foo (* * kwargs), which is passed through the extended location parameter

Let's take a look at python call_ How function handles function information:

Py_LOCAL_INLINE(PyObject *) _Py_HOT_FUNCTION
call_function(PyThreadState *tstate, PyObject ***pp_stack, Py_ssize_t oparg, PyObject *kwnames)
{	
    PyObject **pfunc = (*pp_stack) - oparg - 1;
    PyObject *func = *pfunc;
    PyObject *x, *w;
    /*When python virtual machine starts to execute make_ When the function instruction is, an instruction parameter oparg will be obtained first
    oparg It records the number of parameters of the function, including the number of location parameters and keyword parameters.
    Although extended location parameters and extended keyword parameters are more advanced usage, they are essentially composed of multiple location parameters and multiple keyword parameters.
    This means that although there are four parameters in Python, as long as the number of location parameters and keyword parameters are recorded, we can know how many parameters there are and how much memory is needed to maintain parameters.
    */
    //Nkwags is the number of keyword parameters, and nargs is the number of location parameters 
    Py_ssize_t nkwargs = (kwnames == NULL) ? 0 : PyTuple_GET_SIZE(kwnames);
    Py_ssize_t nargs = oparg - nkwargs;
    PyObject **stack = (*pp_stack) - nargs - nkwargs;

Moreover, each instruction in Python is two bytes. The first byte stores the instruction sequence itself and the second byte stores the number of parameters. Since it is one byte, it means that only 255 parameters are allowed at most, but this is enough. But in Python 3 In 8, this restriction was broken.

[root@iZ2ze3ik2oh85c6hanp0hmZ ~]# python3 1.py 
Traceback (most recent call last):
  File "1.py", line 8, in <module>
    print(exec(s))
  File "<string>", line 2
SyntaxError: more than 255 arguments
[root@iZ2ze3ik2oh85c6hanp0hmZ ~]# 

Take my Python 3.0 on Alibaba cloud 6 as an example, it is found that the number of parameters cannot exceed 255, but in Python 3 8, even if there are 1000000 parameters, it is OK. So Python 3 8's source code changes a little. 3.6 and 3.7 are actually similar. The virtual machine implementation code is even highly similar to Python 2. But in Python 3 8. The change is a little big.

Python function internal local variable information can be obtained through co_nlocals and co_argcount to get. It can also be seen from the name that this is not in PyFunctionObject, but in PyCodeObject. co_nlocals, as we said before, this is the number of local variables inside the function, co_argcount is the number of parameters. In fact, function parameters and function local variables are very close. In a sense, function parameters are a kind of function local variables, which are placed continuously in memory. When Python needs to apply for memory space of local variables for functions, it needs to pass co_nlocals knows the total number of local variables. But in that case, it still needs to be done_ What does argcount do? Don't worry. Look at an example

def foo(a, b, c, d=1):
    pass

print(foo.__code__.co_argcount)  # 4
print(foo.__code__.co_nlocals)  # 4


def foo(a, b, c, d=1):
    a = 1
    b = 1

print(foo.__code__.co_argcount)  # 4
print(foo.__code__.co_nlocals)  # 4


def foo(a, b, c, d=1):
    aa = 1

print(foo.__code__.co_argcount)  # 4
print(foo.__code__.co_nlocals)  # 5

The parameter of the function is also a local variable, so Co_ The number of local variables created in the function plus nlocs. Note that the function parameter is also a local variable. For example, the parameter has an a, but the variable in the function body is still a, which is equivalent to re assignment, so it is still equivalent to a parameter. But co_argcount is the number of stored record parameters. So a very obvious conclusion: for any function, co_nlocals is at least equal to or greater than co_argcount.

def foo(a, b, c, d=1, *args, **kwargs):
    pass


print(foo.__code__.co_argcount)  # 4
print(foo.__code__.co_nlocals)  # 6

In addition, we can see that for extended location parameters and extended keyword parameters, co_argcount is not included, because you can not pass, so it is directly calculated as 0. And for CO_ For nlocals, we can certainly get args and kwargs inside the function body, which can be regarded as two parameters. So co_argcount is 4, co_nlocals is 6. In fact, all extension location parameters exist in a PyTupleObject object, and all extension keyword parameters are stored in a PyDictObject object object. For CO, even if not, but for us_ Argcount and Co_ For nlocals, there will be no change, because the values of these two have been determined at the time of compilation.

Transfer of position parameters

Let's take a look at how location parameters are passed:

s = f"""
def f(name, age):
    age = age + 5
    print(name, age)

age = 5
f("satori", age)
"""

if __name__ == '__main__':
    import dis 
    dis.dis(compile(s, "call_function", "exec"))

The bytecode is as follows. Let's analyze it. Of course, we've covered the basic ones.

  2           0 LOAD_CONST               0 (<code object f at 0x00000224C3941450, file "call_function", line 2>)
              2 LOAD_CONST               1 ('f')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (f)

  6           8 LOAD_CONST               2 (5)
             10 STORE_NAME               1 (age)

  7          12 LOAD_NAME                0 (f)
             14 LOAD_CONST               3 ('satori')
             16 LOAD_NAME                1 (age)
             18 CALL_FUNCTION            2 //oparg is 2, indicating that two parameters were passed during the call
             20 POP_TOP
             22 LOAD_CONST               4 (None)
             24 RETURN_VALUE

Disassembly of <code object f at 0x00000224C3941450, file "call_function", line 2>:
  3           0 LOAD_FAST                1 (age) //At this time, the age and the corresponding value already exist in the symbol table and constant pool of the function
              2 LOAD_CONST               1 (5)  //Load constant 5
              4 BINARY_ADD
              6 STORE_FAST               1 (age) //After adding, reuse age to save

  4           8 LOAD_GLOBAL              0 (print) //Load print
             10 LOAD_FAST                0 (name) //Load local variables name and age
             12 LOAD_FAST                1 (age)
             14 CALL_FUNCTION            2  //Function call, obviously print, with two parameters
             16 POP_TOP
             18 LOAD_CONST               0 (None)
             20 RETURN_VALUE

Although the bytecode has been explained, the most important thing is not said. f(name, age). The name and age here are obviously defined by the outer layer, but how are the two variables defined by the outer layer passed to the function f. Let's re analyze the source code:

  7          12 LOAD_NAME                0 (f)
             14 LOAD_CONST               3 ('satori')
             16 LOAD_NAME                1 (age)

We noticed call_ There are three instructions on function. In fact, after these three instructions are executed, the parameters required by the function have been pushed into the runtime stack.

Pass_ PyFunction_FastCallDict() function, and then execute function_code_fastcall.

//Objects/call.c
static PyObject* _Py_HOT_FUNCTION
function_code_fastcall(PyCodeObject *co, PyObject *const *args, Py_ssize_t nargs,
                       PyObject *globals)
{
    PyFrameObject *f; //Stack frame object
    PyThreadState *tstate = _PyThreadState_GET(); //Thread state object
    PyObject **fastlocals; //F - > localsplus, which will be said later
    Py_ssize_t i;
    PyObject *result;

    assert(globals != NULL);
    /* XXX Perhaps we should create a specialized
       _PyFrame_New_NoTrack() that doesn't take locals, but does
       take builtins without sanity checking them.
       */
    assert(tstate != NULL);
    //Create the PyFrameObject corresponding to the function. We can see that the parameter is co, so it is created according to the bytecode instruction
    //Then there is a globals, which represents the global namespace, so we can see that there is actually no PyFunctionObject in the end. It just plays a conveying role
    f = _PyFrame_New_NoTrack(tstate, co, globals, NULL);
    if (f == NULL) {
        return NULL;
    }

    fastlocals = f->f_localsplus;

    for (i = 0; i < nargs; i++) {
        Py_INCREF(*args);
        fastlocals[i] = *args++;
    }
    //Key: copy function parameters from runtime stack to pyframeobject f_ localsplus
    result = PyEval_EvalFrameEx(f,0);

    if (Py_REFCNT(f) > 1) {
        Py_DECREF(f);
        _PyObject_GC_TRACK(f);
    }
    else {
        ++tstate->recursion_depth;
        Py_DECREF(f);
        --tstate->recursion_depth;
    }
    return result;
}

From the source code, we can see that through_ PyFrame_New_NoTrack creates the PyFrameObject object corresponding to function f, and the parameter is the PyCodeObject object saved in the PyFunctionObject object corresponding to co. Then, the Python virtual machine copies the parameters one by one to the F of the newly created PyFrameObject object object_ In localsplus. When analyzing the Python virtual machine framework, we know that this f_ The memory block pointed to by localsplus also stores the runtime stack used by the Python virtual machine. So what is the relationship between the memory occupied by parameters and the memory occupied by the runtime stack?

//frameobject.c

//This is_ PyFrame_New_NoTrack, the external exposure is PyFrame_New, but this is essentially called
PyFrameObject* _Py_HOT_FUNCTION
_PyFrame_New_NoTrack(PyThreadState *tstate, PyCodeObject *code,
                     PyObject *globals, PyObject *locals)
{
    PyFrameObject *back = tstate->frame;
    PyFrameObject *f;
    PyObject *builtins;
    Py_ssize_t i;

    //...
    //...
        Py_ssize_t extras, ncells, nfrees;
        ncells = PyTuple_GET_SIZE(code->co_cellvars);
        nfrees = PyTuple_GET_SIZE(code->co_freevars);
        extras = code->co_stacksize + code->co_nlocals + ncells + nfrees;
        if (free_list == NULL) {
            //For f_localsplus applies for memory space. The size is extras. Note this extras. We can see that it is actually divided into four parts
            //They are: runtime stack, local variable, cell object and free object. Note: but they are not in this order in memory
            f = PyObject_GC_NewVar(PyFrameObject, &PyFrame_Type,
            extras);
            if (f == NULL) {
                Py_DECREF(builtins);
                return NULL;
            }
        }
        else {
            //...
        }
		
        f->f_code = code;
        //Get the number of local variables + the number of cell objects + the number of free objects
        extras = code->co_nlocals + ncells + nfrees;
        f->f_valuestack = f->f_localsplus + extras;
        for (i=0; i<extras; i++)
            f->f_localsplus[i] = NULL;
        f->f_locals = NULL;
        f->f_trace = NULL;
    }
    //...
    f->f_lasti = -1;
    f->f_lineno = code->co_firstlineno;
    f->f_iblock = 0;
    f->f_executing = 0;
    f->f_gen = NULL;
    f->f_trace_opcodes = 0;
    f->f_trace_lines = 1;

    return f;
}

As mentioned earlier, in the CO of the PyCodeObject object object corresponding to the function_ The nlocals field contains the number of function parameters, because function parameters are also a kind of local symbols. So from F_ Starting with localsplus, there must be memory in extras for function parameters. In other words, the parameters of the function are stored in the memory before the runtime stack.

In addition, from_ PyFrame_ New_ In notrack, we can see that in the array f_ The space in which function parameters are stored in localsplus and the space in the runtime stack are logically separated and do not share the same piece of memory. Although they are continuous, the two are known to each other, but they are completely different and do not communicate with each other.

After processing the parameters, pyeval has not been entered_ Evalframeex, so the runtime stack is empty at this time. But the argument of the function is already in F_ In localsplus. Therefore, the F of the PyFrameObject object object is created_ Localsplus is like this:

Access to location parameters

When the action of parameter copying is completed, it will enter the new pyeval_ The real action of evalmex is called.

  3           0 LOAD_FAST                1 (age) 
              2 LOAD_CONST               1 (5)  
              4 BINARY_ADD
              6 STORE_FAST               1 (age) 

First, the parameters must be read and written through LOAD_FAST,LOAD_CONST,STORE_FAST is completed by these instruction sets.

//ceval.c
PyObject* _Py_HOT_FUNCTION
_PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
{
    ...
    ...
    fastlocals = f->f_localsplus;
    ...
}

//A macro. The fastlocals here is obviously F - > localsplus
#define GETLOCAL(i)     (fastlocals[i])

        case TARGET(LOAD_FAST): {
            //Get the value with index oparg from fastlocals
            PyObject *value = GETLOCAL(oparg);
            if (value == NULL) {
                format_exc_check_arg(tstate, PyExc_UnboundLocalError,
                                     UNBOUNDLOCAL_ERROR_MSG,
                                     PyTuple_GetItem(co->co_varnames, oparg));
                goto error;
            }
            Py_INCREF(value);
            PUSH(value);
            FAST_DISPATCH();
        }


        case TARGET(STORE_FAST): {
            PREDICTED(STORE_FAST);
            PyObject *value = POP(); //Pop up element
            SETLOCAL(oparg, value);  //Set the element with index oparg to value
            FAST_DISPATCH();
        }

So we found that LOAD_FAST and STORE_FAST is a pair of instructions based on f_localsplus is a piece of memory for the operation target, and the instruction 0 load_ The result of fast 1 (age) is that f_ The object corresponding to localsplus [1] is pushed into the runtime stack. After the addition operation is completed, the result is passed through STORE_FAST into f_ In localsplus [1], the update of a is realized, and the result of print(a) in the future is 10.

Now it's clear how Python's positional parameters are passed during function call and accessed during function execution. When calling a function, python pushes the value of the function parameter into the runtime stack from left to right, while in call_ Call in function_ PyFunction_FastCallDict, and then call function_code_fastcall, while in function_ code_ In fastcall, these parameters are copied to f of and PyFrameObject object object in turn_ In localsplus. The final effect is that the python virtual machine stores the parameters used in the function call in the F of the new PyFrameObject object from left to right_ In localsplus.

Therefore, when accessing function parameters, the python virtual machine does not check the namespace according to the usual method of accessing symbols, but directly accesses F through an index (offset position)_ The value corresponding to the symbol stored in localsplus. Yes, f_localsplus stores symbols (variable names), not specific values. Because we say that a variable in Python is just a pointer. Whether the value changes depends on whether the corresponding value is a variable object or an immutable object, rather than whether to change it by passing a value or pointer like other programming languages. Therefore, this way of accessing parameters through index (offset position) is also the origin of position parameters.

Default parameters

A feature of Python functions is that they support default parameters, which is very convenient. Let's take a look at the implementation mechanism.

s = """
def foo(a=1, b=2):
    print(a + b)

foo()
"""

if __name__ == '__main__':
    import dis
    dis.dis(compile(s, "default", "exec"))
  2           0 LOAD_CONST               5 ((1, 2))//We can see that the default value has been loaded in the constructor
              2 LOAD_CONST               2 (<code object foo at 0x000002076ED83BE0, file "default", line 2>)
              4 LOAD_CONST               3 ('foo')
              6 MAKE_FUNCTION            1 (defaults) 
              8 STORE_NAME               0 (foo)

  5          10 LOAD_NAME                0 (foo)
             12 CALL_FUNCTION            0
             14 POP_TOP
             16 LOAD_CONST               4 (None)
             18 RETURN_VALUE

Disassembly of <code object foo at 0x000002076ED83BE0, file "default", line 2>:
  3           0 LOAD_GLOBAL              0 (print)
              2 LOAD_FAST                0 (a)
              4 LOAD_FAST                1 (b)
              6 BINARY_ADD
              8 CALL_FUNCTION            1
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE

Comparing the functions without default parameters at the beginning, we will find that compared with the functions without default parameters, the functions with default parameters, in addition to the PyCodeObject and foo symbols corresponding to the load function body, will first give the value of the default parameters to load and push them into the runtime stack. However, we found that the default parameters are combined into a tuple, and let's take a look at make_ For the function instruction, we found that the following parameters are 1 (defaults), and the previous parameters are 0. What is this 1? It also prompts us with a default. We know that the PyFunctionObject object object has a func_defaults, is there a relationship between the two? Then take a look at make with these questions_ Function instruction.

        case TARGET(MAKE_FUNCTION): {
            PyObject *qualname = POP();
            PyObject *codeobj = POP();
            PyFunctionObject *func = (PyFunctionObject *)
                PyFunction_NewWithQualName(codeobj, f->f_globals, qualname);

            Py_DECREF(codeobj);
            Py_DECREF(qualname);
            if (func == NULL) {
                goto error;
            }

            if (oparg & 0x08) {
                assert(PyTuple_CheckExact(TOP()));
                func ->func_closure = POP();
            }
            if (oparg & 0x04) {
                assert(PyDict_CheckExact(TOP()));
                func->func_annotations = POP();
            }
            if (oparg & 0x02) {
                assert(PyDict_CheckExact(TOP()));
                func->func_kwdefaults = POP();
            }
            
            ////The default parameters, we found, are indeed stored in func_ In defaults
            if (oparg & 0x01) {
                assert(PyTuple_CheckExact(TOP()));
                func->func_defaults = POP();
            }

            PUSH((PyObject *)func);
            DISPATCH();
        }

Through the above command, we can easily see that make_ The function instruction not only creates a PyFunctionObject object, but also handles the default values of parameters. MAKE_ The function instruction parameter indicates that there are default values in the current runtime stack, but the specific number of default values cannot be seen through the parameters, because the default values will be inserted into a PyTupleObject object object in order, so the whole is equivalent to one. Pyfunction is then called_ Setdefaults sets the PyTupleObject object to PyFunctionObject func_ The value of defaults can be used at the python level__ defaults__ visit. In this way, the default value of function parameters has also become a part of PyFunctionObject object. The default value of function and its parameters is finally bound by Python virtual machine. Like PyCodeObject and global namespace, it is also stuffed into the big burden of PyFunctionObject. Therefore, the wedding dress of PyFunctionObject is very thorough. The tool person PyFunctionObject object, give me a praise.

int
PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
{
    if (!PyFunction_Check(op)) {
        PyErr_BadInternalCall();
        return -1;
    }
    if (defaults == Py_None)
        defaults = NULL;
    else if (defaults && PyTuple_Check(defaults)) {
        Py_INCREF(defaults);
    }
    else {
        PyErr_SetString(PyExc_SystemError, "non-tuple default args");
        return -1;
    }
    //The func of the PyFunctionObject object_ The defaults member is set to defaults
    Py_XSETREF(((PyFunctionObject *)op)->func_defaults, defaults);
    return 0;
}

Let's take this foo function as an example to see the underlying implementation corresponding to different call methods:

def foo(a=1, b=2):
    print(a + b)

Execute foo() directly without passing in parameters

PyObject *
_PyFunction_FastCallDict(PyObject *func, PyObject *const *args, Py_ssize_t nargs,
                         PyObject *kwargs)
{	
    //Get PyCodeObject of PyFunctionObject
    PyCodeObject *co = (PyCodeObject *)PyFunction_GET_CODE(func);
    //Get the global namespace of PyFunctionObject
    PyObject *globals = PyFunction_GET_GLOBALS(func);
    //Gets the default value of PyFunctionObject
    PyObject *argdefs = PyFunction_GET_DEFAULTS(func);
    //Some additional attributes
    PyObject *kwdefs, *closure, *name, *qualname;
    PyObject *kwtuple, **k;
    PyObject **d;
    Py_ssize_t nd, nk;
    PyObject *result;

    assert(func != NULL);
    assert(nargs >= 0);
    assert(nargs == 0 || args != NULL);
    assert(kwargs == NULL || PyDict_Check(kwargs));
	
    //Here to judge whether to enter the fast channel, a function must meet two conditions if it wants to enter the fast channel
    //1. Default parameters are not allowed during function definition; 2. When calling a function, it must be specified through the location parameter.
    //So we can detect co here_ Are kwonlyargcount and kwargs both zero
    if (co->co_kwonlyargcount == 0 &&
        (kwargs == NULL || PyDict_GET_SIZE(kwargs) == 0) &&
        (co->co_flags & ~PyCF_MASK) == (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE))
    {
        //Then continue to detect: the nargs here is through call_function passed
        //And this nargs is calling_ Py in function_ ssize_ t nargs = oparg - nkwargs;
        //So nargs here is the number of parameters passed minus the number of parameters passed through keyword parameters
        //And co_argcount is the total number of function parameters, so once even one parameter is passed in the form of keyword parameter, it will cause the two to be unequal and cannot enter the fast channel
        if (argdefs == NULL && co->co_argcount == nargs) {
            return function_code_fastcall(co, args, nargs, globals);
        }
        
        
        //However, such conditions are indeed a little harsh. After all, how can parameters have no default values? So Python also provides a way to get into the fast track
        //We find that under the premise of default, if nargs = = 0 & & Co - > CO can be satisfied_ argcount == PyTuple_ GET_ Size (argdefs) can also enter the fast track
        //co->co_ argcount == PyTuple_ GET_ Size (argdefs) requires that the number of function parameters must be equal to the number of default parameters, that is, all function parameters are default parameters
        //nargs==0 is the number of parameters to be passed in minus the number of parameters passed through keyword parameters, which is equal to 0, that is, either no parameters are passed (all default parameters are used) or all parameters are passed through keyword parameters.
        //This way can also enter the fast track
        else if (nargs == 0 && argdefs != NULL
                 && co->co_argcount == PyTuple_GET_SIZE(argdefs)) {
            args = _PyTuple_ITEMS(argdefs);
            return function_code_fastcall(co, args, PyTuple_GET_SIZE(argdefs),
                                          globals);
        }
    }
	
    
    //If the above two points cannot be satisfied, then there is no way but to take the conventional method
    //Get information about default parameters
    nk = (kwargs != NULL) ? PyDict_GET_SIZE(kwargs) : 0;
    if (nk != 0) {
        Py_ssize_t pos, i;

        kwtuple = PyTuple_New(2 * nk);
        if (kwtuple == NULL) {
            return NULL;
        }

        k = _PyTuple_ITEMS(kwtuple);
        pos = i = 0;
        while (PyDict_Next(kwargs, &pos, &k[i], &k[i+1])) {
            Py_INCREF(k[i]);
            Py_INCREF(k[i+1]);
            i += 2;
        }
        assert(i / 2 == nk);
    }
    else {
        kwtuple = NULL;
        k = NULL;
    }
	
    //Here are some properties of the function, such as default keyword parameters, closures, etc
    kwdefs = PyFunction_GET_KW_DEFAULTS(func);
    closure = PyFunction_GET_CLOSURE(func);
    name = ((PyFunctionObject *)func) -> func_name;
    qualname = ((PyFunctionObject *)func) -> func_qualname;
	
    //Get the address of the value of the default parameter and the number of default parameters
    if (argdefs != NULL) {
        d = _PyTuple_ITEMS(argdefs);
        nd = PyTuple_GET_SIZE(argdefs);
    }
    else {
        d = NULL;
        nd = 0;
    }
	
    //Call_ PyEval_EvalCodeWithName, PyCodeObject object passed in function and parameter information
    result = _PyEval_EvalCodeWithName((PyObject*)co, globals, (PyObject *)NULL,
                                      args, nargs, 
                                      k, k != NULL ? k + 1 : NULL, nk, 2,
                                      d, nd, kwdefs,
                                      closure, name, qualname);
    Py_XDECREF(kwtuple);
    return result;
}

_ PyEval_EvalCodeWithName is a very important function, which will be encountered later when analyzing the extension location parameters and extension keyword parameters.

PyObject *
_PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject *const *args, Py_ssize_t argcount,
           PyObject *const *kwnames, PyObject *const *kwargs,
           Py_ssize_t kwcount, int kwstep,
           PyObject *const *defs, Py_ssize_t defcount,
           PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{	
    //PyCodeObject object, via_ PyFunction_ Get the func received in fastcalldict
    PyCodeObject* co = (PyCodeObject*)_co;
    //Stack frame
    PyFrameObject *f;
    //Return value
    PyObject *retval = NULL;
    //F - > localsplus, and co - > co_freevars, this co_freevars, and co_freevars are all closure related
    PyObject **fastlocals, **freevars;
    PyObject *x, *u;
    //Total number of parameters: the number of parameters that can be passed through location parameters + the number of parameters that can only be passed through keyword parameters
    const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount;
    Py_ssize_t i, j, n;
    PyObject *kwdict;

    PyThreadState *tstate = _PyThreadState_GET();  //Get thread state object
    assert(tstate != NULL);

    if (globals == NULL) {
        _PyErr_SetString(tstate, PyExc_SystemError,
                         "PyEval_EvalCodeEx: NULL globals");
        return NULL;
    }
	
    //Create stack frame
    f = _PyFrame_New_NoTrack(tstate, co, globals, locals);
    if (f == NULL) {
        return NULL;
    }
    fastlocals = f->f_localsplus;
    freevars = f->f_localsplus + co->co_nlocals;
	
    //Remember this co_flags? We say that it is used to judge parameters. If the result of "and operation" with 0x08 is true, it indicates that there are * * kwargs
    //If the result of "and operation" with 0x04 is true, it indicates that there is * args
    if (co->co_flags & CO_VARKEYWORDS) {
        kwdict = PyDict_New();
        if (kwdict == NULL)
            goto fail;
        i = total_args;
        if (co->co_flags & CO_VARARGS) {
            i++;
        }
        SETLOCAL(i, kwdict);
    }
    else {
        kwdict = NULL;
    }
	
    //argcount is the number of location parameters actually transmitted, co - > Co_ argcount is the number of parameters that can be passed through the location parameter
    //If argcount > co - > Co_ Argcount, which proves that there is an extension parameter, otherwise there is no extension parameter    
    if (argcount > co->co_argcount) {
        //So here n equals co - > Co_ argcount
        n = co->co_argcount;
    }
    else {
        //If there is no extended location parameter, the caller passes several through the location parameter, and n is several
        n = argcount;
    }
    
    
    //Then let's take a closer look at this n. suppose we define a function def foo(a, b, c=1,d=2, *args)
    //If argcount > co - > Co_ Argcount, indicating that the number of location parameters we passed exceeds 4, but n is 4
    //But if we only pass two, such as foo ('a ',' B '), then n is obviously 2
    //The following is to set the values of the passed parameters to f in turn_ Go to localsplus, where i is the index and x is the value.
    for (j = 0; j < n; j++) {
        x = args[j];
        Py_INCREF(x);
        SETLOCAL(j, x);
    }
	
    //The following is obviously the logic of extending location parameters. We'll skip it for the time being and talk about it later
    if (co->co_flags & CO_VARARGS) {
        u = _PyTuple_FromArray(args + n, argcount - n);
        if (u == NULL) {
            goto fail;
        }
        SETLOCAL(total_args, u);
    }
	
    //Similarly, after the keyword
    kwcount *= kwstep;
    for (i = 0; i < kwcount; i += kwstep) {
       //......

    //It will be tested again here, argcount > co - > Co_ Argcount indicates that we have passed more, and then check whether * args exists
    //If co - > Co_ flags & CO_ If varargs is False, an error will be reported directly
    if ((argcount > co->co_argcount) && !(co->co_flags & CO_VARARGS)) {
        too_many_positional(tstate, co, argcount, defcount, fastlocals);
        goto fail;
    }
	
    //If the number of parameters passed in is less than the number of parameters defined by the function, it proves that there are default parameters.
    //defcount indicates the number of default parameters set
    if (argcount < co->co_argcount) {
        //Obviously, m = total number of parameters (excluding the number of all formal parameters except * args and * * kwargs) - the number of default parameters
        Py_ssize_t m = co->co_argcount - defcount;
        Py_ssize_t missing = 0;
        //Therefore, m is the total number of parameters that need to be passed without default values
        for (i = argcount; i < m; i++) {
            //i=argcount is the total number of location parameters passed when we call the function
            //For example, a function receives six parameters, but two of them are default parameters, so this means that the caller needs to pass at least four parameters if they are passed by location parameters
            //If we only pass two, m is 4 and i is 2 at the end of the loop
            if (GETLOCAL(i) == NULL) {
                //Then from F - > F via GETLOCAL_ Get the default value in localsplus because in make_ The default value has been saved in the function stage
                //If it is not enough, place your hope on the default value. Once it is not found, missing: the number of missing parameters will be + 1
                missing++;
            }
        }
        //Then, according to our above logic, there are obviously two that have not been passed, but they will use the default value
        //If only three parameters are passed, there are obviously three parameters that are not passed, but the default value is only two, so missing is not 0
        if (missing) {
            //Throw an exception directly
            missing_arguments(tstate, co, missing, defcount, fastlocals);
            goto fail;
        }
        
        
        //It may be difficult to understand below. We say that this m is the number of parameters that need to be passed by the caller
        //N is the number of parameters passed in the form of position parameters. If it is less than the number of function parameters, n is the number of parameters passed in. If it is greater than the number of function parameters, n is the number of function parameters. For example:
        /*
        def foo(a, b, c, d=1, e=2, f=3):
        	pass
        This is a function with 6 parameters. Obviously, m is 3. In fact, when the function is defined, m is a constant value, which is the total number of parameters without default parameters
        But we can call foo(1,2,3), that is, only three are passed, so n here is 3,
        foo(1, 2, 3, 4, 5),So obviously n=5 and m is still 3
        */        
        if (n > m)
            //So now the logic here is well understood. Suppose foo(1, 2, 3, 4, 5) is called
            //Since three parameters are the default parameters, only three can be passed in the call, but five are passed here, and the first three are required
            //As for the latter two, it means that I don't want to use the default value. I want to pass it again, and only the last one uses the default value
            //Therefore, this i is the number of parameters that can use the default value but are not used            
            i = n - m;
        else
            //In addition, if it is passed according to the position parameters, the program can go to this step, indicating that there is no lack of transmission
            //So this n is at least > = m, so if n == m, then n is 0
            i = 0;
        for (; i < defcount; i++) {     
            //The value of the default parameter has been pushed into the stack from the beginning. As a PyTupleObject object, it is set to func as a whole_ Defaults in this field
            //But for the parameters of the function, it must be set to f_localsplus goes inside, and it can only be in the back.
            //Because the order of default parameters should be after non default parameters            
            if (GETLOCAL(m+i) == NULL) {
                //Here is the value corresponding to index i from func_ Take the PyTupleObject corresponding to defaults
                //This i is either n-m or 0. According to the previous example, the function receives six parameters, but we pass five
                //Therefore, we only need to copy the last element with index 2 to F_ Just go inside localsplus.
                //And n=5, m=3, obviously i = 2
                //So what if we pass three?
                //Obviously, i is 0, because at this time n==m, it means that the default parameters use the default value. In this case, copy from the beginning.
                //Similarly, four parameters are passed to prove that the default value of the first default parameter is unnecessary, so you only need to copy the latter two
                //Obviously, you have to copy from the position with index 1 to the end. At this time, n-m, that is, i, is exactly 1
                //Therefore, n-m is "the index of the first value in the PyTupleObject object composed of default parameter values that needs to be copied to f_localsplus"
                //Then I < defcount; I + +, copy to the end                             
                PyObject *def = defs[i];
                Py_INCREF(def);
                //Set the value to F_ In localsplus, the index here is obviously m+i
                //For example: def foo(a,b,c,d=1,e=2,f=3)
                //foo(1, 2, 3, 4), obviously d will not use the default value, so you only need to copy the last two default values to e and f
                //Obviously, e and f are in f according to the order_ The corresponding indexes in localsplus are 4 and 5
                //M is 3, i is n-m equals 4-3 equals 1, so m+i is exactly 4,
                //f_localsplus: [1, 2, 3, 4]
                //PyTupleObject:(1, 2, 3)
                //Therefore, the element with index i in PyTupleObject is copied to f_localsplus is exactly the position corresponding to m+i               
                SETLOCAL(m+i, def);
            }
        }
    }

    //........
    return retval;
}

Therefore, through the above, we know what the default value of the location parameter is.

Pass in a keyword parameter and execute foo(b=3)

In the second call to foo, we specified b=3, but the call method is essentially the same. In call_ Before function, the python virtual machine pushed the pyunicode object object b and PyLongObject object object 3 into the runtime stack.

PyObject *
_PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject *const *args, Py_ssize_t argcount,
           PyObject *const *kwnames, PyObject *const *kwargs,
           Py_ssize_t kwcount, int kwstep,
           PyObject *const *defs, Py_ssize_t defcount,
           PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{	
    PyCodeObject* co = (PyCodeObject*)_co;
    PyFrameObject *f;
    //.......
    f = _PyFrame_New_NoTrack(tstate, co, globals, locals);
    //......
    if (co->co_flags & CO_VARKEYWORDS) {
        //.......
    }
    else {
        //.......
    }

    if (argcount > co->co_argcount) {
        n = co->co_argcount;
    }
    else {
        n = argcount;
    }
    for (j = 0; j < n; j++) {
        //.......
    }

    if (co->co_flags & CO_VARARGS) {
        //......
    }

    /* Traverse the keyword parameters to determine whether the keyword parameter name appears in the def statement of the function */
    kwcount *= kwstep;
    for (i = 0; i < kwcount; i += kwstep) {
        PyObject **co_varnames;  //Symbol table
        PyObject *keyword = kwnames[i]; //Get parameter name
        PyObject *value = kwargs[i];  //Get parameter value
        Py_ssize_t j;
		
        //Obviously, the parameter must be a string, so you can do this in the dictionary: {* * {1: "a", 2: "b"}}
        //But you can't do this: dict(**{1: "a", 2: "b"})
        if (keyword == NULL || !PyUnicode_Check(keyword)) {
            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() keywords must be strings",
                          co->co_name);
            goto fail;
        }
		
        //The logic here will be described in detail later. In short, the core is to detect whether a parameter is passed through the location parameter and keyword parameter at the same time, that is, to judge whether it is passed twice
        co_varnames = ((PyTupleObject *)(co->co_varnames))->ob_item;
        //Look for keyword parameters in the symbol table of the function. Note: j here does not start from 0, but from posonlyargcount
        //Because in Python 3 / is introduced in 8. The parameters before / can only be passed through position parameters
        for (j = co->co_posonlyargcount; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            if (name == keyword) {
                goto kw_found;
            }
        }

        /* The logic is the same as above */
        for (j = co->co_posonlyargcount; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            int cmp = PyObject_RichCompareBool( keyword, name, Py_EQ);
            if (cmp > 0) {
                goto kw_found;
            }
            else if (cmp < 0) {
                goto fail;
            }
        }
			
        assert(j >= total_args);
        if (kwdict == NULL) {
			
            //If the specified symbol does not appear in the symbol table, it indicates that an unnecessary keyword parameter appears (of course * * kwargs later)
            if (co->co_posonlyargcount
                && positional_only_passed_as_keyword(tstate, co,
                                                     kwcount, kwnames))
            {
                goto fail;
            }

            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() got an unexpected keyword argument '%S'",
                          co->co_name, keyword);
            goto fail;
        }

        if (PyDict_SetItem(kwdict, keyword, value) == -1) {
            goto fail;
        }
        continue;

      kw_found:
        if (GETLOCAL(j) != NULL) {
            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() got multiple values for argument '%S'",
                          co->co_name, keyword);
            goto fail;
        }
        Py_INCREF(value);
        SETLOCAL(j, value);
    }

    //.......
    return retval;
}

At compile time: Python will record the symbols appearing in the def statement of the function in the symbol table (co_varnames). As we have seen, in the instruction sequence of foo(b=3), the python virtual machine is executing call_ Before the function instruction, the names of keyword parameters will be pushed into the runtime stack_ PyEval_EvalCodeWithName can use the name of the keyword parameter saved in the runtime stack to get the CO at Python compilation time_ Find in varnames. Best of all, in Co_ The order of variable names defined in varnames is determined by rules. And through the analysis just now, we also know that in the F of PyFrameObject object object_ In the memory maintained by localsplus, the memory used to store function parameters is also arranged according to the same rule. So in Co_ When the parameter name of keyword parameter is searched in varnames, we can directly set f according to the obtained sequence number information_ Memory in localsplus, which sets the default parameter to the value desired by the function caller.

So we can give another simple example to summarize. def foo(a, b, c, d=1,e=2, f=3), for such a function. First, the python virtual machine knows that the caller must pass at least parameters to a, b and c. If it is foo(1), then 1 will be passed to a, but b and c do not receive values, so an error is reported. However, if foo(1, e=4, c=2, b=3) or the old rule 1 is passed to a, it is still not enough. At this time, we will place our hope on the keyword parameters. And we said the definition order and F of def parameters_ The order and CO of parameters stored in memory maintained by localsplus_ The order of parameters in varnames is consistent. Therefore, the keyword parameters do not pay attention to the order. When e=4 is found, the python virtual machine passes co_varnames symbol table, you know to set E to F_ In localsplus, where the index is 4, c=2, set the index to 2, and b=3, set the index to 1. Then, when the location parameters and keyword parameters are set, the python virtual opportunity detects whether all the parameters to be passed, that is, the parameters without default values, have been passed by the caller.

But let's add another sentence here. We say that the keyword parameter setting is specifically set in F_ Where in localsplus is the key parameter name substituted into co_ It is obtained by searching in the varnames symbol table, but if the parameter name of this keyword parameter is not in Co_ What about varnames? In addition, when we talk about location parameters, if the location parameters passed are better than co_argcount is more. What should I do? Yes, if you are smart, you must know. Next, we will introduce extended keywords and extended location parameters.

Extended location parameter and extended keyword parameter

Previously, we saw the value of the number of instruction parameters when using extended location parameters and extended keyword parameters. Let's see it again.

def foo(a, b, *args, **kwargs):
    pass


print(foo.__code__.co_nlocals)  # 4
print(foo.__code__.co_argcount)  # 2

We see that for CO_ For nlocals, it counts the number of all local variables, and the result is 4; But for co_argcount counts the number of all parameters excluding * args * * kwargs, so the result is 2. In that case, as we analyzed earlier, * args can receive multiple location parameters, but eventually these parameters will be placed in the PyTupleObject object object args** Kwargs can receive multiple keyword parameters, but these keyword parameters will form a PyDictObject object object, which is pointed by kwargs. In fact, it is true. Even if we do not analyze it from the perspective of source code, we can draw this conclusion from the actual use of Python.

def foo(*args, **kwargs):
    print(args)
    print(kwargs)


foo(1, 2, 3, a=1, b=2, c=3)
"""
(1, 2, 3)
{'a': 1, 'b': 2, 'c': 3}
"""

foo(*(1, 2, 3), **{"a": 1, "b": 2, "c": 3})
"""
(1, 2, 3)
{'a': 1, 'b': 2, 'c': 3}
"""

Of course, if * is used for a tuple, list or even string, the iteratable object will be directly broken up, which is equivalent to passing multiple location parameters. Similarly, if * * is used for a dictionary, it is equivalent to passing multiple keyword parameters.

Let's take a look at how the extension parameters are implemented. First, enter_ PyEval_EvalCodeWithName comes from this function. Of course, this function should be familiar. Let's take a look at the processing of extension parameters.

PyObject *
_PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
           //Information about position parameters
           PyObject *const *args, Py_ssize_t argcount,
           //Information about keyword parameters                 
           PyObject *const *kwnames, PyObject *const *kwargs,
           Py_ssize_t kwcount, int kwstep, //Number of keyword parameters
           PyObject *const *defs, Py_ssize_t defcount,
           //Default value, closure, function name, fully qualified name and other information              
           PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{
    PyCodeObject* co = (PyCodeObject*)_co;//Get the PyCodeObject of PyFunctionObject
    PyFrameObject *f;//Declare a PyFrameObject
    PyObject *retval = NULL;
    PyObject **fastlocals, **freevars;
    PyObject *x, *u;
    //Get the number of total parameters
    const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount;
    Py_ssize_t i, j, n;
    PyObject *kwdict;
    //........
    //Create a stack frame
    f = _PyFrame_New_NoTrack(tstate, co, globals, locals);
    if (f == NULL) {
        return NULL;
    }
    
    //All parameters of the function
    fastlocals = f->f_localsplus;
    //closure
    freevars = f->f_localsplus + co->co_nlocals;

    //Judge whether to pass the extended keyword parameter, CO_VARKEYWORDS and the following CO_VARARGS are identifiers
    //It is used to judge whether there are extended keyword parameters and extended location parameters
    if (co->co_flags & CO_VARKEYWORDS) {
        //Create a dictionary
        kwdict = PyDict_New();
        if (kwdict == NULL)
            goto fail;
        //i is the total number of parameters, and the assumed value foo(a, b, c, *args, **kwargs)
        i = total_args;
        //If the extended location parameter is also passed, i should add 1
        //Because even for extension, the keyword parameter should still be after the location parameter
        if (co->co_flags & CO_VARARGS) {
            i++;
        }
	    //If there is no extended location parameter, kwdict should be in the position with index 3
        //With the extended location parameter, it is obviously reasonable that kwdit is in the position with index 4
        //Then put it in F_ In localsplus        
        SETLOCAL(i, kwdict);
    }
    else {
        //If not, NULL
        kwdict = NULL;
    }

    /* As we mentioned earlier, the location parameter is copied locally (obviously, the extended location parameter is not included here) */
    if (argcount > co->co_argcount) {
        n = co->co_argcount;
    }
    else {
        n = argcount;
    }
    for (j = 0; j < n; j++) {
        x = args[j];
        Py_INCREF(x);
        SETLOCAL(j, x);
    }

    /* Here's the key. Copy the redundant location parameters into * args */
    if (co->co_flags & CO_VARARGS) {
        //Request a tuple of argcount - n size
        u = _PyTuple_FromArray(args + n, argcount - n);
        if (u == NULL) {
            goto fail;
        }
        //Put in F - > F_ Go inside localsplus
        SETLOCAL(total_args, u);
    }

    //The following is the copy extension keyword parameter, but we found that the symbol and value information are obtained from the two arrays
    //Therefore, combined with the variable declaration at the top, we can see that the keyword parameters we pass are not set in the dictionary
    //Instead, the symbols and values are stored in the corresponding arrays, which are obviously kwnames and kwargs below
    //Then use the index traversal and take out in order. By comparing whether the symbols of the passed keyword parameters have appeared in the parameters defined by the function
    //To determine whether the passed parameter is an ordinary keyword parameter or an extended keyword parameter
    //For example: def foo(a, b, c, **kwargs), then foo(1, 2, c=3, d=4)
    //Obviously, there are two keyword parameters, c=3 and d=4. Then c has appeared in the parameters of the function definition, so c is an ordinary keyword parameter
    //But d doesn't. all d are also extended keyword parameters, so it should be set in the kwargs dictionary
    kwcount *= kwstep;
    //Traverse in order according to the index
    for (i = 0; i < kwcount; i += kwstep) {
        PyObject **co_varnames; //Symbol table
        PyObject *keyword = kwnames[i];//key of keyword parameter
        PyObject *value = kwargs[i];//value of keyword parameter
        Py_ssize_t j;

        if (keyword == NULL || !PyUnicode_Check(keyword)) {
            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() keywords must be strings",
                          co->co_name);
            goto fail;
        }

        //Get the symbol table and get all the symbols, so you can know what the function parameters are
        co_varnames = ((PyTupleObject *)(co->co_varnames))->ob_item;
        //We see another for loop inside
        //First, the outer loop traverses all keyword parameters, that is, the parameters we pass
        //The inner loop traverses the symbol table, that is, all the parameters of the function
        for (j = co->co_posonlyargcount; j < total_args; j++) {
            //The symbols of each keyword parameter we send will be compared with all symbols in the symbol table
            PyObject *name = co_varnames[j];
            //If equal, it means that the keyword parameter is passed, not the extended keyword parameter
            if (name == keyword) {
                //Then kw_ In the found label, it will detect whether the corresponding parameters are passed through the position parameters
                //If the position parameter has been passed, it is obvious that a parameter has been passed twice
                goto kw_found;
            }
        }

        /* The logic is the same as above */
        for (j = co->co_posonlyargcount; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            int cmp = PyObject_RichCompareBool( keyword, name, Py_EQ);
            if (cmp > 0) {
                goto kw_found;
            }
            else if (cmp < 0) {
                goto fail;
            }
        }

        assert(j >= total_args);
        //If you come here, you must have passed in a symbol, which is not in the symbol table_ Keyword parameters in varnames
        //If kwdict is NULL, it proves that the basic function does not define extension parameters at all, then it will directly report an error
        if (kwdict == NULL) {

            if (co->co_posonlyargcount
                && positional_only_passed_as_keyword(tstate, co,
                                                     kwcount, kwnames))
            {
                goto fail;
            }

            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() got an unexpected keyword argument '%S'",
                          co->co_name, keyword);
            goto fail;
        }
		
        //Here, set the keyword and value belonging to the extended keyword parameter to the dictionary created before
        //Then continue goes to the next keyword parameter logic
        if (PyDict_SetItem(kwdict, keyword, value) == -1) {
            goto fail;
        }
        continue;

      kw_found:
        //We said earlier that if it is not an extension, but a common keyword parameter, we will take this step
        //Get the corresponding symbol, but it is not NULL, indicating that it has been passed through the location parameter
        if (GETLOCAL(j) != NULL) {
            //Then a TypeError will be reported here, indicating that a parameter has received multiple values
            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() got multiple values for argument '%S'",
                          co->co_name, keyword);
            //For example: def foo(a, b, c=1, d=2)
            //If foo(1, 2, c=3) is passed in this way, there must be no problem
            /*
            Because the location parameters will be copied to f in the beginning_ In localsplus, so f_localsplus is [a, b, NULL, NULL]
            Then, when setting keyword parameters, the corresponding index of j is 2, then GETLOCAL(j) is NULL, so no error will be reported
            */
            //But if this is passed, foo(1, 2, 3, c=3)
            //So sorry, at this time f_localsplus is [a, b, c, NULL],GETLOCAL(j) is c, not NULL
            //This indicates that the position of c has been passed, so the keyword parameters cannot be passed
            //That's the same sentence f_localsplus stores symbols. Each symbol will correspond to the corresponding value, and these orders are consistent            
            goto fail;
        }
        Py_INCREF(value);
        SETLOCAL(j, value);
    }

    /* Here, check whether the position parameters are passed more */
    if ((argcount > co->co_argcount) && !(co->co_flags & CO_VARARGS)) {
        too_many_positional(tstate, co, argcount, defcount, fastlocals);
        goto fail;
    }

    //......
    return retval;
}

When Python processes parameters, the mechanism is still very complex. We know that when Python defines a function, through / can make the parameters in front of / must be passed through positional parameters, and through * can make the parameters behind * must be passed through positional parameters, which is not considered in our analysis.

In fact, the transmission mechanism of extended keyword parameters is closely related to the transmission mechanism of ordinary keyword parameters. We have seen the transmission mechanism of keyword parameters by analyzing the default value mechanism of function parameters before. Here we see it again. For keyword parameters, whether extended or not, symbols and values will be placed in two arrays in corresponding order. Then Python will traverse the array storing symbols in the order of index, and each symbol will be linked to the symbol table_ The symbols in varnames are compared one by one, and it is found that the symbol of the keyword parameter we passed cannot be found in the symbol table, so it indicates that this is an extended keyword parameter. Then, as we can see in the source code, if the function defines * * kwargs, then kwdict is not empty and the extended keyword parameters will be set directly. Otherwise, an error will be reported and an unexpected keyword parameter will be received.

And the Python virtual machine does put the PyDictObject object object (kwargs) in F_ In localsplus, this f_localsplus contains all the parameters, no matter what they are. But kwargs must be at the end. As for * args, there is no order in theory. You can define it this way: def foo(a, *args, b). There is no problem with this definition, but at this time, B must be passed through keyword parameters, because if it is not through keyword parameters, no matter how many location parameters, it will stop at * args. As mentioned earlier, assuming that only the three parameters of name, age and gender are required, and the gender must be specified through keyword parameters, it can be designed as follows: def foo(name, age, *, gender). We see that args is omitted and only one * is reserved. This is because we can't use args. We just ensure that the following gender must be passed through keyword, so only one * is required.

In addition, in Python 3 8, note that only Python 3 It is only supported from 8. You can force the use of location parameters. The syntax is through /.

Of course, you can access the passed extension location parameters and extension keyword parameters through PyTupleObject corresponding to args and PyDictObject corresponding to kwargs.

In addition, when analyzing parameters, we always intercept some fragments without overall analysis from top to bottom, so we can look at the source code again. Of course, the core is Python's mechanism for processing function parameters. The overall process is as follows (excluding / and *):

  • Get all the numbers passed through the location parameters, and then loop through them to copy them from the runtime stack to F_ In the location specified by localsplus;
  • Judge whether there is * args. If so, apply for tuples of the specified capacity, and then copy them to F_ In localsplus.
  • Get keyword parameters. The parameter name and value are stored separately, but they correspond to each other one by one, and then cycle through them. It also exists F_ In localsplus; At the same time, there is a for loop inside, which will judge whether the parameter is passed by the location parameter at the same time;
  • Judge whether there is * * kwargs. If there is no * * kwargs but a keyword parameter that is not in the symbol table is specified, an error will be reported; Otherwise, combine the keyword parameters that are not in the symbol table with the corresponding values in a dictionary;
  • Then it will detect whether each symbol in all symbol tables corresponds to a value that is not NULL. If any is NULL, the default value is used.

Therefore, the general process of Python in processing parameters is as above, and the specific details are well understood. It just needs to deal with all kinds of situations, which makes it look a little headache. Of course, the logic of generator and asynchronous generator in Python is also in this function, which will be analyzed in our subsequent series.

Summary

This time, we analyzed the scene of function call and how to deal with different forms of parameters. The key point is to have an overall understanding. In the next article, we will analyze closures in the future.

Tags: Python

Posted by EvanAgee on Fri, 20 May 2022 10:49:33 +0300