Python processes and threads


In order to run multiple tasks at the same time, python introduces the concept of multithreading. In Python, you can start multithreading mode in a convenient and quick way. Multithreading is often used in programs that conform to the concurrency mechanism, such as network programs. In order to further subdivide work tasks, multiple threads can be used in a process.

1. The concept of multitasking

Before understanding the process, we need to know the concept of multitasking. Multitasking, as the name suggests, means that the operating system can perform multiple tasks. For example, operating systems such as Windows or Linux or Mac can watch movies, chat and view web pages at the same time. At this time, the operating system is performing multitasking.

1.1 execution mode of multitasking

1.1.1 concurrency

Alternate tasks over a period of time. For example, for single core CPU processing multitasking, the operating system makes each software execute alternately in turn. For example, if software 1 executes for 0.01 seconds, switch to software 2, software 2 executes for 0.01 seconds, and then switch to software 3, execute for 0.01 seconds... This is repeated. On the surface, each software is executed alternately, but because the execution speed of the CPU is too fast, we feel like these software are executed at the same time. Here, we need to pay attention to that the single core CPU executes multiple tasks concurrently.

1.1.2 parallel

For multi-core cpu processing multi tasks, the operating system will arrange an execution software for each core of the cpu. Multiple cores are the real software executed together. It should be noted here that multi-core CPUs execute multiple tasks in parallel, and there are always multiple software executing together.

1.2 summary

  • Simple understanding: if the number of tasks is greater than the number of CPU cores, it is concurrent, and if the number of tasks is less than the number of CPU cores, it is parallel
  • Using multitasking can make full use of CPU resources, improve the execution efficiency of the program, and make your program have the ability to handle multiple tasks
  • There are two ways of multitasking execution: concurrency and parallelism. Here, parallelism is the real meaning of multiple tasks executing together

2. What is a process

Each task mentioned above is a process. We can open the task manager of Windows and check the process being executed by the operating system, as shown in the figure. The processes shown in the figure include not only applications (such as WeGame, LOL, Google browser, etc.), but also system processes.

A process is an entity of a running program in a computer. The process is different from the program. The program itself is only the description of instructions, data and their organizational form. The process is the real running example of the program (instructions and data). For example, when QQ is not turned on, QQ is just a program. After opening QQ, the operating system starts a process for QQ. Open another QQ music, then start a process. As shown in the figure:

3. Common ways to create processes

There are several modules in Python that can create processes, and OS is commonly used Fork() function, multiprocessing package and Pool process Pool. Due to OS The fork() function is only applicable to Unix/Linux/Mac systems and is not available in Windows operating systems. Therefore, I will focus on the two cross platform modules of multiprocessing package and Pool process Pool.

3.1 creating a process using multiprocessing

multiprocessing provides a Process class to represent a Process object. The syntax is as follows:

Process([group [,target [,name [, args [, kwargl]]]])

    
  • 1

The parameters of Process class are described as follows:

  • Group: indicates the process group. At present, it can only be set to None. Generally, it does not need to be set.
  • Target: refers to the callable object executed when the current process starts. That is, the target task executed by the process, usually the function name or method name
  • name: alias of the current process instance. If not set, it defaults to Process-1,Process-2,... Process-N
  • args: represents the parameter tuple passed to the target function.
  • kwargs: represents the parameter dictionary passed to the target function.

Example 1: instantiate the Process class and execute the sub Process. The code is as follows

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 12:55
# @Author: I am wayward Amo
# @FileName: 01_ Use of multiple processes py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680
import multiprocessing  # 1. Import process package
import time

#Dance task
def dance():
for i in range(3):
print("dancing...)
time.sleep(0.2)

#Singing task
def sing():
for i in range(3):
print("singing...)
time.sleep(0.2)

# 2. Create child process
#(the process manually created by yourself is called a child process, which has been created in _init _.py
#Import Process class)
if name == "main":
#When you write code, you'd better add judgment whether it is the main module
#(simple understanding: the py file you want to execute is called the main module) reason:
# 1. Prevent others from executing relevant code when importing
# 2. When creating a process, the data in the main process will be copied in the Windows system
#All code, so it will cause recursion to create sub processes, form dead recursion and report errors.
dance_process = multiprocessing.Process(target=dance)
sing_process = multiprocessing.Process(target=sing)

<span class="token comment"># 3. Start the process and perform the corresponding tasks</span>
dance_process<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>
sing_process<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>

#Note: the process execution is out of order. The specific execution first is determined by the operating system scheduling

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41

In the above code, instantiate the Process class first, and then use dance_ Process. start()/sing_ Process. The start () method starts the child Process and starts executing the dance()/sing() function. In addition to start(), the common methods of Process instances are as follows:

  • is_ alive0): judge whether the process instance is still executing.
  • join(timecout]): whether to wait for the execution of the process instance to end, or how many seconds to wait
  • start(): start a process instance (create a child process).
  • run(): if the target parameter is not given, when the start() method is called on this object, the run() method in the object will be executed.
  • terminate(): terminate the task immediately regardless of whether it is completed or not.

The Process class also has the following common attributes:

  • name: the alias of the current Process instance. The default is Process-n, and N is an integer increasing from 1.
  • PID: the PID value of the current process instance.

The following is a simple example to demonstrate the use of the methods and attributes of the Process class. Create two child processes, use the os module to output the ID of the parent Process and child Process, and call the name and pid attributes of the Process class. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 13:33
# @Author: I am wayward Amo
# @FileName: 02_ Get the process number py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680
import multiprocessing  # 1. Import process package
import time
import os

#Dance task
def dance():
#Gets the number of the current process (child process)
dance_process_id = os.getpid()
#Get the current process object and see which process object executes the current code: multiprocessing current_ process()
print("dance_process_id", dance_process_id, multiprocessing.current_process())
#Gets the parent process number of the current process
dance_process_parent_id = os.getppid()
Print (the parent process number of F "dance_process is: {dance_process_parent_id}")
for i in range(3):
print("dancing...)
time.sleep(0.2)
#Expand: force to kill the specified process according to the process number
# os.kill(dance_process_id, 9)

#Singing task
def sing():
#Gets the number of the current process (child process)
sing_process_id = os.getpid()
#Get the current process object and see which process object executes the current code: multiprocessing current_ process()
print("sing_process_id", sing_process_id, multiprocessing.current_process())
#Gets the parent process number of the current process
sing_process_parent_id = os.getppid()
Print (the parent process number of F "sing_process is: {sing_process_parent_id}")
for i in range(3):
print("singing...)
time.sleep(0.2)

if name == 'main':
#Gets the number of the current process (main process)
main_process_id = os.getpid()
#Get the current process object and see which process object executes the current code: multiprocessing current_ process()
print("main_process_id", main_process_id, multiprocessing.current_process())

<span class="token comment"># 2. Create sub process object</span>
dance_process <span class="token operator">=</span> multiprocessing<span class="token punctuation">.</span>Process<span class="token punctuation">(</span>target<span class="token operator">=</span>dance<span class="token punctuation">,</span> name<span class="token operator">=</span><span class="token string">"dance"</span><span class="token punctuation">)</span>
sing_process <span class="token operator">=</span> multiprocessing<span class="token punctuation">.</span>Process<span class="token punctuation">(</span>target<span class="token operator">=</span>sing<span class="token punctuation">)</span>

<span class="token comment"># 3. Start the process and perform the corresponding tasks</span>
dance_process<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>
sing_process<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>

<span class="token comment"># 4. The console outputs the name of the subprocess with number</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"dance_process_name:"</span><span class="token punctuation">,</span> dance_process<span class="token punctuation">.</span>name<span class="token punctuation">,</span> <span class="token string">"dance_process.pid:"</span><span class="token punctuation">,</span> dance_process<span class="token punctuation">.</span>pid<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"sing_process_name:"</span><span class="token punctuation">,</span> sing_process<span class="token punctuation">.</span>name<span class="token punctuation">,</span> <span class="token string">"sing_process.pid:"</span><span class="token punctuation">,</span> sing_process<span class="token punctuation">.</span>pid<span class="token punctuation">)</span>

<span class="token comment"># 5. Judge whether the child process is still executing. If it is executing, return true</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>dance_process<span class="token punctuation">.</span>is_alive<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>sing_process<span class="token punctuation">.</span>is_alive<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

dance_process<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>  <span class="token comment"># The main process waits for the child process dance_ End of process</span>
sing_process<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>  <span class="token comment"># The main process waits for the child process sing_ End of process</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"--------End of main process---------"</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66

The running results of the program are as follows:

The following is a simple example of a process executing a task with parameters

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 13:39
# @Author: I am wayward Amo
# @FileName: 03_ Processes perform tasks with parameters py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import multiprocessing

#Task to display information
def show_info(name, age):
print(name, age)

#Create child process
#When parameters are passed in tuples, the order of elements in tuples should be consistent with the order of function parameters
# sub_process = multiprocessing.Process(target=show_info, args=("Amo", 18))
#Start the process and execute the corresponding task
# sub_process.start()

#Pass parameters in dictionary mode. The key in the dictionary should be consistent with the parameter name in the function, and there is no order requirement
sub_process = multiprocessing.Process(target=show_info, kwargs={ "age": 18, "name": "Amo"})

#Start process
sub_process.start()

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26

3.2 creating a Process using the Process subclass

For some simple small tasks, Process(target=xxx) is usually used to realize multi Process. However, if you want to Process complex tasks, you usually define a class to inherit the Process class. Each time you instantiate this class, it is equivalent to instantiating a Process object. Next, learn how to create multiple processes by using the Process subclass through an example.
Use the Process subclass method to create two child processes, and output the PID of the parent and child processes, as well as the status and running time of each child Process. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 14:35
# @Author: I am wayward Amo
# @FileName: 04_ Create a Process using the Process subclass py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

# import multiprocessing
from multiprocessing import Process
import time
import os

#Definition class inherits Process class
class SubProcess(Process):
#Because the Process class itself has__ init__ Initialize the method. This subclass is equivalent to overriding the method of the parent class
def init(self, interval, name=""):
# Process.init(self) # calls the initialization method of the process parent class
super().init() # overridden with super
self.interval = interval # receiving parameter interval
if name: # judge whether the passed parameter name exists
self. If the name # parameter is passed as the name attribute, the default is to create a child process; otherwise, the name attribute is used

<span class="token comment"># Overriding the run() method of the Process class</span>
<span class="token keyword">def</span> <span class="token function">run</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Subprocess(%s)Start execution, parent process is(%s) "</span> <span class="token operator">%</span> <span class="token punctuation">(</span>os<span class="token punctuation">.</span>getpid<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> os<span class="token punctuation">.</span>getppid<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

    t_start <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
    time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span>self<span class="token punctuation">.</span>interval<span class="token punctuation">)</span>
    t_stop <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Subprocess(%s )End of execution, time consuming%0 .2f second"</span> <span class="token operator">%</span> <span class="token punctuation">(</span>os<span class="token punctuation">.</span>getpid<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> t_stop <span class="token operator">-</span> t_start<span class="token punctuation">)</span><span class="token punctuation">)</span>

if name == "main":
print("the parent process starts to execute one by one")
print("parent process PID:% s"% os.getpid())
#Output the ID of the current program
p1 = SubProcess(interval=1, name="Amo")
p2 = SubProcess(interval=2)
#Executing the start() method on a Process class that does not contain the target attribute will run the run() method in this class
#So P1.0 will be executed here run()
p1.start()
#Start process p1
p2.start()
#Start process p2
#Output the execution status of p1 and p2 processes. If it is True, return True; Otherwise, False is returned
print("p1.is_ alive=%s" % p1.is_alive())
print("p2.is_ alive=%s" % p2.is_alive())
#Output alias and PID of p1 and p2 processes
print("p1.name=%s" % p1.name)
print("p1.pid=%s" % p1.pid)
print("p2.name=%s" % p2.name)
print("p2. pid=%s" % p2.pid)
print("wait for subprocesses one by one")
p1.join()
#Wait for the p1 process to end
p2.join()
#Wait for the p2 process to end
print("end of parent process execution")

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59

3.3 creating processes using process Pool

In the above, we created two processes using the Process class. If you want to create dozens or hundreds of processes, you need to instantiate more Process classes. Is there a better way to create a Process to solve this kind of problem? The answer is to use the Pool class provided by the multiprocessing module, that is, the Pool Process Pool. In order to better understand the Process Pool, you can compare the Process Pool to a Pool, as shown in the figure. We need to complete the task of filling 10 basins with water. In this Pool, up to 3 basins can be placed to receive water, that is, 3 tasks can be performed at the same time, that is, 3 processes can be started. In order to complete the task faster, now turn on three faucets and start to discharge water. When a basin is full, that is, the Process completes one task, we will pour the water in this basin into the bucket, and then continue to receive water, that is, to perform the next task. If three basins are filled with water at the same time, after the ninth basin is filled with water, the system will randomly assign one basin to receive water and the other two basins will be idle. A simple understanding is that: the Process resources are used repeatedly, and there is no need to create processes frequently and consume system resources.

Next, let's take a look at the common methods of the Pool class. Common methods and descriptions are as follows:

  • apply_ async(func[, args[, kwds]): call func() function in non blocking mode (parallel execution, blocking mode must wait for the previous process to exit before executing the next process), args is the parameter list passed to func() function, and kwds is the keyword parameter list passed to func() function.
  • apply(func[, args[, kwdsI): call func() function in blocking mode.
  • close(): close the Pool so that it will no longer accept new tasks.
  • terminate(): terminate the task immediately regardless of whether it is completed or not.
  • join(): the main process is blocked and waits for the child process to exit. It must be used after close or terminate

Apply is mentioned in the above method_ Async () calls the function in non blocking mode, while apply0 calls the function in blocking mode. So what are blocking and non blocking? In the figure, three tasks are performed in blocking mode and non blocking mode respectively. If you use blocking mode, you must wait for the previous process to exit before executing the next process. If you use non blocking mode, you can execute three processes in parallel.

The following is an example of how to create multiple processes using a process pool. Here, simulate the scene of water discharge from the pool, define a process pool, and set the maximum number of processes to 3. Then perform 10 tasks in a non blocking manner to view the tasks performed by each process. The specific codes are as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 14:39
# @Author: I am wayward Amo
# @FileName: 05_ Create a process using a process pool py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

from multiprocessing import Pool # import
import os
import time

def task(name):
print(f "subprocess {os.getpid()} execute task{name}...)
time.sleep(1)

if name == "main":
# 1. The maximum number of processes to create a process pool is 3
pool = Pool(3)
# 2. Execute 10 times from 0
for i in range(10):
#Call the task() function in a non blocking manner
pool.apply_async(task, args=(i,))
print("wait for all child processes to finish...)
pool.close() # closes the process pool. After closing, the pool will no longer accept new task requests
pool.join() # wait for the child process to finish
print("all child processes end.")

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28

The running results are shown in the figure. It can be seen from the figure that the sub process with PID 2372 executed four tasks, while the other two sub processes executed three tasks respectively.

4. Realize inter process communication through queue

4.1 global variables are not shared between processes

We have learned how to create multiple processes, so in multiple processes, what is the relationship between each process? In fact, each process has its own address space, memory, data stack and other auxiliary data to record its running state. The following is an example to verify whether information can be shared directly between processes.
Define a global variable g_num to create two sub process pairs g_num performs different operations and outputs the results after the operation. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 17:57
# @Author: I am wayward Amo
# @FileName: 06_ Global variables are not shared between processes py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import multiprocessing

#Define a global variable
g_num = 100

#Addition task
def plus():
print("----------- child process 1 start ---------------")
global g_num # declares that the variables defined inside the function are global variables
g_num += 50
print(f"g_num is {g_num}")
print("----------- end of subprocess 1 ---------------")

#Subtraction task
def minus():
print("---------- child process 2 start ---------------")
global g_num # declares that the variables defined inside the function are global variables
g_num -= 50
print(f"g_num is {g_num}")
print("----------- end of subprocess 2 ---------------")

if name == 'main':
print("----------- main process start ---------------")
print(f"g_num is {g_num}")
# 1. Create process object
plus_process = multiprocessing.Process(target=plus)
minus_process = multiprocessing.Process(target=minus)
# 2. Start the process and execute the corresponding task
plus_process.start()
minus_process.start()

plus_process<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>
minus_process<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>  <span class="token comment"># Wait for the child process to end</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"----------End of main process-------------"</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44

The operation results are shown in the figure:

In the above code, two sub processes are created respectively. In one sub process, G_ Another sub process plus 50, num_ Num minus 50. However, it can be seen from the operation results that G_ The initial value of num in the parent process and two child processes is 100. That is, the global variable G_ The result of num in one process is not transferred to the next process, that is, there is no shared information between processes. The schematic diagram between processes is shown in the figure.

Creating a child process will copy the resources of the main process, that is, the child process is a copy of the main process, like a pair of twins. The reason why global variables are not shared between processes is that the global variables in the same process are not operated, but the names of global variables in different processes are the same.

4.2 queue introduction

Through the above, it is understood that communication between processes cannot be carried out. So how to realize the communication between processes? Python multiprocessing wraps the underlying mechanism and provides multiple ways to exchange data, such as queues and Pipes. Here we will explain how to realize the communication between processes through queues.
Queue is to imitate the queue in reality. For example, students line up to buy food in the canteen. The new students lined up at the end of the line. The front student walked away after buying dinner, and the students behind followed. It can be seen that the queue has two characteristics: the new students are at the end of the queue. The first student leaves the team after finishing, and the last one keeps up.
According to the above characteristics, the queue structure can be summarized as shown in the figure

4.3 use of multi process queue

Communication is sometimes needed between processes, and the operating system provides many mechanisms to realize the communication between processes. The queue of mulip processing module can be used to realize data transfer between multiple processes. Queue itself is a message queuing program. Let's introduce the use of queue.
When initializing the Queue() object (for example: q=Queue(num)), if the maximum number of messages that can be received is not specified in parentheses or the number is negative, it means that there is no upper limit on the number of acceptable messages (until the end of memory). The common methods of Queue are as follows:

  • Queue.qsize(): returns the number of messages contained in the current queue.
  • Queue.emptyO: returns True if the queue is empty; Otherwise, it returns False.
  • Queue.full(): returns True if the queue is full; Otherwise, it returns False.
  • Queue.get([block[, timeout]): get a message in the queue and remove it from the queue. The default value of block is True.
    • If block uses the default value and timeout is not set, the message queue is empty, and the program will be blocked (stopped in the read state) until the message is read from the message queue. If timeout is set, it will wait for timeout seconds. If no message has been read, it will throw a "Queue. Empty" exception.
    • If the block value is False and the message queue is empty, a "Queue.Empty" exception will be thrown immediately.
  • Queue.get_nowait(): equivalent to queue get(False).
  • Queue. put(item,[block[, timeout]): writes the item message to the queue. The default value of block is True.
    • If the default value of block is used and timeout (in seconds) is not set, if there is no space for message queuing to write, the program will be blocked (stopped in the write state) until space is made available from the message queuing. If timeout is set, it will wait for timeout seconds. If there is no space, it will throw an "Queue.Full" exception.
    • If the block value is False and the message queue has no space to write, a "Queue.Full" exception will be thrown immediately.
    • Queue.put_nowait(item): equivalent to queue put(item, False).

Next, learn how to use processing. Com through an example Queue. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 19:08
# @Author: I am wayward Amo
# @FileName: 07_ Use of queues py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

from multiprocessing import Queue

if name == 'main':
# 1. Initialize a Queue object, which can accept up to three put messages
q = Queue(3)
q.put("message 1")
q.put("message 2")
print(q.full()) # returns False
q.put("message 3")
print(q.full()) # returns True

<span class="token comment"># At this time, the message queue is full, and the following try will throw exceptions</span>
<span class="token comment"># The first try will throw an exception after waiting for 2 seconds, and the second try will throw an exception immediately</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
    q<span class="token punctuation">.</span>put<span class="token punctuation">(</span><span class="token string">"Message 4"</span><span class="token punctuation">,</span> <span class="token boolean">True</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span>
<span class="token keyword">except</span><span class="token punctuation">:</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>
    <span class="token comment"># print("message queue is full, number of existing messages:% s"% q.qsize())</span>
    <span class="token keyword">pass</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
    q<span class="token punctuation">.</span>put_nowait<span class="token punctuation">(</span><span class="token string">"Message 4"</span><span class="token punctuation">)</span>
<span class="token keyword">except</span><span class="token punctuation">:</span>
    <span class="token comment"># print(f "message queue is full, number of existing messages: {q.qsize()}")</span>
    <span class="token keyword">pass</span>
<span class="token comment"># When reading a message, first judge whether the message queue is empty, and then read it</span>
<span class="token keyword">if</span> <span class="token operator">not</span> q<span class="token punctuation">.</span>empty<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"--------Read message from queue-------------"</span><span class="token punctuation">)</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>q<span class="token punctuation">.</span>qsize<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">print</span><span class="token punctuation">(</span>q<span class="token punctuation">.</span>get_nowait<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># Judge whether the message queue is full before writing</span>
<span class="token keyword">if</span> <span class="token operator">not</span> q<span class="token punctuation">.</span>full<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
    q<span class="token punctuation">.</span>put_nowait<span class="token punctuation">(</span><span class="token string">"Message 4"</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40

4.4 communication between processes using queues

We know how to use multiprocessing Process can create multiple processes, using multiprocessing Queue can realize the operation of queue. Next, through an example, the communication between processes is realized by combining Proess and queue. Create two sub processes. One sub process is responsible for writing data to the queue and the other sub process is responsible for reading data from the queue. To ensure that the data can be read from the queue correctly, set the waiting time of the process reading data to 2 seconds. If the data cannot be read after 2 seconds, an exception is thrown. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 19:38
# @Author: I am wayward Amo
# @FileName: 08_ Use queues to communicate between processes py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

from multiprocessing import Process
from multiprocessing import Queue
import time

#Write data to queue
def write_task(queue):
if not queue.full():
for i in range(5):
Message = "message" + str(i)
queue.put(message)
print(f "write: {message}")

#Read data from queue
def read_task(queue):
time.sleep(1) # sleep for 1 second
while not queue.empty():
#Wait for 2 seconds. If no message has been read, an "Queue.Empty" exception will be thrown
print(f "read: {queue.get(True, 2)}")

if name == 'main':
print("------------ parent process start -------------------")
#The parent process creates a Queue and passes it to each child process
q = Queue()
#Instantiate the child process that writes to the queue and deliver the queue
write_process = Process(target=write_task, args=(q,))
#Instantiate the child process of the read queue and deliver the queue
read_process = Process(target=read_task, args=(q,))
#Start the process and execute the corresponding task
write_process.start()
read_process.start()
#Wait for the child process to end
write_process.join()
read_process.join()
print("------------ end of parent process ------------")

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44

The operation results are shown in the figure below:

5. What is thread

If you need to handle multiple tasks at the same time, one is that you can use multiple processes in one application, and each process is responsible for completing part of the work; Another way to subdivide work into multiple tasks is to use multiple threads within a process. So, what is a thread? Thread is the smallest unit that the operating system can schedule operations. It is included in the process and is the actual operation unit in the process. A thread refers to a single sequential control flow in a process. Multiple threads can be concurrent in a process, and each thread executes different tasks in parallel. For example, for a video player, one thread is used to display video and another thread is used to play audio. Only when two threads work at the same time can we watch the video with picture and sound synchronization normally.
Take a life example to better understand the relationship between process and thread. A process is like a house. It is a container with corresponding attributes, such as floor area, bedroom, kitchen and bathroom. The house itself did not take the initiative to do anything. Thread is the resident of the house. He can use every room in the house, cook, take a bath and so on.

6. Create thread

Since threads are the execution unit directly supported by the operating system, high-level languages (such as python, Java, etc.) usually have built-in multithreading support. Python's standard library provides two modules:_ thread and threading_ thread is a low-level module and threading is a high-level module, right_ thread is encapsulated. In most cases, we only need to use the advanced module threading.

6.1 creating threads using threading module

The threading module provides a Thread class to represent a Thread object. The syntax is as follows:
Thread([group [, target [ , name [, args [,kwargs]]] ]]). The parameters of thread class are described as follows:

  • group: the value is None, which is reserved for later versions.
  • target: indicates a callable object. When the thread starts, the run() method will call this object. The default value is None, indicating that nothing will be called.
  • Name: indicates the name of the current Thread. A unique name in "Thread-N" format is created by default.
  • args: represents the parameter tuple passed to the target() function.
  • kwargs: represents the parameter dictionary passed to the target() function.

Through comparison, it is found that the method of Thread class is basically the same as that of the Porcess class explained earlier, which will not be repeated here. Next, let's learn how to use the threading module to create threads through an example. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 20:12
# @Author: I am wayward Amo
# @FileName: 09_ The use of multithreading py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading # 1. Import thread module
import time

#Dance task
def dance():
#Get current thread
print("dance_thread:", threading.current_thread())
for i in range(4):
print("dancing...)
time.sleep(0.2)

#Singing task
def sing():
#Get current thread
print("sing_thread:", threading.current_thread())
for i in range(4):
print("singing...)
time.sleep(0.2)

if name == 'main':
#Get current thread
print("main_thread:", threading.current_thread())
# 1. Create child process
dance_thread = threading.Thread(target=dance, name="dance_thread")
sing_thread = threading.Thread(target=sing, name="sing_thread")
# 2. Start the child thread to execute the corresponding task
dance_thread.start()
sing_thread.start()

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38

The operation results are as follows:

The following is a simple example of a thread executing a task with parameters:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 20:40
# @Author: I am wayward Amo
# @FileName: 11_ A thread executes a task with parameters py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading
import time

def show_info(name, age):
print(f "my name is {name} and my age is {age}")

if name == 'main':
#Create thread object
#When passing parameters in the form of tuples, it is necessary to ensure that the order of elements in tuples is consistent with the order of function parameters
# sub_thread = threading.Thread(target=show_info, args=("Amo", 18))
## start the thread to execute the corresponding task
# sub_thread.start()

<span class="token comment"># Pass parameters in the form of a dictionary to ensure that the key in the dictionary is consistent with the name of the function parameter without sequence requirements</span>
sub_thread <span class="token operator">=</span> threading<span class="token punctuation">.</span>Thread<span class="token punctuation">(</span>target<span class="token operator">=</span>show_info<span class="token punctuation">,</span> kwargs<span class="token operator">=</span><span class="token punctuation">{<!-- --></span><span class="token string">"age"</span><span class="token punctuation">:</span> <span class="token number">18</span><span class="token punctuation">,</span> <span class="token string">"name"</span><span class="token punctuation">:</span> <span class="token string">"Amo"</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment"># Start the thread to execute the corresponding task</span>
sub_thread<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26

The following is a simple example to demonstrate that tasks executed by threads are out of order:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 20:53
# @Author: I am wayward Amo
# @FileName: 12_ The execution of tasks between threads is out of order py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading
import time

def task():
time.sleep(1)
#Get current thread
print(threading.current_thread())

if name == 'main':
#Loop creates a large number of threads to test whether the execution between threads is out of order
for i in range(20):
#Create one child thread per loop
sub_thread = threading.Thread(target=task)
#Start the child thread to execute the corresponding task
sub_thread.start()

<span class="token comment"># Note: the execution between threads is out of order. The specific thread task to execute first is determined by cpu scheduling</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27

The operation results are as follows:

The following is a simple example to demonstrate that the main thread will wait for the execution of all child threads to end:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/23 12:55
# @Author: I am wayward Amo
# @FileName: 14_ The main thread will wait for all child threads to finish executing py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading
import time

def task():
while True:
print("task execution...)
time.sleep(0.3)

if name == 'main':
#Create child thread
#daemon=True means that the created sub thread guards the main thread. The main thread exits and the sub thread is destroyed directly
# sub_thread = threading.Thread(target=task, daemon=True)
sub_thread = threading.Thread(target=task)
#Set the child thread as the main thread of guard
sub_thread.setDaemon(True)
sub_thread.start()

<span class="token comment"># The main thread delays execution for 1 second</span>
time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>

<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"over"</span><span class="token punctuation">)</span>
<span class="token comment"># exit()</span>

#Conclusion: the main thread will wait for the execution of the child thread to end
#Solution: set the sub thread as the main thread of guardian

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34

Here's the demonstration to add a knowledge point. In the process of realizing multitasking, the main process will also wait for the execution of all sub processes to end. The following is a simple example:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/22 20:33
# @Author: I am wayward Amo
# @FileName: 09_ The main process will wait for all child processes to finish executing py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import multiprocessing
import time

def task():
while True:
Task in progress ("print...)
time.sleep(0.2)

if name == 'main':
# 1. Create child process
#Set the subprocess as the guardian of the main process. The main process exits and the subprocess is destroyed directly
# sub_process = multiprocessing.Process(target=task, daemon=True)
sub_process = multiprocessing.Process(target=task)
# sub_process.daemon = True
# 2. Start the sub process to execute the corresponding task
sub_process.start()

<span class="token comment"># The main process is delayed for 2 seconds</span>
time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">)</span>
<span class="token comment"># Before exiting the main process, let the child process destroy</span>
sub_process<span class="token punctuation">.</span>terminate<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"-----End of main process-----"</span><span class="token punctuation">)</span>
<span class="token comment"># Conclusion: the main process will wait for the execution of the sub process, and then exit the program</span>
<span class="token comment"># Solution: the main process exits and the sub process is destroyed</span>
<span class="token comment"># 1. Set the subprocess as the guardian process. The main process exits and the subprocess is destroyed. The subprocess will depend on the main process</span>
<span class="token comment"># 2. Let the child process destroy before the main process exits</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36

Program running results:

6.2 creating threads using Thread subclass

Thread thread class and Process process class are used in a very similar way. You can also create threads by defining a subclass to inherit thread thread class. Let's learn how to create a thread using the thread subclass through an example. Create a subclass SubThread and inherit threading Thread thread class and define a run() method. Instantiate the SubThread class to create two threads, and call the start() method to start the thread. The program will automatically call the run() method. The code is as follows:

import threading
import time

class SubThread(threading.Thread):
def run(self):
for i in range(3):
time.sleep(1)
msg = "sub thread" + self Name + "execution, i =" + str(i) # name attribute saves the name of the current thread
print(msg)

if name == 'main':
print("------ main process start -----")
t1 = SubThread()
t2 = SubThread()
t1.start()
t2.start()
t1.join()
t2.join()
print("------ end of main process ------")

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21

7. Inter thread communication

7.1 sharing global variables between threads

In the previous study, we have known that information cannot be shared directly between processes. Can information be shared between threads? Let's verify it with an example. Define a global variable g_ mum and create two sub thread pairs g_ mum performs different operations and outputs the output after the operation Results. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/23 13:29
# @Author: I am wayward Amo
# @FileName: 16_ Global variables are shared between threads py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading
import time

#Define a global variable
g_num = 100

#Addition task
def plus():
print("start of sub thread 1 ---------------")
global g_num # declares that the variables defined inside the function are global variables
g_num += 50
print(f"g_num is {g_num}")
print("end of sub thread 1 ---------------")

#Subtraction task
def minus():
time.sleep(1)
print("---------- start of sub thread 2 ---------------")
global g_num # declares that the variables defined inside the function are global variables
g_num -= 50
print(f"g_num is {g_num}")
print("---------- end of sub thread 2 ---------------")

if name == 'main':
print("---------- main thread start ---------------")
print(f"g_num is {g_num}")
# 1. Create thread object
plus_thread = threading.Thread(target=plus)
minus_thread = threading.Thread(target=minus)
# 2. Start the thread to execute the corresponding task
plus_thread.start()
minus_thread.start()

plus_thread<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>
minus_thread<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>  <span class="token comment"># Wait for the child thread to end</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"----------End of main thread-------------"</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47

In the above code, a global variable G is defined_ Num, with a value of 100, and then create 2 threads. A thread will g_ When num is increased by 50, a thread will g_num reduced by 50. If G_ The final result of num is 100, which means that data can be shared between threads. The operation results are shown in the figure:

From the above example, it can be concluded that all threads in a process share global variables, which can complete the data sharing among multiple threads without using other methods. However, in this way, it is easy to make errors when sharing global variable data between threads. Here is a simple example:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/23 13:37
# @Author: I am wayward Amo
# @FileName: 17_ Error in sharing global variable data between threads py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading
import time

g_num = 0

#Tasks executed in 1 million cycles
def task1():
for i in range(1000000):
global g_num # indicates that this variable is a global variable
g_num += 1 # add 1 to the global variable every cycle

<span class="token comment"># This code indicates that the data calculation is completed</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"task1:"</span><span class="token punctuation">,</span> g_num<span class="token punctuation">)</span>

def task2():
for i in range(1000000):
global g_num # indicates that this variable is a global variable
g_num += 1 # add 1 to the global variable every cycle

<span class="token comment"># This code indicates that the data calculation is completed</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"task2:"</span><span class="token punctuation">,</span> g_num<span class="token punctuation">)</span>

if name == 'main':
#Create two child threads
sub_thread1 = threading.Thread(target=task1)
sub_thread2 = threading.Thread(target=task2)
#Start the thread to execute the task
sub_thread1.start()
# time.sleep(1)
sub_thread2.start()

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40

Program running results:

According to the normal result, the final result should be 2000000, but the final result found by executing the program is less than 2000000. The reasons are as follows:

Solution 1:

# Thread waiting, let the first thread execute first, and then let the second thread execute again to ensure that there is no problem with the data
sub_thread1.join() # The main thread waits for the execution of the first sub thread, and then the code continues to execute
sub_thread2.start()

 
  • 1
  • 2
  • 3

Solution 2: use mutex

7.2 what is a mutex

Because threads can modify global variables at will, it may cause chaotic operations on global variables among multiple threads. Still take the house as an example. When there is only one resident in the house (single thread), he can use any room at any time, such as kitchen, bedroom and bathroom. However, when the house has multiple occupants (multithreading), he can't use some rooms at any time, such as the bathroom, otherwise it will cause chaos.
How to solve this problem? A simple way to prevent others from entering is to put a lock on the door. Those who arrive first lock the door, and those who arrive later queue at the door and wait until the lock is opened. As shown in the figure.

This is called "Mutual exclusion" (Mutex), which prevents multiple threads from reading and writing a memory area at the same time. Mutex introduces a state for resources: locked and unlocked. When a thread wants to change the shared data, it locks it first. At this time, the state of the resource is "locked", and other threads cannot change it; Until the thread releases the resource and changes the state of the resource to "unlocked", other threads can lock the resource again. Mutex ensures that only one thread can write at a time, which ensures the correctness of data in the case of multithreading. However, in this case, multitasking becomes a single task operation, which reduces the efficiency of program execution. Moreover, the improper use of mutex is easy to cause deadlock.

7.3 using mutex

Using the Lock class in the threading module makes it easy to handle locks. The Lock class has two methods: acquire() to Lock and release() to release the Lock. The example usage is as follows:

mutex = threading.Lock()  # Create lock
mutex.acquire([blocking])  # locking
mutex.release()  # Release lock

 
  • 1
  • 2
  • 3

The syntax is as follows:

  • acquire([blocking]): acquire the lock. If necessary, block it until the lock is released. If the blocking parameter is provided and set to False, False will be returned immediately when the lock cannot be obtained; Returns True if the lock was successfully acquired.
  • release(): release one lock by one. An error occurs when the lock is unlocked, or when this method is called from a different thread from the one that originally called the acquire() method.

Next, learn how to use mutex through an example. Here, multithreading and mutex lock are used to simulate the function of multiple people ordering movie tickets at the same time. It is assumed that there are only 100 movie tickets in a movie theater, and 10 users rush to buy the movie tickets at the same time. Each time one ticket is sold, the number of remaining movie tickets will be displayed. The code is as follows:

 # -*- coding: utf-8 -*-
# @Time    : 2020/2/23 13:52
# @Author: I am wayward Amo
# @FileName: 18_ Mutex py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

import threading
from threading import Lock
import time

n = 100 # defines 100 movie tickets

#Definitional lock
mutex = Lock() # creates a mutex. Lock is essentially a function. You can create a mutex by calling the function

def task():
global n
mutex. Acquire (# lock)
temp = n # assigned to temporary variable
time.sleep(0.1)
n = temp - 1 # quantity minus 1
print("successful ticket purchase,% d movie tickets remaining"% n)
mutex. Release (# release lock)

if name == 'main':
t_l = [] # initialize a list
for i in range(10):
t = threading.Thread(target=task)
t_l.append(t) # stores threads in the list
t.start() # starts the thread to execute the corresponding task
for t in t_l:
t.join() # wait for the child thread to end

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36

In the above code, 10 threads are created and all execute the task() function. In order to solve the problem of resource competition, mutex acquire() function is used to realize resource locking. After the first thread to obtain resources is locked, other threads wait for mutex Release(). So only one thread executes the task function at a time. The operation results are shown in the figure.

7.4 deadlock


Here is a simple example to demonstrate deadlock. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/23 14:38
# @Author: I am wayward Amo
# @FileName: 19_ Deadlock py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

#Deadlock: the situation of waiting for the other party to release the lock is called deadlock
import threading

#Create mutex
lock = threading.Lock()

#Requirement: multiple threads take values in the list according to the subscript at the same time. It is necessary to ensure that only one thread can take values at the same time
def get_value(index):
#Lock
lock.acquire()
my_list = [1, 4, 6]
#Judge whether the subscript is out of bounds
if index >= len(my_list):
print("subscript out of bounds:", index)
#If the value is not successful, you also need to release the mutex. Do not affect the subsequent threads to take the value
#The lock needs to be released in a suitable place to prevent deadlock
# lock.release()
return

<span class="token comment"># Value according to subscript</span>
value <span class="token operator">=</span> my_list<span class="token punctuation">[</span>index<span class="token punctuation">]</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>value<span class="token punctuation">)</span>
<span class="token comment"># Release lock</span>
lock<span class="token punctuation">.</span>release<span class="token punctuation">(</span><span class="token punctuation">)</span>

if name == 'main':
#Create a large number of threads and execute tasks based on subscript values at the same time
for i in range(10):
#Create one child thread per loop
sub_thread = threading.Thread(target=get_value, args=(i,))
#Start the thread to execute the task
sub_thread.start()

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41

The operation results are as follows:

7.5 inter process communication using queues

We know that the Queue queue of mulprocessing module can realize inter process communication. Similarly, between processes, Queue: Queue can also be used to realize inter thread communication. The difference is that we need to use the Queue queue of the Queue module instead of the Queue queue of the multprocessing module, but the use method of the Queue is the same.
Inter process communication using Queue is usually applied to producer consumer mode. The module that generates the data is called the producer, while the module that processes the data is called the consumer. The buffer between producers and consumers is called warehouse. Producers are responsible for transporting goods to warehouses, while consumers are responsible for taking goods out of warehouses, which constitutes the producer consumer model. Let's learn how to use Queue to communicate between processes through an example.
Define a Producer class and a Consumer class. The generator generates five products and writes them into the queue in turn, while the Consumer takes them out of the queue in turn. The code is as follows:

# -*- coding: utf-8 -*-
# @Time    : 2020/2/23 14:41
# @Author: I am wayward Amo
# @FileName: 20_ Use queues to communicate between processes py
# @Software: PyCharm
# @Blog    : https://blog.csdn.net/xw1680

from queue import Queue
import random
import threading
import time

#Producer class
class Producer(threading.Thread):
def init(self, name, queue):
threading.Thread.init(self, name=name)
self.data = queue

<span class="token keyword">def</span> <span class="token function">run</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">5</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"producer%s Will product%d Join queue!"</span> <span class="token operator">%</span> <span class="token punctuation">(</span>self<span class="token punctuation">.</span>getName<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> i<span class="token punctuation">)</span><span class="token punctuation">)</span>
        self<span class="token punctuation">.</span>data<span class="token punctuation">.</span>put<span class="token punctuation">(</span>i<span class="token punctuation">)</span>
        time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span>random<span class="token punctuation">.</span>random<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"producer%s complete!"</span> <span class="token operator">%</span> self<span class="token punctuation">.</span>getName<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

#Consumer category
class Consumer(threading.Thread):
def init(self, name, queue):
threading.Thread.init(self, name=name)
self.data = queue

<span class="token keyword">def</span> <span class="token function">run</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">5</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        val <span class="token operator">=</span> self<span class="token punctuation">.</span>data<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token punctuation">)</span>
        <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"consumer%s Will product%d Remove from queue!"</span> <span class="token operator">%</span> <span class="token punctuation">(</span>self<span class="token punctuation">.</span>getName<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> val<span class="token punctuation">)</span><span class="token punctuation">)</span>
        time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span>random<span class="token punctuation">.</span>random<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"consumer%s complete!"</span> <span class="token operator">%</span> self<span class="token punctuation">.</span>getName<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

if name == 'main':
print("the main thread starts one by one")
queue = Queue() # instantiate queue

producer <span class="token operator">=</span> Producer<span class="token punctuation">(</span><span class="token string">"Producer"</span><span class="token punctuation">,</span> queue<span class="token punctuation">)</span>  <span class="token comment"># Instantiate the thread Producer and pass in the queue as a parameter</span>
consumer <span class="token operator">=</span> Consumer<span class="token punctuation">(</span><span class="token string">"Consumer"</span><span class="token punctuation">,</span> queue<span class="token punctuation">)</span>  <span class="token comment"># # Instantiate the thread Consumer and pass in the queue as a parameter</span>
producer<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>  <span class="token comment"># Start thread producer</span>
consumer<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>  <span class="token comment"># Start thread consumer</span>
producer<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>
consumer<span class="token punctuation">.</span>join<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"End of main thread"</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52

The running results of the program are as follows:

8. Process and thread comparison

8.1 three directions of process and thread comparison

  1. Relationship comparison
  2. Difference comparison
  3. Comparison of advantages and disadvantages

8.2 relationship comparison

  1. Threads are attached to processes. Without processes, there will be no threads.
  2. A process provides one thread by default, and a process can create multiple threads.

8.3 difference and comparison

  1. Global variables are not shared between processes
  2. Threads share global variables, but pay attention to the problem of resource competition. The solution: mutual exclusion or thread synchronization
  3. The resource cost of creating a process is greater than that of creating a thread
  4. Process is the basic unit of operating system resource allocation, and thread is the basic unit of CPU scheduling
  5. Threads cannot execute independently and must exist in the process
  6. Multi process development is more stable than single process multithreading development

8.4 comparison of advantages and disadvantages

  • Process advantages and disadvantages:
    Advantages: multiple cores can be used
    Disadvantages: high resource overhead
  • Advantages and disadvantages of threads:
    Advantages: low resource overhead
    Disadvantages: multi core cannot be used

8.5 summary

  1. Both processes and threads are a way to multitask
  2. Multi process consumes more resources than multi thread, but multi process development is more stable than single process multi thread development. If a process hangs, it will not affect other processes.
  3. Multiple processes can run with multiple cores of cpu, and multiple threads can share global variables.
  4. Threads cannot be executed alone and must be attached to the process

Tags: Python Multithreading multiple processes

Posted by chrisprse on Sat, 07 May 2022 21:03:36 +0300