python foundation-21-process and thread

Process and thread

1, What is multitasking
Multitasking is that the operating system can run multiple tasks at the same time. For example, you are surfing the Internet with your browser, learning on the official website of muke.com and listening to music. This is multitasking. At least three tasks are running at the same time. There are many tasks running quietly in the background at the same time, but they are not displayed on the desktop.
2, What is a process
For the operating system, a task is a Process. For example, opening a browser starts a browser Process, opening a word starts a word Process, and opening two notepads starts two Notepad processes. You can check the Process of opening the operating system on the computer. The specific method is to take the win8 operating system as an example: right click the taskbar on the win8 traditional desktop and select task manager to quickly open the win8 task manager as follows:

After opening, you can view which application processes and background processes are opened on the computer, as well as the computer resources CPU, memory, disk, etc. occupied by each process.
3, What kind of task is a process? When a task is started, the operating system will allocate the system resources it needs, including memory, I/O and CPU
Not enough, there will be a system crash, and such a task can be called a process.
4, How to create a process in python
Module used: multiprocessing method created: multiprocessing Process(…)
5, What is a thread
Thread is the smallest execution unit of the operating system, and the process is composed of at least one thread. How to schedule processes and threads is entirely determined by the operating system. The program itself cannot decide when to execute and how long to execute. Some processes do more than one thing at the same time, such as wechat, which can do voice, text, browsing information and other things at the same time. Simple understanding: in a process, if you want to do multiple things at the same time, you need to run multiple "subtasks" at the same time. We call these "subtasks" in the process threads.
6, What kind of task is a thread
After a process is run, it is a thread. If a process does not run, the thread will run. If a process has multiple threads, it is related to the number of modules that can be called by the cpu alone. This calling module can be established by manually creating threads.
7, How to create threads in python
Module used: threading
Method of creation: threading Thread(…)

Process module multiprocessing

example

Normal operation

import time


def work_a():
    for i in range(10): # Cycle 0-9
        print(i,"a") # Print cycle
        time.sleep(1) # Wait 1 second
def work_b():
    for i in range(10):
        print(i,"b")
        time.sleep(1)

if __name__ == "__main__" :
    start = time.time() # start time
    work_a()
    work_b()
    print("Time consuming:",time.time() - start) # Finish time minus start time

View process

import multiprocessing
import time
import os

def work_a():
    for i in range(10): # Cycle 0-9
        print(i,"a",os.getpid()) # Print cycle #os.getpid() view the pid process of the system
        time.sleep(1) # Wait 1 second
def work_b():
    for i in range(10):
        print(i,"b",os.getpid())
        time.sleep(1)

if __name__ == "__main__" :
    start = time.time() # start time
    work_a()
    work_b()
    print("Time consuming:",time.time() - start) # Finish time minus start time
    print("parent pid is %s" % os.getpid())

As shown in the figure, we can see that both a and b are in the same process

Multi process

import multiprocessing
import time
import os

def work_a():
    for i in range(10): # Cycle 0-9
        print(i,"a",os.getpid()) # Print cycle #os.getpid() view the pid process of the system
        time.sleep(1) # Wait 1 second
def work_b():
    for i in range(10):
        print(i,"b",os.getpid())
        time.sleep(1)

if __name__ == "__main__" :
    start = time.time() # start time
    a_p = multiprocessing.Process(target=work_a)# Create multiple processes
    a_p.start()
    work_b()
    print("Time consuming:",time.time() - start) # Finish time minus start time
    print("parent pid is %s" % os.getpid())

The child process and the main process do not affect each other

import multiprocessing
import time
import os

def work_a():
    for i in range(10): # Cycle 0-9
        print(i,"a",os.getpid()) # Print cycle #os.getpid() view the pid process of the system
        time.sleep(1) # Wait 1 second
def work_b():
    for i in range(10):
        print(i,"b",os.getpid())
        time.sleep(1)

if __name__ == "__main__" :
    start = time.time() # Main process
    a_p = multiprocessing.Process(target=work_a)# Subprocess 1
    a_p.start() #Subprocess 1 execution
    b_p = multiprocessing.Process(target=work_b)  # Subprocess 2
    b_p.start() # Subprocess 2 execution
    print("Time consuming:",time.time() - start) # Main process
    print("parent pid is %s" % os.getpid()) # Main process

Process blocking

import multiprocessing
import time
import os

def work_a():
    for i in range(10): # Cycle 0-9
        print(i,"a",os.getpid()) # Print cycle #os.getpid() view the pid process of the system
        time.sleep(1) # Wait 1 second
def work_b():
    for i in range(10):
        print(i,"b",os.getpid())
        time.sleep(1)

if __name__ == "__main__" :
    start = time.time() # Main process
    a_p = multiprocessing.Process(target=work_a)# Subprocess 1
    # a_p.start() #Subprocess 1 execution
    b_p = multiprocessing.Process(target=work_b)  # Subprocess 2
    # b_p.start() # Subprocess 2 execution

    for p in (a_p,b_p): # All subprocesses execute
        p.start()
    for p in (a_p,b_p): ## After executing the sub process, execute the main process of print below
        p.join()  # block

    print("Time consuming:",time.time() - start) # Main process
    print("parent pid is %s" % os.getpid()) # Main process





– if the subprocess 1 is blocked and the subprocess 2 is not blocked, the subprocess 1 will be executed first, then the main process and finally the subprocess 2

import multiprocessing
import time
import os

def work_a():
    for i in range(10): # Cycle 0-9
        print(i,"a",os.getpid()) # Print cycle #os.getpid() view the pid process of the system
        time.sleep(1) # Wait 1 second
def work_b():
    for i in range(10):
        print(i,"b",os.getpid())
        time.sleep(1)

if __name__ == "__main__" :
    start = time.time() # Main process
    a_p = multiprocessing.Process(target=work_a)# Subprocess 1
    a_p.start() #Subprocess 1 execution
    a_p.join() # Subprocess blocking
    b_p = multiprocessing.Process(target=work_b)  # Subprocess 2
    b_p.start() # Subprocess 2 execution

    # for p in (a_p,b_p): # All subprocesses execute
    #     p.start()
    # for p in (a_p,b_p):
    #     p.join()  # block

    print("Time consuming:",time.time() - start) # Main process
    print("parent pid is %s" % os.getpid()) # Main process




Process pool and process lock

# -*- coding: utf-8 -*-
import os
import time
import multiprocessing

def work(count):
    print(count,os.getpid()) # Print process number
    time.sleep(5)

if __name__ == "__main__":
    pool = multiprocessing.Pool(5) # The process pool is limited to 5
    for i in range(20):
        pool.apply_async(func=work,args=(i,))
    time.sleep(20)

# -*- coding: utf-8 -*-
import os
import time
import multiprocessing

def work(count):
    print(count,os.getpid()) # Print process number
    time.sleep(5)

if __name__ == "__main__":
    pool = multiprocessing.Pool(5) # The process pool is limited to 5
    for i in range(20):
        pool.apply_async(func=work,args=(i,))
    pool.close()#Close the process pool and no longer accept new processes
    pool.join()#The main process is blocked waiting for the child process to exit
# -*- coding: utf-8 -*-
import os
import time
import multiprocessing

def work(count):
    print(count,os.getpid()) # Print process number
    time.sleep(5)
    return "result is %s,pid is %s"%(count,os.getpid())

if __name__ == "__main__":
    pool = multiprocessing.Pool(5) # The process pool is limited to 5
    results = []
    for i in range(20):
        result = pool.apply_async(func=work,args=(i,))
        results.append(result)

    for res in results:
        print(res.get())


# -*- coding: utf-8 -*-
import os
import time
import multiprocessing

def work(count,lock):
    lock.acquire()# Lock
    print(count,os.getpid()) # Print process number
    time.sleep(5)
    lock.release() # Unlock

if __name__ == "__main__":
    pool = multiprocessing.Pool(5) # The process pool is limited to 5
    manager = multiprocessing.Manager() # Process locks are only effective for one process at a time
    lock = manager.Lock()
    for i in range(20):
        pool.apply_async(func=work,args=(i,lock))
    pool.close()#Close the process pool and no longer accept new processes
    pool.join()#The main process is blocked waiting for the child process to exit

thread

1, Possible problems and solutions when multiple threads are running
Functions executed through threads cannot obtain return values - how to communicate between threads: modifying files simultaneously through multiple threads in the queue may cause data errors and confusion - how to avoid resource preemption between threads: create thread locks
Too many threads may lead to insufficient resources or even crash - how to avoid creating too many threads: creating a thread pool
2, It is solved by column to column communication

3, Create thread lock
Write the lock code where the lock needs to be added in the thread code, and write the unlock code where the lock needs to be released
Using module: threading
How to lock: threading Lock(). acquire()
How to unlock: threading Lock(). release()
4, Create thread pool
First write out the method of creating thread pool, and then put threads into the thread pool
Module used: concurrent futures


ordinary

import random
import time
import threading # Multithreading module
lists = ["python","django","tornado",
         "flask","bs5","requests"]
new_lists = []

def work():
    if len(lists) == 0 :
        return
    data = random.choice(lists) #Randomly select a data from a non empty sequence and return it
    lists.remove(data)# The remove() function deletes the specified element from the list
    new_data = "%s_new"% data # rename
    new_lists.append(new_data) # Append to new list
    time.sleep(1) # Wait 1 second

if __name__=="__main__":
    start = time.time() # start time
    for i in range(len(lists)): # Run the work function in a loop according to the length of the list
        work()
    print("old list:",lists)  # Print the contents of lists
    print("new list:",new_lists) # Print the contents of the new list
    print("time is %s"%(time.time()-start)) # time consuming


Thread instance

import random
import time
import threading # Multithreading module
lists = ["python","django","tornado",
         "flask","bs5","requests"]
new_lists = []

def work():
    if len(lists) == 0 :
        return
    data = random.choice(lists) #Randomly select a data from a non empty sequence and return it
    lists.remove(data)# The remove() function deletes the specified element from the list
    new_data = "%s_new"% data # rename
    new_lists.append(new_data) # Append to new list
    time.sleep(1) # Wait 1 second

if __name__=="__main__":
    start = time.time() # start time
    t_list = []
    for i in range(len(lists)): # Run the work function in a loop according to the length of the list
        t = threading.Thread(target=work) # Instantiate a thread
        t_list.append(t)
        t.start() # Start thread
    for t in t_list:
        t.join()# After blocking the subroutine, run the following print

    print("old list:",lists)  # Print the contents of lists
    print("new list:",new_lists) # Print the contents of the new list
    print("time is %s"%(time.time()-start)) # time consuming


Thread pool


import time
import threading
import concurrent
from concurrent.futures import ThreadPoolExecutor


lock = threading.Lock() # Instantiate a thread lock

def work(i):
    # lock.acquire()
    print(i)
    time.sleep(1)
    # lock.release()


if __name__=="__main__":
    t  = ThreadPoolExecutor(2) # Thread pool 2 threads
    for i in range(20):
        t.submit(work,(i,))


Wire program lock

import time
import threading
import concurrent
from concurrent.futures import ThreadPoolExecutor


lock = threading.Lock() # Instantiate a thread lock

def work(i):
    lock.acquire()
    print(i)
    time.sleep(1)
    lock.release()


if __name__=="__main__":
    t  = ThreadPoolExecutor(2) # Thread pool 2 threads
    for i in range(20):
        t.submit(work,(i,))

asynchronous

Asynchrony is relative to synchronization. Synchronization means order, and asynchrony means disorder. It is precisely because of the disorder of asynchrony that the coordination between various programs has become a big problem. Asynchronous programming is the programming to solve this problem. It takes process, thread, coordination and function / method as the basic unit of task execution program, combined with callback, event loop, semaphore and other mechanisms, A programming method to improve the overall execution efficiency and concurrency of the program
How to implement asynchrony in Python
Method 1:
How to define an asynchronous, use the keyword: async
In an asynchronous program, how to call another asynchronous program, use the keyword: await
How to call an asynchronous function, use the module: asyncio

Synchronous execution instance

import time

def a():
    for i in range(10):
        print(i,"I am A")
        time.sleep(1)# waiting time
    return "a function"

def b():
    for i in range(10):
        print(i,"I am B")
        time.sleep(1)
    return "b function"

if __name__=="__main__":
    start = time.time()
    a()
    b()
    print("time consuming%s"%(time.time()-start))


Asynchronous execution instance

import time
import asyncio

async def a():
    for i in range(10):
        print(i,"I am A")
        await asyncio.sleep(1) #  time.sleep is cpu level blocking. We need to change asyncio sleep
    return "a function"

async def b():
    for i in range(10):
        print(i,"I am B")
        await asyncio.sleep(1)
    return "b function"

async def main():
    result = await asyncio.gather(
        a(),
        b()
    )
    print(result)

if __name__=="__main__":
    start = time.time()
    asyncio.run(main())
    print("time consuming%s"%(time.time()-start))


import time
import gevent

def gevent_a():
    for i in range(10):
        print(i,"I am A")
        gevent.sleep(1) # block
    return "I am gevent a result"

def gevent_b():
    for i in range(10):
        print(i,"I am B")
        gevent.sleep(1)
    return "I am gevent b result"


if __name__=="__main__":
    start = time.time()
    g_a = gevent.spawn(gevent_a)
    g_b = gevent.spawn(gevent_b)
    gevent_list = [g_a,g_b]
    result = gevent.joinall(gevent_list)
    print(result)
    print("time consuming%s"%(time.time()-start))

Tags: Python programming language

Posted by BuzzLY on Thu, 05 May 2022 14:23:43 +0300