Day 37 Coroutine related

Day 37

1. shutdown of thread pool and process pool

self.shutdown(wait=True)

​ Wait for all line/process tasks to complete and close the pool before executing the contents of the main process. Generally placed outside the for loop, it should be noted that after closing the line/process pool, you can no longer submit tasks. Re-execute the contents of the main process

  • wait=True, wait for all tasks in the pool to finish executing and reclaiming resources before continuing

  • wait=False, return immediately, and will not wait for the tasks in the pool to finish executing

  • But no matter what the value of the wait parameter is, the entire program will wait until all tasks are executed.

  • submit and map must be before shutdown

    import time
    from concurrent.futures import ThreadPoolExecutor
    
    pool = ThreadPoolExecutor(3)
    
    
    def task():
        print("i'm starting......")
        time.sleep(2)
        print("i'm over....")
    
    
    def call_back(f):
        pass
    
    
    if __name__ == '__main__':
        for i in range(10):
            pool.submit(task).add_done_callback(call_back)
    
        pool.shutdown(wait=True)
        # pool.submit(task).add_done_callback(call_back)
        # cannot schedule new futures after shutdown
        print("The main process ends!")
        
    ---------------------------
    >>> i'm starting......
    >>> i'm starting......
    >>> i'm starting......
    >>> ....
    >>> The main process ends!
    

2. Timer

class Timer(Thread):

Timer (timer) is a derived class of Thread, which is used to call a method after a specified time. Timer is derived from Thread and has no instance method added.

  • t = Timer(30.0, f, args=None, kwargs=None) Basic usage
  • t.start() start
  • t.cancel() if the timer is still waiting, stop its action
from threading import Timer, current_thread


def print_data(name):
    print(f"{name} Playing "Black Myth"·Goku!")
    print(f"i am thread{current_thread().getName()}")


if __name__ == '__main__':
    for i in range(10):
        p = Timer(2, print_data, kwargs={"name": "Mao Cheng"})
        p.start()
-------------------------------------
>>> Mao Cheng is playing "Black Myth"·Goku!
	i am thread Thread-1
    ........
>>> Mao Cheng is playing "Black Myth"·Goku!
	i am thread Thread-10

3. Introduction to coroutines

A coroutine is a lightweight thread in user mode, that is, a coroutine is controlled and scheduled by the user program itself.

Coroutine is to achieve concurrency under single thread, under attribute thread

The problem to be solved by the coroutine: save state + switch

yield: generator, as long as there is a yield keyword in the function, this function is a generator, which can be saved state + switch through yield

advantage:

  • The switching overhead of the coroutine is smaller, it belongs to the program-level switching, and the operating system cannot perceive it, so it is more lightweight
  • Concurrency can be achieved in a single thread, maximizing the use of CPU

shortcoming:

  • The essence of coroutines is that they are single-threaded and cannot use multiple cores. One program can open multiple processes, each process opens multiple threads, and each thread opens a coroutine.
  • A coroutine refers to a single thread, so once a coroutine is blocked, the entire thread will be blocked

Features:

Concurrency must be implemented in only one single thread

Modifying shared data does not require locking

The user program saves the context stack of multiple control flows by itself (the state that needs to be saved)

Additional: When a coroutine encounters an IO operation, it automatically switches to other coroutines (how to implement detection of IO, yield, and greenlet cannot be achieved, the gevent module (select mechanism) is used)

Fourth, the greetlet module

switch switches execution to this greenlet

from greenlet import greenlet
import time


def eat():
    print("I took a bite!")
    time.sleep(1)
    p.switch()  # 2. Switch to play print("I played for a while!")
    print("I took another bite!")
    p.switch()  # 4. Switch to play print("I played for a while!")


def play():
    print("I played for a while!")
    e.switch()  # 3. Switch to eat print("I ate another bite!")
    time.sleep(1)
    print("I played for a while!")


if __name__ == '__main__':
    e = greenlet(eat)
    p = greenlet(play)
    e.switch()  # 1. Switch to eat print("I took a bite!")
-----------------------------------
>>> I took a bite!
>>> I played for a while!
>>> I took another bite!
>>> I played for a while!

5. gevent

  • gevent is encapsulated on the basis of greenlet to make gevent easier to use.
  • gevent uses an implicit start event loop, that is, a special coroutine is started to start the event loop when blocking is required;
  • If a task has no io operation, it will be executed until completion; other coroutines have no chance to execute;
  • Automatically identify io events and give up CPU control time

Instructions

  • gevent.spawn(): Create a normal Greenlet object and switch;

  • # uncommonly used
    - gevent.spawn_later(seconds=3) # Delay creating a normal Greenlet object and switching 
    - gevent.spawn_raw() # The created coroutine object belongs to a group
    - gevent.getcurrent() # returns the currently executing
    - greenlet gevent.joinall(jobs): Add a coroutine task to the event loop and receive a list of tasks
    - gevent.wait() # You can replace the join function to wait for the end of the loop, or you can pass in a list of coroutine objects, ,
    - gevent.kill() # kill a coroutine
    - gevent.killall() # kills all coroutines in a coroutine list 
    
  • monkey.patch_all(): very important, will automatically replace some standard modules of python with gevent framework

import gevent
import time
from gevent import monkey;monkey.patch_all()  # Similar to monkey patch


def eat():
    print("I took a bite!")
    time.sleep(1)
    print("I took another bite!")


def play(name):
    print(f"{name}Play for a while!")
    time.sleep(2)
    print(f"{name}Played for a while!")


if __name__ == '__main__':
    e = gevent.spawn(eat)
    p = gevent.spawn(play, "Trump")
    e.join()  #Fixed usage If not used, it will exit when encountering IO operations
    p.join()

6. asyncio

# Libraries that officially support coroutines



# import time
# import asyncio
#
# # Turn ordinary functions into coroutine functions
# # 3.5 was written like this before
# @asyncio.coroutine
# def task():
#     print('started')
#     yield from asyncio.sleep(1)  #asyncio.sleep(1) simulate io
#     print('End')
#
#
# loop=asyncio.get_event_loop()  # Get a time loop object#
#
# # The coroutine function is parenthesized, it will not actually be executed, it needs to be submitted to the loop, so that the loop loops to execute
# # List of coroutine functions
#
# ctime=time.time()
# t=[task(),task()]
# loop.run_until_complete(asyncio.wait(t))
# loop.close()
# print(time.time()-ctime)


import time
import asyncio
from threading import current_thread
# Indicates that I am a coroutine function, equivalent to the decorator before 3.5
async def task():
    print('here we go')
    print(current_thread().name)
    await asyncio.sleep(3)  # await is equivalent to the original yield from
    print('ended')

async def task2():
    print('here we go')
    print(current_thread().name)
    await asyncio.sleep(2)
    print('ended')

loop=asyncio.get_event_loop()

ctime=time.time()
t=[task(),task2()]
loop.run_until_complete(asyncio.wait(t))
loop.close()
print(time.time()-ctime)

7 io models (emphasis, abstract, difficult, interview focus)

1 The memory is divided into the kernel buffer and the user buffer (resources downloaded from the network, resources loaded from the hard disk, first put into the kernel buffer---->copy To the application's buffer, the application can use this data)

2 io Model:
	-block io(BIO)
    -non-blocking io(NIO)
    -io multiplex()  select(windows support, windows not support epoll,Officially not available redis of window Version), poll,epoll(linux support)
    -asynchronous io: 
    -signal driven io(theory, not considered)

Supplementary virtual environment

1 Solve the problem of different versions of modules that different projects depend on
2 pycharm Select when creating a project in
	-Can this virtual environment be used by other projects (depending on your choice)
    -Create a virtual environment based on the current state of the system interpreter or the clean state
3 Install the module:
	-cmd Under the window: pip3 install flask   (Under whom, you must confirm it)
    -recommend you to use pycharm: setting--->that set
    -pycharm down terminal download (equivalent to cmd),Compare cmd Well, it has a hint
4 How to switch to the system environment using the virtual environment
    
4 The role of environment variables
	-Add a path to the environment variable, and the commands under the path can be executed at any location.

Tags: Python epoll

Posted by richardw on Fri, 20 May 2022 17:44:24 +0300