The most powerful error retry Library in Python

When we write programs, especially those related to network requests, such as calling web interfaces, running web crawlers and other tasks, we often encounter some accidental request failures. In this case, if we simply catch errors and skip the corresponding tasks, it is certainly not rigorous, especially in web crawlers, there will be the risk of losing valuable data.

In such cases, it is necessary to add some * * error retry * * strategies to our program logic. Teacher Fei, I wrote an article a few years ago to introduce the retry Library in Python, but it has a relatively single function and can only meet basic needs.

The tenacity library I want to introduce to you today may be the best error retry Library in Python ecology at present. Let's take a look at its main functions ~ like to remember collection, attention and praise.

1. Common functions in tenacity

As a third-party Python library, we can use pip install tenacity to install it. After installation, let's learn the main usage methods and features of tenacity:

1.1 basic use of tenacity

The core function of tenacity's error retry is implemented by its retry decorator. By default, when no parameters are passed to the retry decorator, it will keep trying again when the decorated function throws an error. For example, the following simple example:

import random
from tenacity import retry

@retry
def demo_func1():

    a = random.random()
    print(a)
    
    if a >= 0.1:
        raise Exception

demo_func1()

It can be seen that every time a random number between 0 and 1 is generated in our function body, the error will be thrown when the random number does not exceed 0.1. Otherwise, tenacity will catch the error throwing behavior every time and try again immediately.

1.2 set the maximum number of retries

Sometimes we have a limit to the tolerance of retrying due to logical errors of a certain function. For example, when we call a network interface, if we fail to execute it n times in a row, we may think that the task itself has defects and will be normal one day instead of retrying.

At this time, we can use stop in tenacity_ after_ The attempt function is passed in as the stop parameter in retry(), which adds an end point to our "endless" error retry process, where stop_after_attempt() accepts an integer as the number of * * maximum retries * *:

from tenacity import retry, stop_after_attempt

@retry(stop=stop_after_attempt(3))
def demo_func2():
    
    print('Function execution')
    
    raise Exception
    
demo_func2()

It can be seen that after limiting the maximum number of retries, after three retries, our function still throws an error after the fourth execution, and officially throws the corresponding Exception error in the function, ending the retry process.

1.3 set the maximum timeout length of retry

In addition to setting the maximum number of error retries as in the previous section, tenacity also provides us with stop_after_delay() function to set the maximum time-consuming of the whole retry process. Beyond this time, the retry process will end:

import time
from tenacity import retry, stop_after_delay

# Set the maximum retry timeout to 5 seconds
@retry(stop=stop_after_delay(5))
def demo_func3():
    
    time.sleep(1)
    print(f'Past {time.time() - start_time} second')
    
    raise Exception

# Record start time
start_time = time.time()
demo_func3()

1.4 combined retry stop conditions

If our task needs to add the maximum number of retries and the maximum timeout time limit at the same time, in tenacity, we only need to use the | operator to combine different restrictions and then pass in the stop parameter of retry(). For example, in the following example, when our function retries for more than 3 seconds or more than 5 times, we can end the retry:

import time
import random
from tenacity import retry, stop_after_delay, stop_after_attempt

@retry(stop=(stop_after_delay(3) | stop_after_attempt(5)))
def demo_func4():
    
    time.sleep(random.random())
    print(f'Past {time.time() - start_time} second')
    
    raise Exception

# Record start time
start_time = time.time()
demo_func4()

As you can see, in the above demonstration, the "maximum retry 5 times" limit was reached first, thus ending the retry process.

1.5 set the time interval between adjacent retries

In some cases, we don't want to start the next retry immediately after each retry throws an error. For example, in the crawler task, in order to better disguise our program, tenacity provides a series of very practical functions, combined with the wait parameter of retry(), to help us properly deal with the time interval between adjacent retries. The more practical methods are as follows:

1.5.1 set fixed time interval

We use wait in tenacity_ Fixed() can set a fixed waiting interval in seconds between adjacent retries, just like the following simple example:

import time
from tenacity import retry, wait_fixed, stop_after_attempt

# Set the retry wait interval to 1 second
@retry(wait=wait_fixed(1), stop=stop_after_attempt(3))
def demo_func5():
    
    print(f'Past {time.time() - start_time} second')
    
    raise Exception
    
# Record start time
start_time = time.time()
demo_func5()

1.5.2 setting random time interval

In addition to setting a fixed time interval, tenacity can also use wait_random() helps us set evenly distributed random numbers for adjacent retries. Just set the range of uniform distribution:

import time
from tenacity import retry, wait_random, stop_after_attempt

# Set the retry waiting interval to a random number between 1 and 3
@retry(wait=wait_random(min=1, max=3), stop=stop_after_attempt(5))
def demo_func6():
    
    print(f'Past {time.time() - start_time} second')
    
    raise Exception

# Record start time
start_time = time.time()
demo_func6()

It can be observed that the waiting time after each retry is random~

1.6 customize whether to trigger retry

The default strategy of retry() in tenacity is to retry when the decorated function execution process "throws any error". However, in some cases, we may need to catch / ignore specific error types or catch exception calculation results.

tenacity also has built-in relevant practical functions:

1.6.1 capture or ignore specific error types

Use retry in tenacity_ if_ exception_ Type () and retry_if_not_exception_type(), together with the retry parameter of retry(), we can catch or ignore specific error types:

from tenacity import retry, retry_if_exception_type, retry_if_not_exception_type

@retry(retry=retry_if_exception_type(FileExistsError))
def demo_func7():
    
    raise TimeoutError
    
@retry(retry=retry_if_not_exception_type(FileNotFoundError))
def demo_func8():

    raise FileNotFoundError

1.6.2 user defined function result condition judgment function

We can write additional conditional judgment functions to cooperate with retry in tenacity_ if_ Result(), which is used to judge the return result of the function by user-defined conditions. The retry operation will be triggered only when it returns True:

import random
from tenacity import retry, retry_if_result

@retry(retry=retry_if_result(lambda x: x >= 0.1))
def demo_func9():
    a = random.random()
    print(a)
    return a

# Record start time
demo_func9()

1.7 statistics on error retry of function

For the function decorated by tenacity's retry(), we can print its retry The statistics property is used to view the statistics record results of error retries. For example, here we demo the example function executed earlier_ Print the statistical results of func9():

demo_func9.retry.statistics

In addition to the above functions, tenacity also has many special features, which can be combined with other Python functions such as logging module, asynchronous function and coroutine to realize more advanced functions. Interested friends can go there https://github.com/jd/tenacity Learn more.

The above is the whole content of this article. Welcome to discuss it in the comment area~

Recommended articles

Technical exchange

For complete code and data acquisition, you can ask me for it

Technical exchange, job promotion, dry goods summary, and interactive exchange with 3000 + developers from famous universities and enterprises such as Alibaba, Peking University, Tsinghua University, Chinese Academy of Sciences, CMU, Tencent and Baidu~

At present, a technical exchange group has been opened, with more than 3000 group friends. The best way to add notes is: source + Interest direction, which is convenient to find like-minded friends

  • Method ① send the following pictures to wechat, long press to identify, and the background replies: add group;
  • Mode ②. Add micro signal: dkl88191, remarks: from CSDN
  • Method ③. Wechat search official account: Python learning and data mining, background reply: add group

Tags: Java Back-end Interview

Posted by joyser on Sat, 14 May 2022 21:43:46 +0300