Spring cloud course: 15 Introduction and service degradation of hystrix circuit breaker

Hystrix circuit breaker

1, Overview

Problems faced by distributed systems

Applications in complex distributed architecture have dozens of dependencies, and each dependency inevitably fails at some time.

Service avalanche

When calling between multiple microservices, suppose that microservice A calls microservice B and microservice C, and microservice B and microservice C call other microservices, which is the so-called "fan out effect". If the call response time of A microservice on the fan out link is too long or unavailable, the call to microservice A will occupy more and more system resources, resulting in system crash, which is the so-called "avalanche effect"

Hystrix is an open source library for dealing with delay and fault tolerance of distributed systems. In distributed systems, many dependencies inevitably fail to call, such as timeout and exception. Hystrix can ensure that in the case of a dependency failure, it will not lead to overall service failure, avoid cascading failures, and improve the elasticity of distributed systems.

"Circuit breaker" itself is a kind of switching device. When a service unit fails, it returns an expected and processable alternative response (Fallback) to the caller through the fault monitoring of the circuit breaker (similar to a blown fuse) Instead of waiting for a long time or throwing exceptions that cannot be handled by the caller, this ensures that the thread sent by the service call will not be occupied unnecessarily for a long time, so as to avoid the spread and even avalanche of faults in the distributed system


 

1. Function

service degradation
Service fuse
Near real-time monitoring

2. Current situation: stop, change and maintain

 

Official website: https://github.com/Netflix/Hystrix/wiki/How-To-Use

Hystrix official announcement, stop and enter the dimension

Hystrix is no longer in active development, and is currently in maintenance mode.

2, Key concepts of Hystrix

1. Service degradation FallBack

Returns a composite expected processable alternative response to the caller, such as "server busy, please try again later!" Don't let the customer wait and return a friendly prompt immediately. FallBack
What conditions trigger a downgrade

  • Abnormal program operation
  • Timeout
  • Service fuse triggers service degradation
  • Full thread pool / semaphore will also cause service degradation

2. Service Break

After reaching the maximum service access, the user will directly refuse the access, and then return the friendly prompt with the method of service degradation.
That is, fuse - > service degradation - and then blow - restore the call link

3. Service flow limit

Second kill, high concurrency and other operations are strictly prohibited. Queue up for N for 1 second and proceed in an orderly manner

3, Hystrix case

1. Construction

1.1 create a new module

cloud-provider-hystrix-payment8001

1.2 pom

Spring cloud starter Netflix hystrix and Eureka need to be introduced


    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>
        <!--eureka-client-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

1.3 yml

server:
  port: 8001
spring:
  application:
    name: cloud-provider-hystrix-payment
eureka:
  client:
    register-with-eureka: true
    fetchRegistry: true
    service-url:
      defaultZone: http://eureka7001.com:7001/eureka

1.4 main

@SpringBootApplication
@EnableEurekaClient
public class HystrixPaymentMain8001 {
    public static void main(String[] args) {
        SpringApplication.run(HystrixPaymentMain8001.class,args);
    }
}

1.5 service

@Service
public class PaymentService {
    /*Normal access OK*/
    public String paymentInfo_OK(Integer id){
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_OK,ID:  "+id;
    }
    public String paymentInfo_Timeout(Integer id) {
        try{
            TimeUnit.SECONDS.sleep(3);}
        catch (    InterruptedException e){
            e.printStackTrace();
        }
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_Timeout,ID: "+id;
    }
}

1.6 controller

@RestController
@Slf4j
public class PaymentController {
    @Resource
    private PaymentService paymentService;
    @Value("${server.port}")
    private String serverPort;

    @GetMapping("/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id")Integer id){
        String result=paymentService.paymentInfo_OK(id);
        log.info("****result:"+result);
        return  result;
    }
    @GetMapping("/payment/hystrix/timeout/{id}")
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id){
        String result=paymentService.paymentInfo_Timeout(id);
        log.info("****result:"+result);
        return  result;
    }


}

1.7 start this service and Eureka service

1.8 viewing normal access time

3000ms 

Less than 10ms

 

 

2.Jmeter high concurrency test 8001

2.1 create a new thread group

 

2.2 after opening the pressure measurement, the time consumption of the two interfaces increases significantly

Check the time consumption after opening the pressure measurement

ok:1700ms

timeout:  4200ms

The default number of working threads of tomcat is full, and there are no extra threads to decompose the pressure and processing.

2.3 conclusion

The above is the self-test of the service provider. If the consumer visits, he can only wait, which eventually leads to the dissatisfaction of the consumer 80 and the death of the service 8001.

3. Create a new consumer 80 and use the consumer test interface

3.1. Create a new module cloud consumer feign hystrix order80

3.2 pom

    <dependencies>
        <!--open feign-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-openfeign</artifactId>
        </dependency>
        <!--eureka-client-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
        <dependency>
            <groupId>com.zhl.springcloud</groupId>
            <artifactId>cloud-api-commons</artifactId>
            <version>${project.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

3.3. yml

server:
  port: 80

spring:
  application:
    name: cloud-consumer-feign-hystrix-order80

eureka:
  client:
    register-with-eureka: true
    fetchRegistry: true
    service-url:
      defaultZone: http://eureka7001.com:7001/eureka

3.4 main

@SpringBootApplication
@EnableFeignClients
public class FeignHystixOrderMainF80 {
    public static void main(String[] args) {
        SpringApplication.run(FeignHystixOrderMainF80.class,args);
    }
}

3.5 Service

com.zhl.springcloud.service.PaymentHystrixService

@Component
@FeignClient(value = "CLOUD-PROVIDER-HYSTRIX-PAYMENT")
public interface PaymentHystrixService {
    @GetMapping("/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id")Integer id);
    @GetMapping("/payment/hystrix/timeout/{id}")
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id);
}

3.6 Controller

com.zhl.springcloud.controller.OrderHystrixController

@RestController
@Slf4j
public class OrderHystrixController {
    @Resource
    PaymentHystrixService paymentHystrixService;

    @GetMapping("/consumer/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id")Integer id){
        return paymentHystrixService.paymentInfo_OK(id);
    }
    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id){
        return paymentHystrixService.paymentInfo_TimeOut(id);
    }
}

3.7 call test

Testing the OK interface takes 8ms

Open JMeter for pressure measurement, and then visit

4. Fault phenomenon and cause

8001. The access is busy and the client is slow accordingly.

5. Conclusion

It is precisely because of the above faults or poor performance that the technology of} degradation, fault tolerance and current limiting was born.

6. How to solve it? Need to be addressed

  • The service provider times out and the server slows down: consumers can't wait all the time. There must be service degradation
  • Service provider error (downtime or program running error): consumers cannot wait all the time, and there must be service degradation
  • The waiting time of the consumer is less than the running time of the service provider, and the consumer handles the degradation by himself.

7. Service degradation

7.1. Degraded configuration

    @HystrixCommand

7.2 analysis of service providers

Set the peak value of the self call timeout time. It can run normally within the peak value. If it exceeds the need for a thorough method, it will be used as a service degradation fallback

7.3 service provider 8001 service degradation code optimization

Once calling the service method fails and an error message is thrown, it will automatically call the method specified in the fallbackMethod call class marked by @ HystrixCommand

Timeout or degradation are handled by fallbackMethod

Annotate the timeout handling method with @ HystrixCommand +,

7.4 service provider 8001 service degradation optimization

7.4.1 Service:

    com.zhl.springcloud.service.PaymentService

@Service
public class PaymentService {
    /*Normal access OK*/
    public String paymentInfo_OK(Integer id){
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_OK,ID:  "+id;
    }

    /*Timeout or degradation are handled by fallbackMethod*/
    @HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            //The normal timeout time is 3 seconds, and the fallbackMethod will be executed in case of timeout
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "3000")
    })
    public String paymentInfo_Timeout(Integer id) {
        int timenumner=5;
        try{
            TimeUnit.SECONDS.sleep(timenumner);}
        catch (    InterruptedException e){
            e.printStackTrace();
        }
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_Timeout,ID: "+id+"\t"+"Time consuming:"+timenumner;
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_Timeout,ID: "+id+"\t"+"handle at a lower grade";
    }
}

7.4.2 activate the main startup class and add a new annotation @ enablercircuitbreaker

@SpringBootApplication
@EnableEurekaClient
@EnableCircuitBreaker
public class HystrixPaymentMain8001 {
    public static void main(String[] args) {
        SpringApplication.run(HystrixPaymentMain8001.class,args);
    }
}

7.4.3 timeout degradation test

The service takes 5 seconds, and the degradation information is returned in 3 seconds.

7.4.4 abnormal test

Modify business class

    /*Timeout or degradation are handled by fallbackMethod*/
    @HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            //The normal timeout time is 3 seconds, and the fallbackMethod will be executed in case of timeout
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "3000")
    })
    public String paymentInfo_Timeout(Integer id) {
        int timenumner=5;
        /*Test error*/
        int age=10/0;
        try{
            TimeUnit.SECONDS.sleep(timenumner);}
        catch (    InterruptedException e){
            e.printStackTrace();
        }
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_Timeout,ID: "+id+"\t"+"Time consuming:"+timenumner;
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "Thread pool:"+Thread.currentThread().getName()+" The system is busy or running. Please try again later, ID: "+id+"\t"+"handle at a lower grade";
    }

Access test:

7.5 customer service 80 service degradation

7.5.1 # YML # configure OpenFeign # support # Hystrix

server:
  port: 80

spring:
  application:
    name: cloud-consumer-feign-hystrix-order80
eureka:
  client:
    service-url:
      defaultZone: http://eureka7001.com:7001/eureka
#OpenFeign opens Hystrix
feign:
  hystrix:
    enabled: true

7.5.2 Main open fuse @ enablercircuitbreaker

@SpringBootApplication
@EnableFeignClients
@EnableCircuitBreaker
public class FeignHystixOrderMainF80 {
    public static void main(String[] args) {
        SpringApplication.run(FeignHystixOrderMainF80.class,args);
    }
}

7.5.3 Controller

com.zhl.springcloud.controller.OrderHystrixController

@RestController
@Slf4j
public class OrderHystrixController {
    @Resource
    PaymentHystrixService paymentHystrixService;

    @GetMapping("/consumer/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id")Integer id){
        return paymentHystrixService.paymentInfo_OK(id);
    }
    /*1.5 seconds until timeout*/
    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    @HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "1500")
    })
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id){
        return paymentHystrixService.paymentInfo_TimeOut(id);
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "I'm consumer 80. The other party's payment system is busy. Please try again in 10 seconds or make an error. Please check yourself (╥╯^╰╥)";
    }
}

7.5.4 service provider business code: 5 seconds timeout, 3 seconds business processing time

@Service
public class PaymentService {
    /*Normal access OK*/
    public String paymentInfo_OK(Integer id){
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_OK,ID:  "+id;
    }

    /*Timeout or degradation are handled by fallbackMethod*/
    @HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            //The normal timeout time is 3 seconds, and the fallbackMethod will be executed in case of timeout
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "5000")
    })
    public String paymentInfo_Timeout(Integer id) {
        int timenumner=3;
        /*Test error*/
        try{
            TimeUnit.SECONDS.sleep(timenumner);}
        catch (    InterruptedException e){
            e.printStackTrace();
        }
        return  "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_Timeout,ID: "+id+"\t"+"Time consuming:"+timenumner;
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "Thread pool:"+Thread.currentThread().getName()+" The system is busy or running. Please try again later, ID: "+id+"\t"+"handle at a lower grade";
    }
}

7.5.5 timeout test effect: 3 seconds for the provider and 1.5 seconds for the consumer

7.5.6 consumer error test

int age=10/0} test error

    /*1.5 seconds until timeout*/
    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    @HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "1500")
    })
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id){
        int age=10/0;
        return paymentHystrixService.paymentInfo_TimeOut(id);
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "I'm consumer 80. The other party's payment system is busy. Please try again in 10 seconds or make an error. Please check yourself, (╥╯^╰╥)";
    }

7.6 global configuration of service degradation

At present, each business corresponds to a thorough method. The code is inflated and needs to be unified and customized separately.

7.6.1 Controller level configuration

Add @ defaultproperty annotation to the Controller to specify the global processing method in the Controller

com.zhl.springcloud.controller.OrderHystrixController

Only @ HystrixCommand is referenced on the interface without specifying fallbackMethod  

Define a global processing method of Controller, here is ControllerGlobalFallBack

Note:

The Fallback Method at the interface level requires that the parameters and return types be consistent with the interface.

The FallbackMethod at class level is ControllerGlobalFallBack, and cannot have parameters. Otherwise, an error is reported: FallbackMethod was't found:

@RestController
@Slf4j
/*Controller Level default processing method*/
@DefaultProperties(defaultFallback = "ControllerGlobalFallBack")
public class OrderHystrixController {
    @Resource
    PaymentHystrixService paymentHystrixService;

    @GetMapping("/consumer/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id")Integer id){
        return paymentHystrixService.paymentInfo_OK(id);
    }
    /*1.5 seconds until timeout*/
    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    /*Custom specified processing method*/
    /*@HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "1500")
    })*/
    @HystrixCommand
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id){
        //int age=10/0;
        return paymentHystrixService.paymentInfo_TimeOut(id);
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "I'm consumer 80. The other party's payment system is busy. Please try again in 10 seconds or make an error. Please check yourself, (╥╯^╰╥)";
    }
    /*Controller Global FallBack processing*/
    public String ControllerGlobalFallBack(){
        return "ControllerGlobalFallBack";
    }
}

7.6.2 unified configuration of service call interface

Create a new class to implement the interface of FeignClient

@Component
public class HystrixPaymentFallBackService implements PaymentHystrixService {
    @Override
    public String paymentInfo_OK(Integer id) {
        return "---HystrixPaymentServiceImpl_paymentInfo_OK fallback ";
    }

    @Override
    public String paymentInfo_TimeOut(Integer id) {
        return "----HystrixPaymentServiceImpl_paymentInfo_TimeOut fallback";
    }
}

YML start Feign

#OpenFeign opens Hystrix
feign:
  hystrix:
    enabled: true

Uncomment @ HystrixCommand in controller

@RestController
@Slf4j
/*Controller Level default processing method*/
//@DefaultProperties(defaultFallback = "ControllerGlobalFallBack")
public class OrderHystrixController {
    @Resource
    PaymentHystrixService paymentHystrixService;

    @GetMapping("/consumer/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id")Integer id){
        return paymentHystrixService.paymentInfo_OK(id);
    }
    /*1.5 seconds until timeout*/
    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    /*Custom specified processing method*/
    /*@HystrixCommand(fallbackMethod = "paymentInfo_TimeoutHandler",commandProperties = {
            @HystrixProperty( name="execution.isolation.thread.timeoutInMilliseconds",value = "1500")
    })*/
    /* @HystrixCommand*/
    public String paymentInfo_TimeOut(@PathVariable("id")Integer id){
        //int age=10/0;
        return paymentHystrixService.paymentInfo_TimeOut(id);
    }
    /*handle at a lower grade*/
    public   String paymentInfo_TimeoutHandler(Integer id){
        return  "I'm consumer 80. The other party's payment system is busy. Please try again in 10 seconds or make an error. Please check yourself, (╥╯^╰╥)";
    }
    /*Controller Global FallBack processing*/
    public String ControllerGlobalFallBack(){
        return "ControllerGlobalFallBack";
    }
}

Access timeout interface test

 

Posted by lbaxterl on Wed, 04 May 2022 04:02:40 +0300