k8s install istio

Istio servicemesh solution: k8s install istio

https://istio.io/

https://www.servicemesher.com/tags/istio/page/3/

https://www.jianshu.com/p/e23e3e74538e

https://time.geekbang.org/article/2355

1. Service Mesh concept

Service Mesh is also translated as "service grid" as the infrastructure layer of communication between services.

 

Service Mesh is an infrastructure layer used to handle communication between services. Cloud native applications have complex service topologies, and Service Mesh ensures that requests can shuttle reliably in these topologies. In practical applications, Service Mesh is usually composed of a series of lightweight network agents, which are deployed with the application, but the application does not need to know their existence.

Service Mesh is actually an abstract layer above TCP/IP. It assumes that the underlying L3/L4 network can transmit bytes point-to-point (of course, it also assumes that the network environment is unreliable, so Service Mesh must have the ability to deal with network faults).

Service mesh # has the following characteristics:

  1. The middle layer of communication between applications;
  2. Lightweight network agent;
  3. Application is not aware;
  4. Decouple application retry, timeout, monitoring, tracking and service discovery;

2. Principle

Basic principles of Service Mesh

If we use one sentence to explain what Service Mesh is, we can compare it to TCP/IP between applications or micro services, which is responsible for network call, current limiting, fusing and monitoring between services. For writing applications, you generally don't need to care about the TCP/IP layer (such as RESTful applications through HTTP protocol). Similarly, using Service Mesh doesn't need to be related. Things between services that were originally implemented through applications or other frameworks, such as Spring Cloud and OSS, can now be handed over to Service Mesh.

Phil Cal ç ado explains the context of Service Mesh in detail in his blog Pattern: Service Mesh:

  1. Connect directly from the original hosts with network cables
  2. Emergence of network layer
  3. Control flow integrated into the application
  4. Decompose the control flow outside the application
  5. Integrated service discovery and circuit breaker in application
  6. There are software packages / libraries dedicated to service discovery and circuit breakers, Twitter's Finagle and Facebook's Proxygen. At this time, it is still integrated inside the application
  7. Open source software dedicated to service discovery and circuit breaker has emerged, such as NetflixOSS ecosystem
  8. Finally, Service Mesh appears as the middle layer of micro services

The architecture of Service Mesh is shown in the following figure

 

 

 

 

3. Solutions

At present, the open source solutions of community Service Mesh include linkerd launched by Buyant and Istio led by Google, IBM and other manufacturers. Linkerd is more mature and stable, Istio has richer functions, more powerful design, and the community is relatively stronger. Therefore, it is generally believed that Istio will have a better prospect, but after all, it is still in the early stage of the project and there are still many problems.

4. Introduction to Istio

Istio is a microservice management, protection and monitoring framework open source by Google, IBM and Lyft. Istio is Greek, which means "sail". The official Chinese document address is: https://istio.doczh.cn

https://blog.51cto.com/brucewang/2149104

4.1 what is ISTIO

Istio is an open platform for connecting, managing and protecting microservices.

Istio provides a simple way to establish a network for deployed services. The network has the functions of load balancing, inter service authentication, monitoring and so on, without any change to the service code. To enable the service to support istio, you only need to deploy a special sidecar in your environment, configure and manage agents using istio control plane functions, and intercept all network communications between microservices.

  • Automatic load balancing of HTTP, gRPC, WebSocket and TCP traffic.
  • Through rich routing rules, Retry, failover and fault injection, the traffic behavior can be fine-grained controlled.
  • Pluggable policy layer and configuration API, supporting access control, rate restriction and quota.
  • Automatic measurement, logging and tracking of all traffic in and out of the cluster entrance and exit.
  • Through powerful identity based authentication and authorization, secure inter service communication can be realized in the cluster.

4.2 what is the use of ISTIO

Istio provides a complete solution to meet the diversified needs of microservice applications by providing behavior insight and operation control for the whole service grid. It provides many key functions in the service network:

  • Traffic management. Control the flow between services and the flow direction of API calls, so as to make the calls more reliable and make the network more robust under bad conditions.
  • Service identity and security. Provide services in the grid with verifiable identity and the ability to protect service traffic, so that it can flow on networks with different credibility.
  • Policy execution. Apply the organizational policy to the interaction between services to ensure that the access policy is implemented and the resources are well distributed among consumers. Policy changes can be accomplished by configuring the grid rather than modifying the application code.
  • Telemetry: understand the dependencies between services and the nature and direction of traffic between them, so as to provide the ability to quickly identify problems.

 

4.3 architecture

Istio service grid is logically divided into data plane and control plane.

  • The data plane consists of a set of intelligent agents (Envoy) deployed in sidecar mode. These agents can regulate and control all network communication between microservices and mixers.
  • The control plane is responsible for managing and configuring agents to route traffic. In addition, the control plane configures the Mixer to implement policies and collect telemetry data.
    The following figure shows the different components that make up each panel:

 

 

 

Envoy

  • Dynamic service discovery
  • load balancing
  • TLS termination
  • Http / 2 & grpc proxy
  • Fuse
  • Health check, gray publishing based on percentage traffic split
  • fault injection
  • Rich metrics
    Envoy is deployed as a sidecar in the same Kubernetes pod as the corresponding service. This allows Istio to extract a large number of signals about traffic behavior as attributes, which can be used in Mixer to execute policy decisions and send them to the monitoring system to provide information about the behavior of the whole grid.

Mixer

Mixer is a platform independent component responsible for executing access control and usage policies on the service grid and collecting telemetry data from Envoy agents and other services. The agent extracts request level attributes and sends them to mixer for evaluation.

Pilot

Pilot provides Envoy sidecar with service discovery function and traffic management function for intelligent routing (such as A/B test, Canary deployment, etc.) and elasticity (timeout, Retry, fuse, etc.). It translates advanced routing rules that control traffic behavior into Envoy specific configurations and propagates them to sidecar at run time.

Citadel

Citadel provides powerful inter service and end-user authentication through built-in identity and credential management. It can be used to upgrade the unencrypted traffic in the service grid and provide operation and maintenance personnel with the ability to enforce policies based on service identification rather than network control. Starting with version 0.5, Istio supports role-based access control to control who can access your services

 

 

5. Istio and Kubernetes

http://blog.itpub.net/31543630/viewspace-2213615

Kubernetes provides deployment, upgrade and limited operational traffic management capabilities; The mechanism of service is used for service registration, discovery and forwarding. kubeproxy has certain forwarding and load balancing capabilities. However, it does not have the ability of upper layer, such as fusing, current limiting degradation, call chain management and so on  

Istio complements this part of k8s's ability in microservice governance. At the same time, it is built based on k8s, but it is not completely rebuilt like SpringCloud Netflix. Istio is a key link in Google's microservice governance.

 

 

 

Istio is closely combined with k8s, including: Sicecar runs in k8s pod and is deployed together as a proxy and business container. The deployment process is transparent to users. Mesh requires that the operation of business programs cannot perceive the existence of sidecar. The design of pod based on k8sd is more thorough and transparent to users. Users can't even perceive the process of deploying sidecar. Imagine that it would not be so convenient to deploy an agent through the VM.  

The pilot contains a controller, which can automatically discover the services and endpoints in K8S through list / watch Kube apiserver. It monitors events by registering a controller in Kubernetes, so as to obtain the relationship between Service and Kubernetes' Endpoint and Pod. However, at the forwarding level, Kube proxy forwarding is no longer used, but these mapping relationships are transformed into pilot's own forwarding model and distributed to envoy for forwarding.

 

 

 

K8s orchestration container service has become a de facto standard; Because of the matching of the characteristics of lightweight, rapid deployment, operation and maintenance between microservices and containers, microservices running in containers is also becoming a standard practice; For cloud native applications, using Kubernetes to build microservice deployment and cluster management capabilities and Istio to build service governance capabilities will gradually become the standard configuration for application microservice transformation.  

 

 

6. Install istio based on k8s

6.1}k8s environment

be based on kubernetes(3): kubedm installation k8s1 fifteen

# kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   26h   v1.15.3
k8s-node-1   Ready    node     25h   v1.15.3
k8s-node-2   Ready    node     25h   v1.15.3
k8s-node-3   Ready    node     25h   v1.15.3

 

 

6.2 downloading and preparing for installation

Get the latest Istio installation package

 

curl -L https://git.io/getLatestIstio | sh -

# curl -L https://git.io/getLatestIstio | sh -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100  3015  100  3015    0     0    997      0  0:00:03  0:00:03 --:--:-- 12108
Downloading istio-1.3.4 from https://github.com/istio/istio/releases/download/1.3.4/istio-1.3.4-linux.tar.gz ...  % Total% Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   614    0   614    0     0    614      0 --:--:-- --:--:-- --:--:--   614
  1 36.3M    1  390k    0     0   2413      0  4:23:25  0:02:45  4:20:40     0
  1 36.3M    1  390k    0     0   2408      0  4:23:58  0:02:46  4:21:12     0


...The download is too slow. I press the link wget Yes
wget  https://github.com/istio/istio/releases/download/1.3.4/istio-1.3.4-linux.tar.gz

[root@k8s-master istio]# tar -zxf istio-1.3.4-linux.tar.gz
[root@k8s-master istio]# ls
istio-1.3.4  istio-1.3.4-linux.tar.gz
[root@k8s-master istio]# cp -r  istio-1.3.4 /usr/local/istio
[root@k8s-master istio]# echo 'PATH=$PATH:/usr/local/istio/bin:' >>  /etc/profile

 

 

After entering the installation package directory, enter the install/kubernetes / directory to install the yaml files required by Kubernetes

 

[root@k8s-master istio]# cd istio-1.3.4/
[root@k8s-master istio-1.3.4]# ls
bin  install  istio.VERSION  LICENSE  README.md  samples  tools
[root@k8s-master istio-1.3.4]# cd install/
[root@k8s-master install]# ls
consul  gcp  kubernetes  README.md  tools
[root@k8s-master install]# cd  kubernetes/
[root@k8s-master kubernetes]# ls
global-default-sidecar-scope.yaml  istio-demo-auth.yaml  mesh-expansion.yaml  operator
helm                               istio-demo.yaml       namespace.yaml       README.md
[root@k8s-master kubernetes]#

 

6.3 installation

Select the installation method that does not enable TLS authentication by default. The yaml file that enables TLS installation method is istio demo auth yaml)

 

kubectl create -f istio-demo.yaml

 

6.4 confirmation of installation

 

# kubectl get pod -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
grafana-59d57c5c56-x6rqv                  1/1     Running     0          17m
istio-citadel-555bff45bf-nhxbw            1/1     Running     0          17m
istio-egressgateway-7fd45cc845-rd5gs      1/1     Running     0          17m
istio-galley-7f8b6db7d7-9x4rw             1/1     Running     0          17m
istio-grafana-post-install-1.3.4-7frpb    0/1     Completed   0          17m
istio-ingressgateway-64456bb8b-pg2cs      1/1     Running     0          17m
istio-pilot-85894cd4f5-t9n6t              2/2     Running     0          17m
istio-policy-7d8f44db64-8f6xv             2/2     Running     0          17m
istio-security-post-install-1.3.4-rw55q   0/1     Completed   0          17m
istio-sidecar-injector-65d5f8db56-7bk9g   1/1     Running     0          17m
istio-telemetry-85c7c59b86-xgmdl          2/2     Running     0          17m
istio-tracing-795c9c64c4-5f4kd            1/1     Running     0          17m
kiali-8c9d6fbf6-9hp75                     1/1     Running     0          17m
prometheus-7d7b9f7844-p79gv               1/1     Running     0          17m


# kubectl get svc -n istio-system
NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                          AGE
grafana                     ClusterIP      10.110.104.169   <none>        3000/TCP                                                                                          18m
istio-citadel               ClusterIP      10.105.113.216   <none>        8060/TCP,15014/TCP                                                                                          18m
istio-egressgateway         ClusterIP      10.98.158.85     <none>        80/TCP,443/TCP,15443/TCP                                                                                          18m
istio-galley                ClusterIP      10.103.48.216    <none>        443/TCP,15014/TCP,9901/TCP                                                                                          18m
istio-ingressgateway        LoadBalancer   10.110.13.142    <pending>     15020:32111/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32114/TCP,15030:30450/TCP,15031:31750/TCP,15032:30788/TCP,15443:32083/TCP   18m
istio-pilot                 ClusterIP      10.111.22.59     <none>        15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                          18m
istio-policy                ClusterIP      10.102.139.26    <none>        9091/TCP,15004/TCP,15014/TCP                                                                                          18m
istio-sidecar-injector      ClusterIP      10.102.102.11    <none>        443/TCP,15014/TCP                                                                                          18m
istio-telemetry             ClusterIP      10.106.211.184   <none>        9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                          18m
jaeger-agent                ClusterIP      None             <none>        5775/UDP,6831/UDP,6832/UDP                                                                                          17m
jaeger-collector            ClusterIP      10.98.254.9      <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP                                                                                          25h
jaeger-collector-headless   ClusterIP      None             <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP                                                                                          25h
jaeger-operator             ClusterIP      10.109.247.90    <none>        8383/TCP                                                                                          26h
jaeger-query                ClusterIP      10.111.157.104   <none>        16686/TCP                                                                                          25h
kiali                       ClusterIP      10.107.66.48     <none>        20001/TCP                                                                                          18m
prometheus                  ClusterIP      10.98.18.46      <none>        9090/TCP                                                                                          18m
tracing                     ClusterIP      10.97.13.237     <none>        80/TCP                                                                                          17m
zipkin                      ClusterIP      10.109.176.69    <none>        9411/TCP                                                                                          17m

 

6.5 operational components

grafana

Grafana supports many different time series database data sources. Grafana provides different query methods for each data source, and can well support the characteristics of each data source.

istio-citadel

Components that implement the Citadel function of istio

istio-egressgateway

The agent that controls internal access to external links. If no rules are set here, Envoy will abandon the access request because it cannot recognize the access rules.

 

istio-ingressgateway

The policy of controlling external access to internal data. Only after configuration can external services (outside the kubernetes cluster) access services in the cluster.

 

istio-pilot

pilot component of ISTIO

 

istio-policy

Policy control component of ISTIO

 

istio-sidecar-injector

Provide automatic injection function, so that yaml files do not need to be manually patched and injected

 

istio-telemetry

ISTIO is a component that implements telemetry data

 

prometheus

A monitoring data collection tool to replace heapster

 

servicegraph

The services in the Istio grid generate a graph. In this task, you will install the Servicegraph plug-in and use the web-based interface to view the service graph of the service grid.

 

 

 

tracing,zipkin

Service track tracking, and business visits can be recorded at the same time

 

 

7. Bookinfo example

https://www.qikqiak.com/istio-book/install/1.Docker%20for%20Mac%E5%AE%89%E8%A3%85istio.html

https://blog.51cto.com/brucewang/2149104 https://blog.csdn.net/M2l0ZgSsVc7r69eFdTj/article/details/81571517

 

At present, most of the usage examples of istio are official Bookinfo application examples:

 

7.1 the bookinfo application is divided into four separate microservices:

  • Productpage: the productpage microservice will call two microservices, details and reviews, to generate pages.
  • details: this micro service contains information about books.
  • Reviews: this micro service contains book related reviews. It also calls the ratings microservice.
  • Ratings: ratings micro service contains rating information composed of book reviews.

There are three versions of reviews micro service:

  • The v1 version does not call the ratings service.
  • v2 version will call the ratings service and use 1 to 5 Black Star icons to display the rating information.
  • The v3 version will call the ratings service and use 1 to 5 Red Star icons to display the rating information

     

 

 

 

 

This example is a multilingual microservice application. First, there is a ProductPage portal application implemented in python to display the details and evaluation of books. It will call the Detail application implemented in Ruby to obtain the details of books, and call the evaluation application implemented in Java to obtain the evaluation of books.

The figure above shows the actual structure of the whole application after using istio. All micro services are encapsulated with an Envoy sidecar, which intercepts all requests to arrive and leave the service.

7.2 deployment

First enter the decompressed istio directory and execute the following command:

 

kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml)

 

 

[root@k8s-master istio-1.3.4]# ls
bin  install  istio.VERSION  LICENSE  README.md  samples  tools
[root@k8s-master istio-1.3.4]#  kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml)
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

 

Including bookinfo Yaml is the yaml file of k8s Deployment and Service, and istioctl Kube inject will add content to the Deployment based on this file. Usually, an initContainer(image:docker.io/istio/proxy_init) will be added as a sidecar and run in the same pod as the application container.

After a while, you can see the following service and pod startup:

# kubectl get pod
NAME                                      READY   STATUS    RESTARTS   AGE
details-v1-5557659779-hlr52               2/2     Running   0          6m28s
productpage-v1-58f967648c-vhtxx           2/2     Running   0          6m27s
ratings-v1-68c58c7975-6fbkm               2/2     Running   0          6m27s
reviews-v1-9cd479947-p44ln                2/2     Running   0          6m26s
reviews-v2-567f5dcdfb-4klvt               2/2     Running   0          6m27s
reviews-v3-86bdc9c869-24ppb               2/2     Running   0          6m27s

# kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.109.59.71    <none>        9080/TCP   6m40s
productpage   ClusterIP   10.106.157.87   <none>        9080/TCP   6m39s
ratings       ClusterIP   10.107.134.22   <none>        9080/TCP   6m40s
reviews       ClusterIP   10.105.11.68    <none>        9080/TCP   6m39s

 

 

Now that the application services have been successfully deployed and started, you need to access them outside the cluster and add an istio gateway. Gateway is equivalent to k8s's ingress controller and ingress. It configures a load balancer for HTTP/TCP traffic, which is usually managed at the edge of the service grid as an ingress trafic of the application.

Then create a gateway:

1 kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

 

 

# kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io/bookinfo-gateway created
The VirtualService "bookinfo" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"networking.istio.io/v1alpha3", "kind":"VirtualService", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"bookinfo\",\"namespace\":\"default\"},\"spec\":{\"gateways\":[\"bookinfo-gateway\"],\"hosts\":[\"*\"],\"http\":[{\"match\":[{\"uri\":{\"exact\":\"/productpage\"}},{\"uri\":{\"prefix\":\"/static\"}},{\"uri\":{\"exact\":\"/login\"}},{\"uri\":{\"exact\":\"/logout\"}},{\"uri\":{\"prefix\":\"/api/v1/products\"}}],\"route\":[{\"destination\":{\"host\":\"productpage\",\"port\":{\"number\":9080}}}]}]}}\n"}, "creationTimestamp":"2019-11-08T03:23:26Z", "generation":1, "name":"bookinfo", "namespace":"default", "uid":"819111f5-99ec-4db4-8182-dcbbb46e1f93"}, "spec":map[string]interface {}{"gateways":[]interface {}{"bookinfo-gateway"}, "hosts":[]interface {}{"*"}, "http":[]interface {}{map[string]interface {}{"match":[]interface {}{map[string]interface {}{"uri":map[string]interface {}{"exact":"/productpage"}}, map[string]interface {}{"uri":map[string]interface {}{"prefix":"/static"}}, map[string]interface {}{"uri":map[string]interface {}{"exact":"/login"}}, map[string]interface {}{"uri":map[string]interface {}{"exact":"/logout"}}, map[string]interface {}{"uri":map[string]interface {}{"prefix":"/api/v1/products"}}}, "route":[]interface {}{map[string]interface {}{"destination":map[string]interface {}{"host":"productpage", "port":map[string]interface {}{"number":9080}}}}}}}}: validation failure list:
spec.http.route.weight in body is required

 

I'm at samples / bookinfo / Networking / bookinfo gateway Yaml added a

      weight: 1

 

 

 

[root@k8s-master istio-1.3.1]# vim  samples/bookinfo/networking/bookinfo-gateway.yaml
[root@k8s-master istio-1.3.1]# kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io/bookinfo-gateway unchanged
virtualservice.networking.istio.io/bookinfo created

#Verify that the gateway is started successfully:

[root@k8s-master istio-1.3.4]# istioctl get gateway
Command "get" is deprecated, Use `kubectl get` instead (see https://kubernetes.io/docs/tasks/tools/install-kubectl)
GATEWAY NAME       HOSTS     NAMESPACE   AGE
bookinfo-gateway   *         default     1m
[root@k8s-master istio-1.3.4]# kubectl get gateway
NAME               AGE
bookinfo-gateway   2m
[root@k8s-master istio-1.3.4]#

 

 

In the official document, we will get the ip and port of ingress later. In fact, we don't need it. In the service, we can see such a service information:

 

# kubectl get svc -n istio-system | grep istio-ingressgateway
istio-ingressgateway        LoadBalancer   10.110.13.142    <pending>     15020:32111/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32114/TCP,15030:30450/TCP,15031:31750/TCP,15032:30788/TCP,15443:32083/TCP   19h

 

The LoadBalancer type service used here is actually used to connect with the cloud service provider. If we do not connect with the cloud service provider, we can change the type here to nodePort. However, if we want to access our service, we have to add nodePort port. In fact, we can access Bookinfo directly by using localhost and open the address in the browser: http://localhost/productpage

It can also be verified by the following methods

#Determine the IP and port of Ingress:

#port: 
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
# secure port: 
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')
#host
export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o 'jsonpath={.items[0].status.hostIP}')
# GATEWAY_URL: 
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
#Test 200 is normal
curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage

 

 

7.3 test whether the deployment is successful

You can use url200 above

You can also view it through the page

The Bookinfo application consists of four micro services, namely the web micro service page productpage. The left part of the page is the Book Details service, and the right part is the Book Reviews service. The reviews service is currently in v1 status, that is, there is no star rating. Since the request route is not set, the page will be refreshed several times, and the request route traffic will switch randomly in the reviews services v1, v2 and v3.

 

 

 

 

 


If you think reading this article is helpful to you, please click the "recommend" button. Your "recommend" will be my biggest writing motivation! You are welcome to reprint, but without the consent of the author, you must give the connection between the author and the original text on the obvious position of the article page after reprinting the article, otherwise you reserve the right to investigate legal responsibility.
 
 

Posted by tanju on Sat, 07 May 2022 22:32:34 +0300