New in Kubernetes: programming in yaml

Author | Wu Peng

Introduction

Performance testing is a routine requirement in daily development work to find out the performance of services.

So how to do performance testing? Either it is done by coding, writing a bunch of scripts, and throwing it away after use; or it is based on the platform, and it is carried out in the process defined by the platform. For the latter, usually due to the complexity of target scenarios, such as deploying specific workload s, observing specific performance items, network access problems, etc., performance testing platforms often require high costs to meet the needs of changing development scenarios.

Could this kind of problem be better solved in a cloud-native context?

First look at the two yaml files:

  • performance-test.yaml describes the operation flow in K8s:

    1. Create a Namespace for testing
    2. Start monitoring of Deployment creation efficiency and creation success rate
    3. Repeat the following actions N times: ① Create a Deployment using the workload template; ② Wait for the Deployment to become Ready
    4. Delete the Namespace for testing
  • basic-1-pod-deployment.yaml describes the workload template used

performance-test.yaml :

apiVersion: aliyun.com/v1alpha1
kind: Beidou
metadata:
  name: performance
  namespace: beidou
spec:
  steps:
  - name: "Create Namespace If Not Exits"
    operations:
    - name: "create namespace"
      type: Task
      op: CreateNamespace
      args:
      - name: NS
        value: beidou
  - name: "Monitor Deployment Creation Efficiency"
    operations:
    - name: "Begin To Monitor Deployment Creation Efficiency"
      type: Task
      op: DeploymentCreationEfficiency
      args:
      - name: NS
        value: beidou
    - name: "Repeat 1 Times"
      type: Task
      op: RepeatNTimes
      args:
      - name: TIMES
        value: "1"
      - name: ACTION
        reference:
          id: deployment-operation
  - name: "Delete namespace"
    operations:
    - name: "delete namespace"
      type: Task
      op: DeleteNamespace
      args:
      - name: NS
        value: beidou
      - name: FORCE
        value: "false"
  references:
  - id: deployment-operation
    steps:
    - name: "Prepare Deployment"
      operations:
      - name: "Prepare Deployment"
        type: Task
        op: PrepareBatchDeployments
        args:
        - name: NS
          value: beidou
        - name: NODE_TYPE
          value: ebm
        - name: BATCH_NUM
          value: "1"
        - name: TEMPLATE
          value: "./templates/basic-1-pod-deployment.yaml"
        - name: DEPLOYMENT_REPLICAS
          value: "1"
        - name: DEPLOYMENT_PREFIX
          value: "ebm"
      - name: "Wait For Deployments To Be Ready"
        type: Task
        op: WaitForBatchDeploymentsReady
        args:
        - name: NS
          value: beidou
        - name: TIMEOUT
          value: "3m"
        - name: CHECK_INTERVAL
          value: "2s"

basic-1-pod-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: basic-1-pod
spec:
  selector:
    matchLabels:
      app: basic-1-pod
  template:
    metadata:
      labels:
        app: basic-1-pod
    spec:
      containers:
      - name: nginx
        image: registry-vpc.cn-hangzhou.aliyuncs.com/xxx/nginx:1.17.9
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 2
            memory: 4Gi

Then execute performance-test.yaml via a command line tool:

$ beidou server -c ~/.kube/config services/performance-test.yaml

The execution effect is as follows (the time-consuming creation of each Deployment, the TP95 value of the time-consuming creation of all Deployments, and whether each Deployment is successfully created):

These metrics are output according to the Prometheus standard, which can be collected by the Prometheus server, and combined with Grafana to visualize the performance test data.

By expressing ideas in yaml, arranging the operation and monitoring of K8s resources, you no longer have to worry about the implementation of performance testing 😄

Why program in yaml?

Performance testing, regression testing, etc. are very helpful for service quality assurance and need to be done, but the conventional implementation method requires a lot of time and energy in the initial stage, and the maintenance cost after new changes is relatively high.

Generally, this process implements atomic operations in the form of code, such as creating Deployment, detecting Pod configuration, etc., and then combines atomic operations to meet requirements, such as creating Deployment - > waiting for Deployment ready - > detecting Pod configuration, etc

Is there a way to implement it at a low cost and reuse the existing experience in the process of implementation?

Atomic operations can be encapsulated as primitives, such as createdeployment and checkpod, and then the process can be expressed through the yaml structure. Then ideas can be described through yaml instead of code, and yaml files written by others can be reused to solve certain needs of the scene

That is, programming in yaml, reducing repetitive code work, describing logic in a declarative way, and using yaml files to meet scene-level reuse.

There are many types of declarative operation services in the industry, such as in the field of operation and maintenance. Ansible,SaltStack , in Kubernetes Argo Workflow,clusterloader2. Their overall ideas are similar, encapsulating frequently used operations into primitives, and users express operational logic through primitives.

Through a declarative method, K8s-oriented operations are abstracted into keywords in yaml, and control logic such as serial and parallel is provided in yaml, then the work you want to do can be completely described through the yaml file.

this idea and Argo Workflow Similar, but with a finer granularity than Argo, focusing on operation functions:

The design and implementation of the service are briefly described below.

design and implementation

1. Service form

  • The user describes the operation logic in a declarative way in yaml;
  • Delivered as an all-in-one binary tool or Operator;
  • The implementation of common primitives built into the service is provided in yaml in the form of keywords;
  • Supports configuring native K8s resources.

2. Design

The core of this solution lies in the design of configuration management, which configures the operation process, and has the following concepts from top to bottom:

  • Service: Orchestration of Modules or Tasks;

  • Module: A task scenario, which is a collection of operation units (including the templates/ directory, which represents a collection of template files, which can be used to configure K8s native resources);

  • Task: operation unit, use plugin and parameters to perform operations;

  • Plugin: Operation instructions, similar to functions in development languages.

Common operations in the abstract target scene, these common operations are primitives that can be used in yaml, corresponding to the above Plugin:

  • K8s related

    • CreateNamespace
    • DeleteNamespace
    • PrepareSecret
    • PrepareConfigMap
    • PrepareBatchDeployments
    • WaitForBatchDeploymentsReady
    • etc.
  • observational correlation

    • DeploymentCreationEfficiency
    • PodCreationEfficiency
    • etc.
  • Detection item related

    • CheckPodAnnotations
    • CheckPodObjectInfo
    • CheckPodInnerStates
    • etc.
  • control statement

    • RepeatNTimes
    • etc.

The relationship between the above four concepts is as follows:

An example can be found in the yaml file at the beginning of the article, corresponding to form two.

3. Core Implementation

CRD Design:

package v1alpha1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// BeidouType is the type related to Beidou execution.
type BeidouType string

const (
	// BeidouTask represents the Task execution type.
	BeidouTask BeidouType = "Task"
)

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Beidou represents a crd used to describe serices.
type Beidou struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

	Spec   BeidouSpec   `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
	Status BeidouStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

// BeidouSpec is the spec of a Beidou.
type BeidouSpec struct {
	Steps      []BeidouStep      `json:"steps" protobuf:"bytes,1,opt,name=steps"`
	References []BeidouReference `json:"references" protobuf:"bytes,2,opt,name=references"`
}

// BeidouStep is the spec of step.
type BeidouStep struct {
	Name       string            `json:"name" protobuf:"bytes,1,opt,name=name"`
	Operations []BeidouOperation `json:"operations" protobuf:"bytes,2,opt,name=operations"`
}

// BeidouOperation is the spec of operation.
type BeidouOperation struct {
	Name string      `json:"name" protobuf:"bytes,1,opt,name=name"`
	Type BeidouType  `json:"type" protobuf:"bytes,2,opt,name=type"`
	Op   string      `json:"op" protobuf:"bytes,3,opt,name=op"`
	Args []BeidouArg `json:"args" protobuf:"bytes,4,opt,name=args"`
}

// BeidouArg is the spec of arg.
type BeidouArg struct {
	Name        string                   `json:"name" protobuf:"bytes,1,opt,name=name"`
	Value       string                   `json:"value,omitempty" protobuf:"bytes,2,opt,name=value"`
	Reference   BeidouOperationReference `json:"reference,omitempty" protobuf:"bytes,3,opt,name=reference"`
	Tolerations []corev1.Toleration      `json:"tolerations,omitempty" protobuf:"bytes,4,opt,name=tolerations"`
	Checking    []string                 `json:"checking,omitempty" protobuf:"bytes,5,opt,name=checking"`
}

// BeidouOperationReference is the spec of operation reference.
type BeidouOperationReference struct {
	ID string `json:"id" protobuf:"bytes,1,opt,name=id"`
}

// BeidouReference is the spec of reference.
type BeidouReference struct {
	ID    string       `json:"id" protobuf:"bytes,1,opt,name=id"`
	Steps []BeidouStep `json:"steps" protobuf:"bytes,2,opt,name=steps"`
}

// BeidouStatus represents the current state of a Beidou.
type BeidouStatus struct {
	Message string `json:"message" protobuf:"bytes,1,opt,name=message"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// BeidouList is a collection of Beidou.
type BeidouList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata" protobuf:"bytes,1,opt,name=metadata"`

	Items []Beidou `json:"items" protobuf:"bytes,2,opt,name=items"`
}

Core process:

// ExecSteps executes steps.
func ExecSteps(ctx context.Context, steps []v1alpha1.BeidouStep, references []v1alpha1.BeidouReference) error {
    logger, _ := ctx.Value(CtxLogger).(*log.Entry)

	var hasMonitored bool
	for i, step := range steps {
		for j, op := range step.Operations {
			switch op.Op {
			case "DeploymentCreationEfficiency":
				if !hasMonitored {
					defer func() {
						err := monitor.Output()
						if err != nil {
							logger.Errorf("Failed to output: %s", err)
						}
					}()
				}
				hasMonitored = true
			}

			err := ExecOperation(ctx, op, references)
			if err != nil {
				return fmt.Errorf("failed to run operation %s: %s", op.Name, err)
			}
		}
	}

	return nil
}

// ExecOperation executes operation.
func ExecOperation(ctx context.Context, op v1alpha1.BeidouOperation, references []v1alpha1.BeidouReference) error {
	switch op.Type {
	case v1alpha1.BeidouTask:
		if !tasks.IsRegistered(op.Op) {
			return ErrNotRegistered
		}

		if !tasks.DoesSupportReference(op.Op) {
			return ExecTask(ctx, op.Op, op.Args)
		}

		return ExecTaskWithRefer(ctx, op.Op, op.Args, references)
	}

	return nil
}

// ExecTask executes a task.
func ExecTask(ctx context.Context, opname string, args []v1alpha1.BeidouArg) error {
	switch opname {
	case tasks.CreateNamespace:
		var ns string
		for _, arg := range args {
			switch arg.Name {
			case "NS":
				ns = arg.Value
			}
		}

		return op.CreateNamespace(ctx, ns)
    // ...
    }
    // ...
}

// ExecTaskWithRefer executes a task with reference.
func ExecTaskWithRefer(ctx context.Context, opname string, args []v1alpha1.BeidouArg, references []v1alpha1.BeidouReference) error {
	switch opname {
	case tasks.RepeatNTimes:
		var times int
		var steps []v1alpha1.BeidouStep
		var err error
		for _, arg := range args {
			switch arg.Name {
			case "TIMES":
				times, err = strconv.Atoi(arg.Value)
				if err != nil {
					return ErrParseArgs
				}
			case "ACTION":
				for _, refer := range references {
					if refer.ID == arg.Reference.ID {
						steps = refer.Steps
						break
					}
				}
			}
		}

		return RepeatNTimes(ctx, times, steps)
	}

	return ErrNotImplemented
}

An example implementation of an operation primitive:

// PodAnnotations is an operation used to check whether annotations of Pod are expected.
func PodAnnotations(ctx context.Context, data PodAnnotationsData) error {
	kclient, ok := ctx.Value(tasks.KubernetesClient).(kubernetes.Interface)
	if !ok {
		return tasks.ErrNoKubernetesClient
	}

	pods, err := kclient.CoreV1().Pods(data.Namespace).List(metav1.ListOptions{})
	if err != nil {
		return fmt.Errorf("failed to list pods in ns %s: %s", data.Namespace, err)
	}

	for _, pod := range pods.Items {
		if pod.Annotations == nil {
			return fmt.Errorf("pod %s in ns %s has no annotations", pod.Name, data.Namespace)
		}

		for _, annotation := range data.Exists {
			if _, exists := pod.Annotations[annotation]; !exists {
				return fmt.Errorf("annotation %s does not exist in pod %s in ns %s", annotation, pod.Name, data.Namespace)
			}
		}

		for k, v := range data.Equal {
			if pod.Annotations[k] != v {
				return fmt.Errorf("value of annotation %s is not %s in pod %s in ns %s", k, v, pod.Name, data.Namespace)
			}
		}
	}

	return nil
}

follow-up

At present, the Alibaba Cloud Container Service team has implemented the first version internally, which has been used for internal performance testing of some cloud products and regular regression testing, which greatly improves our work efficiency.

Programming in yaml is an embodiment of declarative operations in cloud-native scenarios and a practice of declarative services. For repeated coding or repeated operations in routine work scenarios, similar methods can be considered for satisfaction.

You are welcome to discuss such service forms and projects and explore the value of this model.

Alibaba Cloud Container Service is continuously recruiting. Welcome to join us to explore the fields of K8s, edge computing, Serverless, etc., to make the present better and bring possibilities to the future! Contact Email: flyer.zyf@alibaba-inc.com

Spring Cloud Alibaba seven-day training camp

In seven days, I will understand the implementation principle of each module of microservices, teach how to independently develop a microservice application, and help Xiaobai developers to establish a systematic knowledge system from 0 to 1. Click on the link to register for the experience: https://developer.aliyun.com/learning/trainingcamp/spring/1

"Alibaba Cloud Native Focus on microservices, Serverless, containers, Service Mesh and other technical fields, focus on cloud-native popular technology trends, and cloud-native large-scale implementation practices, and be the official account of the most cloud-native developers. "

Tags: Kubernetes Cloud Native

Posted by almightyad on Thu, 30 Jun 2022 21:09:31 +0300