Design of high availability cluster based on CentOS + docker + kubedm

🌟 preface

Hello, I'm Edison 😎

Today is the third in our "cloud native" series: "deploying k8s clusters";

Let's get it!


1. Installation requirements

Before starting, the deployment of Kubernetes cluster machines needs to meet the following conditions:

  • One or more machines, operating system centos7 8-86_ x64
  • Hardware configuration: 2GB or more RAM, 2 CPUs or more CPUs, 30GB or more hard disk
  • Network interworking between all machines in the cluster
  • You can access the Internet and need to pull the image
  • Disable swap partition

🍑 Node planning

Nodes deploying k8s clusters can be divided into the following two types of roles according to their purposes:
 
(1) Master: the master node of the cluster and the initialization node of the cluster. The basic configuration is no less than 2C4G
 
(2) Slave: the slave node of the cluster, which can be multiple, and the basic configuration is no less than 2C4G

This article demonstrates the addition of slave nodes. One master + two slave nodes will be deployed. The node planning is as follows:

🍑 Prepare the environment

2. Set hosts resolution

Operation node: all nodes (k8s master, k8s slave) need to execute

  • Modify hostname: hostname must contain only lowercase letters, numbers, ",", "-", and the beginning and end must be lowercase letters or numbers
# On the master node
$ hostnamectl set-hostname k8s-master #Set the hostname of the master node

# On the slave-1 node
$ hostnamectl set-hostname k8s-slave1 #Set the hostname of the slave1 node

# On the slave-2 node
$ hostnamectl set-hostname k8s-slave2 #Set the hostname of the slave2 node

  • Add hosts resolution
$ cat >>/etc/hosts<<EOF
172.21.51.143 k8s-master
172.21.51.67 k8s-slave1
172.21.51.68 k8s-slave2
EOF

🍑 Adjust system configuration

Operation node: all master and slave nodes (k8s master, k8s slave) need to be executed

The following operations take k8s master as an example, and other nodes are the same operations (the values of ip and hostname are replaced with the real values of the corresponding machine)

  • Set security group open port

If there is no security group restriction between nodes (any access between intranet machines), it can be ignored. Otherwise, at least the following ports can be connected:
 
K8s master node: TCP: 6443237923806008060081; Open all UDP protocol ports
 
K8s slave node: all UDP protocol ports are open

  • Set iptables
$ iptables -P FORWARD ACCEPT
  • Close swap
$ swapoff -a
# Prevent automatic mount of swap partition after startup

$ sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  • Modify kernel parameters
$ cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.max_map_count=262144
EOF

$ modprobe br_netfilter

$ sysctl -p /etc/sysctl.d/k8s.conf
  • Set yum source
$ curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo

$ curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

$ yum clean all && yum makecache 

🍑 Install docker

All nodes should be installed

## View all available versions
$ yum list docker-ce --showduplicates | sort -r

##Install old version Yum install docker-ce-cli-18.09.9-3.0 el7  docker-ce-18.09.9-3. el7

## Install the latest version in the source
$ yum install docker-ce-20.10.12 -y

## Configure docker acceleration
$ mkdir -p /etc/docker
vi /etc/docker/daemon.json
{
  "insecure-registries": [    
    "172.21.51.143:5000" 
  ],                          
  "registry-mirrors" : [
    "https://8xpk5wnt.mirror.aliyuncs.com "## use your own Alibaba image accelerator website
  ]
}
## Start docker
$ systemctl enable docker && systemctl start docker

3. Install kubedm, kubelet and kubectl

kubeadm is a tool launched by the official community for rapid deployment of kubernetes clusters.

Operation node: all master and slave nodes (k8s master, k8s slave) need to be executed

$ yum install -y kubelet-1.21.5 kubeadm-1.21.5 kubectl-1.21.5 --disableexcludes=kubernetes

## View kubedm version
$ kubeadm version

## Set kubelet startup
$ systemctl enable kubelet 

🍑 Initialize profile

Operation node: only in the master node (k8s-master)

$ kubeadm config print init-defaults > kubeadm.yaml

$ cat kubeadm.yaml ##Write yaml file
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.21.51.143
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.21.5
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

🍑 Download Image in advance

Operation node: only in the master node (k8s-master)

 # Check the list of images to be used. If there is no problem, you will get the following list
$ kubeadm config images list --config kubeadm.yaml
registry.aliyuncs.com/google_containers/kube-apiserver:v1.16.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.16.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.16.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.16.0
registry.aliyuncs.com/google_containers/pause:3.1
registry.aliyuncs.com/google_containers/etcd:3.3.15-0
registry.aliyuncs.com/google_containers/coredns:1.21.5

  # Download Image to local in advance
$ kubeadm config images pull --config kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.3.15-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.21.5

🍑 Initialize the master node

Operation node: only in the master node (k8s-master)

$ kubeadm init --config kubeadm.yaml

If the initialization is successful, the following information will be prompted at the end:

...
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.21.51.143:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:1c4305f032f4bf534f628c32f5039084f4b103c922ff71b12a5f0f98d1ca9a4f

Next, follow the above prompt information to configure the authentication of kubectl client

 $ mkdir -p $HOME/.kube
 
 $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 
 $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

🍑 Add slave nodes to the cluster

Operation node: all slave nodes (k8s slave) need to execute the following commands on each slave node;
 
This command is printed in the prompt message after kubedm init succeeds. It needs to be replaced with the command printed after the actual init.

$ kubeadm join 172.21.51.143:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:1c4305f032f4bf534f628c32f5039084f4b103c922ff71b12a5f0f98d1ca9a4f

If you forget to add a command, it can be generated by the following command:

$ kubeadm token create --print-join-command

4. Network plug-in

Operation node: only in the master node (k8s-master), CNI

  • Download the yaml file of flannel
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  • Modify the configuration, specify the network card name, and add a line of configuration in line 190 of the file:
$ vi kube-flannel.yml
...      
      containers:
      - name: kube-flannel
        image: rancher/mirrored-flannelcni-flannel:v0.16.1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=eth0  # If the machine has multiple network cards, specify the name of the intranet network card. If it is not specified by default, the first network card will be found
        resources:
          requests:
            cpu: "100m"
...
  • Execute and install the flannel network plug-in
# First pull the image, which is slow in C
$ docker pull rancher/mirrored-flannelcni-flannel:v0.16.1

# Perform flannel installation
$ kubectl apply -f kube-flannel.yml

5. Cluster settings

Set whether the master node is schedulable

Operation node: k8s master

After the default deployment is successful, the master node cannot schedule business pods. If you need to set the master node to participate in the scheduling of pods, you need to perform the following steps:

$ kubectl taint node k8s-master node-role.kubernetes.io/master:NoSchedule-

Set kubectl auto completion

$ yum install bash-completion -y

$ source /usr/share/bash-completion/bash_completion

$ source <(kubectl completion bash)

$ echo "source <(kubectl completion bash)" >> ~/.bashrc

6. The adjustment certificate has expired

For the cluster installed with kubedm, the default validity period of the certificate is 1 year, which can be modified to 10 years in the following ways.

$ cd /etc/kubernetes/pki

# View current certificate validity
$ for i in $(ls *.crt); do echo "===== $i ====="; openssl x509 -in $i -text -noout | grep -A 3 'Validity' ; done

$ mkdir backup_key; cp -rp ./* backup_key/

$ git clone https://github.com/yuyicai/update-kube-cert.git

$ cd update-kube-cert/ 

$ bash update-kubeadm-cert.sh all

# Reconstruction Management Services
$ kubectl -n kube-system delete po kube-apiserver-k8s-master kube-controller-manager-k8s-master kube-scheduler-k8s-master

7. Verify cluster

Operation node: executed at the master node (k8s-master)

$ kubectl get nodes  #Observe whether all cluster nodes are Ready

Create test nginx service

$ kubectl run  test-nginx --image=nginx:alpine

Check whether the pod is created successfully, and access the pod ip to test whether it is available

$ kubectl get po -o wide

NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
test-nginx-5bd8859b98-5nnnw   1/1     Running   0          9s    10.244.1.2   k8s-slave1   <none>           <none>

$ curl 10.244.1.2
...
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

8. Deploy dashboard

  • Deployment Services
$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml

$ vi recommended.yaml
# Modify the Service to NodePort type, up and down 45 lines of the file
......
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  type: NodePort  # Add type=NodePort to become a NodePort type service
......
  • Check the access address. This example is port 30133
$ kubectl apply -f recommended.yaml

$ kubectl -n kubernetes-dashboard get svc
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.105.62.124   <none>        8000/TCP        31m
kubernetes-dashboard        NodePort    10.103.74.46    <none>        443:30133/TCP   31m 
  • visit
  • Create ServiceAccount for access
$ vi dashboard-admin.conf
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kubernetes-dashboard

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kubernetes-dashboard

$ kubectl apply -f dashboard-admin.conf
$ kubectl -n kubernetes-dashboard get secret |grep admin-token
admin-token-fqdpf                  kubernetes.io/service-account-token   3      7m17s

# Use this command to get the token and paste it into the browser
$ kubectl -n kubernetes-dashboard get secret admin-token-fqdpf -o jsonpath={.data.token}|base64 -d

eyJhbGciOiJSUzI1NiIsImtpZCI6Ik1rb2xHWHMwbWFPMjJaRzhleGRqaExnVi1BLVNRc2txaEhETmVpRzlDeDQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi10b2tlbi1mcWRwZiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjYyNWMxNjJlLTQ1ZG...
  • Paste here
  • Successfully built

🍑 Clean up cluster

If we encounter other problems during cluster installation, we can use the following command to reset:

# Execute on all cluster nodes
kubeadm reset

ifconfig cni0 down && ip link delete cni0

ifconfig flannel.1 down && ip link delete flannel.1

rm -rf /run/flannel/subnet.env

rm -rf /var/lib/cni/

mv /etc/kubernetes/ /tmp

mv /var/lib/etcd /tmp

mv ~/.kube /tmp

iptables -F

iptables -t nat -F

ipvsadm -C

ip link del kube-ipvs0

ip link del dummy0

Tags: CentOS Docker Kubernetes cloud computing Cloud Native

Posted by airric on Mon, 18 Apr 2022 11:49:49 +0300