Virtual platform, distributed storage, high-availability k8s cluster environment construction of micro-service architecture series

1. Construction of physical machine virtual platform

1. Architecture conversion diagram from practical to required (each physical machine requires two hard disks for distributed storage and system disks, mine is an old machine 10 years ago and there is no big requirement):

 

2. System installation to Official website download Proxmox VE 7.x ISO Installer (Choose the version according to your needs) Download the promox image, download u disk creation tool , and finally select the DD mode in the pop-up box; if the machine is relatively new, you can choose the u disk creation or installation tool by yourself. (The u disk needs to be manually formatted for daily use: cmd-->diskpart-->list disk-->select 1 is subject to the actual disk number corresponding to u--->clean--->create partition primary--> active--->format fs=ntfs label="my u pan" quick--->assign)

3. The installation follows the interface step by step and there is nothing to pay attention to in the next step. It is best to have two disks for one machine, one for the system and one larger for use as a distributed storage. Just select the drive letter.

2. Establishment of distributed storage ceph environment (at least three machines must be installed on each)

1. nano /etc/apt/sources.list replace the following sources into the sources.list file and then save and exit (ctrl +o to save ---> enter to replace the same name ---> ctrl +x to exit)

#deb http://ftp.debian.org/debian bullseye main contrib
#deb http://ftp.debian.org/debian bullseye-updates main contrib
# security updates
#deb http://security.debian.org bullseye-security main contrib
# debian aliyun source
deb https://mirrors.aliyun.com/debian/ bullseye main non-free contrib
deb-src https://mirrors.aliyun.com/debian/ bullseye main non-free contrib
deb https://mirrors.aliyun.com/debian-security/ bullseye-security main
deb-src https://mirrors.aliyun.com/debian-security/ bullseye-security main
deb https://mirrors.aliyun.com/debian/ bullseye-updates main non-free contrib
deb-src https://mirrors.aliyun.com/debian/ bullseye-updates main non-free contrib
deb https://mirrors.aliyun.com/debian/ bullseye-backports main non-free contrib
deb-src https://mirrors.aliyun.com/debian/ bullseye-backports main non-free contrib
# proxmox source
# deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
deb https://mirrors.ustc.edu.cn/proxmox/debian/pve bullseye pve-no-subscription

2. nano /etc/apt/sources.list.d/pve-enterprise.list comment the original synchronization address save and exit

# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

3. nano /etc/apt/sources.list.d/ceph.list replaces the source of ceph Save and exit

deb http://mirrors.ustc.edu.cn/proxmox/debian/ceph-pacific bullseye main

4. Update source

root@masterxxx:~# apt-get update 
root@masterxxx:~# apt-get upgrade 
root@masterxxx:~# apt-get dist-upgrade 

5. Check whether the nano /etc/apt/sources.list.d/ceph.list file has changed, and repeat step 3 again if there is a change

6. Configure the Promox7.1 cluster. I created it by logging in to master001, and 002 and 003 join directly

7. Enter the ceph button under the cluster button and pop up a configuration button in the pop-up box. Click to configure the internal and external network ip and you will be done (I use master001 as the management node)

8. Then enter each ceph node (mine is also a Promox node) to check and bind the spare hard disk. My mine is /dev/sda, and check the binding according to the actual node.

root@masterxxx:~# fdisk -l
root@masterxxx:~# pveceph osd create /dev/sda

9. Create the pool-k8s and pool-vm pools of ceph for later use. If you change the size, click Advanced and change the pgs to the corresponding value. I created it directly by default, and finally uploaded the system image.

10. Install the basic system of k8s (hard disk and image are created and uploaded in the previous step, follow the prompts to set up step by step and wait for the creation completion point to run on the right to complete the basic system installation), and finally remove the CD-ROM drive (do not remove the migration exception) and convert to template

3. 3 masters and 3 slaves k8s cluster construction

1. Introduction to the environment

System: RockyLinux8.6 (4 cores 8G) * 6 units ----> the basic system runs at the same time, only 3.5G virtual memory is actually required

High availability middleware: nginx+KeepAlive (to achieve high availability of control nodes)

Network middleware: Calico

Time synchronization: chrony

Specific node detail diagram:

2. Schematic diagram of k8s installation My physical machine is in the 172.16.0 segment and the virtual machine is in the 172.16.1 segment. They communicate with each other, and the same ip segment can be set according to the actual situation

3. Template full clone (do not link clone) a virtual machine (for k8s basic environment)

3.1. Basic tool installation

[root@anonymous ~]# yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo  libaio-devel wget vim ncurses-devel autoconf automake zlib-devel  epel-release openssh-server socat ipvsadm conntrack telnet

3.2. Time synchronization configuration

[root@anonymous ~]# yum -y install chrony
[root@anonymous ~]# systemctl enable chronyd --now
[root@anonymous ~]# vim /etc/chrony.conf
 delete:
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
 In its original position, insert the domestic NTP server address
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp1.tencent.com iburst
server ntp2.tencent.com iburst
[root@anonymous ~]# systemctl restart chronyd

3.3. Turn off the firewall and swap partition

[root@anonymous ~]# systemctl stop firewalld ; systemctl disable firewalld
# Permanently shutting down SELINUX requires restarting the host to take effect
[root@anonymous ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
# Temporarily close SELINUX 
[root@anonymous ~]# setenforce 0
 
#temporarily close swap
[root@anonymous ~]# swapoff -a
#Permanent shutdown: comment swap mount /dev/mapper/centos-swap swap
[root@anonymous ~]# vim /etc/fstab

3.4. Modify kernel parameters

[root@anonymous ~]# modprobe br_netfilter
[root@anonymous ~]# lsmod | grep br_netfilter
 
[root@anonymous ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
 
[root@anonymous ~]# sysctl -p /etc/sysctl.d/k8s.conf

3.5, configure the installation source

# docker:
[root@anonymous ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# k8s:
[root@anonymous ~]# tee /etc/yum.repos.d/kubernetes.repo <<-'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF

3.6. Install k8s and container runtime (docker I chose)

#install k8s
[root@anonymous ~]# yum install docker-ce -y
[root@anonymous ~]# systemctl start docker && systemctl enable docker.service
 
[root@anonymous ~]# tee /etc/docker/daemon.json << 'EOF'
{
 "registry-mirrors":["https://vh3bm52y.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
} 
EOF
 
[root@anonymous ~]# systemctl daemon-reload
[root@anonymous ~]# systemctl restart docker
[root@anonymous ~]# systemctl enable docker
[root@anonymous ~]# systemctl status docker
 
# Install k8s components
[root@anonymous ~]# yum install -y kubelet-1.23.1 kubeadm-1.23.1 kubectl-1.23.1
[root@anonymous ~]# systemctl enable kubelet
 
# kubelet: A tool that runs on all nodes in the cluster and is used to start objects such as Pod s and containers
# kubeadm: a command tool for initializing the cluster and starting the cluster
# kubectl : The command line used to communicate with the cluster. Through kubectl, you can deploy and manage applications, view various resources, create, delete and update various components

3.7. Close the current virtual machine generation template and right-click --> Clone --> Full Clone --> Generate a total of 6 virtual machines until complete

3.8. Configure a fixed ip and host name for the cloned template machine

# Set a fixed ip (my network card is eth0, the specific network card is modified according to the actual situation)
[root@192 network-scripts]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
# Configuration reference x is filled according to the actual host ip
BOOTPROTO=static
IPADDR=172.16.1.X
NETMASK=255.255.255.0
GATEWAY=172.16.1.1
DNS1=172.16.1.1

# set hostname
[root@anonymous ~]# hostnamectl set-hostname master083 && bash
[root@master083 ~]# vim /etc/hosts
# New host resolution configuration
172.16.1.83 master083
172.16.1.78 master078
172.16.1.79 master079
172.16.1.80 worker080
172.16.1.81 worker081
172.16.1.82 worker082

4. Master node (master083, master078, master079) configuration

4.1. Select a virtual machine as the master node machine of the control node (I chose: 172.16.1.83 master083)

# Enter all the way without entering a password
[root@master083 ~]# ssh-keygen
# Install the local ssh public key file to the account corresponding to the remote host, yes enter the remote machine password
[root@master083 ~]# ssh-copy-id master078
[root@master083 ~]# ssh-copy-id master079
[root@master083 ~]# ssh-copy-id master083
[root@master083 ~]# ssh-copy-id worker080
[root@master083 ~]# ssh-copy-id worker081
[root@master083 ~]# ssh-copy-id worker082

4.2. Master node nginx+KeepAlive (3 master nodes must be installed)

[root@master083 ~]# yum install nginx keepalived nginx-mod-stream -y

4.3. Modify or replace nignx configuration (all 3 master nodes are changed, file address: /etc/nginx/nginx.conf)

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
stream {
    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;
    upstream k8s-apiserver {
       server 172.16.1.83:6443 weight=5 max_fails=3 fail_timeout=30s;
       server 172.16.1.78:6443 weight=5 max_fails=3 fail_timeout=30s;
       server 172.16.1.79:6443 weight=5 max_fails=3 fail_timeout=30s;
    }
 
    server {
       listen 16443; # Since nginx is multiplexed with the master node, the listening port cannot be 6443, otherwise it will conflict
       proxy_pass k8s-apiserver;
    }
}
 
http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    server {
        listen       80 default_server;
        server_name  _;
 
        location / {
        }
    }
}

4.4. Modify or configure KeepAlive (change all 3 master nodes, file address: /etc/keepalived/keepalived.conf)

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}
 
vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}
 
vrrp_instance VI_1 {
    state MASTER           # Use BACKUP for the backup server (multiple MASTERs will not be able to ping each other) I am 83 MASTER 
    interface ens18        # Change to the actual network card name
    virtual_router_id 83   # VRRP route ID instance, each instance is unique
    priority 100           # Priority, standby server setting 78 and 79 can be set 90/80 less than 100 priority is not the same
    advert_int 1           # Specify the VRRP heartbeat packet notification interval, the default is 1 second
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # Virtual IP
    virtual_ipaddress {
        172.16.1.83/24     # One of the three control nodes is exposed in a virtual unified manner, and the 12 in my intranet is not occupied, so choose according to the actual situation
    }
    track_script {
        check_nginx
    }
}

4.5, check_nginx.sh reference attention plus execution permission

#!/bin/bash
#1. Determine whether Nginx is alive
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    #2. If it does not survive, try to start Nginx
    service nginx start
    sleep 2
    #3. Wait for 2 seconds and get the Nginx status again
    counter=`ps -C nginx --no-header | wc -l`
    #4. Judge again, if Nginx is not alive, stop Keepalived and let the address drift
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi

4.6, 3 master nodes start nginx and KeepAlive

[root@master083 ~]# chmod +x /etc/keepalived/check_nginx.sh
[root@master083 ~]# systemctl daemon-reload
[root@master083 ~]# systemctl enable nginx keepalived
[root@master083 ~]# systemctl start nginx
[root@master083 ~]# systemctl start keepalived

4.7. I am creating a k8s cluster configuration file (I choose the keepalived master-->master083)

# yaml configuration reference
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.23.1
# The virtual host ip configured above
controlPlaneEndpoint: 172.16.1.46:16443
imageRepository: registry.aliyuncs.com/google_containers
apiServer:
  certSANs:
    - 172.16.1.86
    - 172.16.1.78
    - 172.16.1.79
    - 172.16.1.80
    - 172.16.1.81
    - 172.16.1.82
    - 172.16.1.46
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.10.0.0/16
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind:  KubeProxyConfiguration
mode: ipvs

4.8. Execute the configuration file to initialize the cluster (I chose the keepalived master-->master083)

# initialization
[root@master083 ~]# kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=SystemVerification
# Copy admin user configuration
[root@master083 ~]# mkdir -p $HOME/.kube
[root@master083 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master083 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# view node
[root@master083 ~]# kubectl get nodes

5. The remaining nodes are connected to the cluster

5.1. Control node access, take master079 as an example (just repeat this step for master078)

# Create a certificate storage directory
[root@master079 ~]# cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/
# copy certificate
[root@master083 ~]# scp /etc/kubernetes/pki/ca.crt master079 :/etc/kubernetes/pki/
[root@master083 ~]# scp /etc/kubernetes/pki/ca.key master079 :/etc/kubernetes/pki/
[root@master083 ~]# scp /etc/kubernetes/pki/sa.key master079 :/etc/kubernetes/pki/
[root@master083 ~]# scp /etc/kubernetes/pki/sa.pub master079 :/etc/kubernetes/pki/
[root@master083 ~]# scp /etc/kubernetes/pki/front-proxy-ca.crt master079 :/etc/kubernetes/pki/
[root@master083 ~]# scp /etc/kubernetes/pki/front-proxy-ca.key master079 :/etc/kubernetes/pki/
[root@master083 ~]# scp /etc/kubernetes/pki/etcd/ca.crt master079 :/etc/kubernetes/pki/etcd/
[root@master083 ~]# scp /etc/kubernetes/pki/etcd/ca.key master079 :/etc/kubernetes/pki/etcd/
# View the token and sha256 of the joining information
[root@master083 ~]# kubeadm token create --print-join-command
# xxx is replaced according to the information viewed in the previous command (system rollback may cause system time problems, you can manually refresh the system time: chronyc -a makestep)
[root@master079 ~]# kubeadm join 172.16.1.83:16443  --token xxx --discovery-token-ca-cert-hash sha256:xxx --control-plane --ignore-preflight-errors=SystemVerification
# copy copy admin
[root@master079 ~]# mkdir -p $HOME/.kube
[root@master079 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master079 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# View node information
[root@master079 ~]# kubectl get nodes

5.2. Work node node access

[root@worker080 ~]# kubeadm join 192.168.0.12:16443 --token *** --discovery-token-ca-cert-hash sha256:****** --ignore-preflight-errors=SystemVerification
[root@worker081 ~]# kubeadm join 192.168.0.12:16443 --token *** --discovery-token-ca-cert-hash sha256:****** --ignore-preflight-errors=SystemVerification
[root@worker082 ~]# kubeadm join 192.168.0.12:16443 --token *** --discovery-token-ca-cert-hash sha256:****** --ignore-preflight-errors=SystemVerification
# Label the working node
[root@master083 ~]# kubectl label node worker080 node-role.kubernetes.io/worker=worker
[root@master083 ~]# kubectl label node worker081 node-role.kubernetes.io/worker=worker
[root@master083 ~]# kubectl label node worker082 node-role.kubernetes.io/worker=worker
[root@master083 ~]# kubectl get nodes

6. Network plug-in installation and detection, and the basic deployment of k8s is completed

6.1. Install calico (calico.yaml download address: https://docs.projectcalico.org/manifests/calico.yaml ), which will involve the Google source causing the installation to fail and some custom configurations, calico file reference used in this article , if you are familiar with it, you can also configure it yourself

# Install
[root@master083 ~]# kubectl apply -f calico.yaml
# Wait for the automatic configuration to check whether the network takes effect
[root@master083 ~]# kubectl get nodes -owide
NAME        STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION              CONTAINER-RUNTIME
master078   Ready    control-plane,master   37h   v1.23.1   172.16.1.78   <none>        Rocky Linux 8.6 (Green Obsidian)   4.18.0-372.9.1.el8.x86_64   docker://20.10.21
master079   Ready    control-plane,master   37h   v1.23.1   172.16.1.79   <none>        Rocky Linux 8.6 (Green Obsidian)   4.18.0-372.9.1.el8.x86_64   docker://20.10.21
master083   Ready    control-plane,master   37h   v1.23.1   172.16.1.83   <none>        Rocky Linux 8.6 (Green Obsidian)   4.18.0-372.9.1.el8.x86_64   docker://20.10.21
worker080   Ready    worker                 37h   v1.23.1   172.16.1.80   <none>        Rocky Linux 8.6 (Green Obsidian)   4.18.0-372.9.1.el8.x86_64   docker://20.10.21
worker081   Ready    worker                 37h   v1.23.1   172.16.1.81   <none>        Rocky Linux 8.6 (Green Obsidian)   4.18.0-372.9.1.el8.x86_64   docker://20.10.21
worker082   Ready    worker                 37h   v1.23.1   172.16.1.82   <none>        Rocky Linux 8.6 (Green Obsidian)   4.18.0-372.9.1.el8.x86_64   docker://20.10.21

6.2. Create and enter the pod to check whether dns and external network are unobstructed

[root@master083 ~]# docker pull busybox:1.28

# External network detection
[root@master083 ~]# kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh
/ # ping www.baidu.com

# dns detection
[root@master083 ~]# kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh
/ # nslookup kubernetes.default.svc.cluster.local

7. Build a distributed storage environment for k8s (you can ignore this step if you use nfs shared storage)

7.1. Go to the ceph master node to create a distributed storage operation account kube

root@master001:~# ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=pool-k8s' -o ceph.client.kube.keyring

# 2. View the key s of the kube user and the admin user 
root@master001:~# ceph auth get-key client.admin
root@master001:~# ceph auth get-key client.kube

7.2. Go to k8s master node to associate ceph account information

# 1. Create a dev namespace
[root@master083 ~]# kubectl create ns dev
# 2. Create an admin secret to replace the value obtained by client.admin in step 7.2
[root@master083 ~]# kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" \
--from-literal=key='AQDSdZBjX15VFBAA+zJDZ8reSLCm2UAxtEW+Gw==' \
--namespace=kube-system
# 3. Create pvc in the dev namespace to access the secret of ceph to replace the value obtained by client.kube in step 7.2
[root@master083 ~]# kubectl create secret generic ceph-user-secret --type="kubernetes.io/rbd" \
--from-literal=key='AQCizZJjB19ADxAAmx0yYeL2QDJ5j3WsN/jyGA==' \
--namespace=dev 
# 4. View the secret
[root@master083 ~]# kubectl get secret ceph-user-secret -o yaml -n dev
[root@master083 ~]# kubectl get secret ceph-secret -o yaml -n kube-system

7.3. Create an account, which is mainly used to manage the right of ceph provisioner to run in the k8s cluster (rbac-ceph.yaml)

apiVersion: v1
kind: ServiceAccount  #Create an account, which is mainly used to manage the rights of ceph provisioner to run in the k8s cluster
metadata:
  name: rbd-provisioner
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns"]
    verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
subjects:
  - kind: ServiceAccount
    name: rbd-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: rbd-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rbd-provisioner
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rbd-provisioner
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbd-provisioner
subjects:
  - kind: ServiceAccount
    name: rbd-provisioner
    namespace: kube-system

7.4. Create a ceph provider (provisioner-ceph.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rbd-provisioner
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rbd-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
        - name: rbd-provisioner
          image: "quay.io/external_storage/rbd-provisioner:latest"
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: ceph-conf
              mountPath: /etc/ceph
          env:
            - name: PROVISIONER_NAME
              value: ceph.com/rbd
      serviceAccount: rbd-provisioner

      volumes:
        - name: ceph-conf
          hostPath:
            path: /etc/ceph

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
provisioner: ceph.com/rbd
parameters:
  monitors: 172.16.0.143:6789,172.16.0.211:6789,172.16.0.212:6789
  adminId: admin                      # k8s users accessing ceph
  adminSecretName: ceph-secret        # secret name
  adminSecretNamespace: kube-system   # secret plus namespace
  pool: pool-k8s       # ceph's rbd process pool
  userId: kube          # k8s users accessing ceph
  userSecretName: ceph-user-secret    # secret name, no need to add a namespace
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"
reclaimPolicy: Retain

7.5. Copy the files under /etc/ceph on the ceph server (mine is the physical machine master001) to each working node. The following is the copy structure reference of /etc/ceph of the worker081 working node

7.6. Perform initial deployment on k8s control node master083

[root@master083 ~]# kubectl apply -f rbac-ceph.yaml
[root@master083 ~]# kubectl apply -f provisioner-ceph.yaml
# View provider launch node information
[root@master083 ~]#kubectl get pod -n kube-system -owide
................                           ...     .......   ........        ...
kube-proxy-s5mfz                           1/1     Running   4 (22h ago)     37h     172.16.1.81      worker081   <none>           <none>
kube-proxy-tqksh                           1/1     Running   4               38h     172.16.1.79      master079   <none>           <none>
kube-proxy-w4h57                           1/1     Running   4 (22h ago)     37h     172.16.1.82      worker082   <none>           <none>
kube-scheduler-master078                   1/1     Running   30              38h     172.16.1.78      master078   <none>           <none>
kube-scheduler-master079                   1/1     Running   24              38h     172.16.1.79      master079   <none>           <none>
kube-scheduler-master083                   1/1     Running   18 (127m ago)   38h     172.16.1.83      master083   <none>           <none>
rbd-provisioner-579d59bb7b-ssd8b           1/1     Running   6               3h45m   10.244.129.207   worker081   <none>           <none>

7.7, Troubleshooting summary (here you can skip directly to see 7.8)

7.7.1, the first time the supplier is not Running, please go to the corresponding node to pull the image (I have to go to the node worker081)

[root@worker081 ~]# docker pull quay.io/external_storage/rbd-provisioner:latest

7.7.2. After copying, it is best to upgrade the ceph in the supplier

# into the container
[root@master083 ~]# kubectl exec -it rbd-provisioner-579d59bb7b-ssd8b -c rbd-provisioner -n kube-system -- sh
# The view version is too old to be compatible
sh-4.2# ceph -v
# Update container yum source
sh-4.2# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
# 2.3, configure ceph source
sh-4.2# cat >>/etc/yum.repos.d/ceph.repo<< eof
[ceph]
name=ceph
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/
gpgcheck=0
priority=1
enable=1
[ceph-noarch]
name=cephnoarch
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/
gpgcheck=0
priority=1
enable=1
[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS/
gpgcheck=0
priority=1
enable=1
eof
# renew
sh-4.2# yum -y update
# View installable versions
sh-4.2# yum list ceph-common --showduplicates | sort -r
ceph-common.x86_64                     2:14.2.9-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.8-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.7-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.6-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.5-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.4-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.3-0.el7                      ceph 
ceph-common.x86_64                     2:14.2.22-0.el7                     ceph 
ceph-common.x86_64                     2:14.2.22-0.el7                     @ceph
ceph-common.x86_64                     2:14.2.21-0.el7                     ceph 
ceph-common.x86_64                     2:14.2.20-0.el7                     ceph 
ceph-common.x86_64                     2:14.2.2-0.el7                      ceph 
..................                     ...............                     ....
# install latest version
sh-4.2# yum install -y ceph-common-14.2.21-0.el7
sh-4.2# ceph -v

# Enter the master081 server, view the docker container and recreate the image (9fb54e49f9bf is the id of the running container, not the id of the image) docker images and docker ps can manually find the image associated with the running container
[root@master081 ~]# sudo docker commit -m "update ceph-common 14.2.22 " -a "morik" 9fb54e49f9bf provisioner/ceph:14.2.22
[root@master081 ~]# docker save ceph_provisioner_14.2.22.tar.gz provisioner/ceph:14.2.22
# Go to 83 and 81 to delete the Deployment and docker images successively
[root@master083 ~]# kubectl delete Deployment rbd-provisioner -n rbd-provisioner 
[root@master081 ~]# docker rmi -f 9fb54e49f9bf
# Copy the newly generated ceph_provisioner_14.2.22.tar.gzdocker image to the working node and load it
[root@master080 ~]# docker load -i /home/ceph_provisioner_14.2.22.tar.gz
[root@master082 ~]# docker load -i /home/ceph_provisioner_14.2.22.tar.gz

7.8. Modify the image name (you can directly use the docker image , you can also generate it by yourself in the way of 7.6)

7.9. Test pvc (test-ceph.yaml)

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ceph-sc-claim
  namespace: dev
spec:
  storageClassName: ceph-rbd
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  resources:
    requests:
      storage: 500Mi

8.0, test to see the effect, the initial 12k ---> 38k after formatting

[root@master083 ~]# kubectl apply -f test-ceph.yaml
[root@master083 ~]# kubectl get pvc -n dev
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-sc-claim   Bound    pvc-406871be-069e-4ac1-84c9-ccc1589fd880   500Mi      RWO,ROX        ceph-rbd       4h35m
[root@master083 ~]# kubectl get pv -n dev
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-406871be-069e-4ac1-84c9-ccc1589fd880   500Mi      RWO,ROX        Retain           Bound    dev/ceph-sc-claim   ceph-rbd                4h35m
[root@master083 ~]# 

Tags: Kubernetes server architecture Microservices cluster

Posted by chapm4 on Sun, 11 Dec 2022 10:51:34 +0300