kubernetes 1.18 binary deployment

Chapter I pre knowledge points
1.1. Kubernetes cluster can be deployed in production environment in two ways
1.1.1 at present, there are two main ways to deploy Kubernetes cluster in production:
kubeadm

Kubeadm It's a K8s Deployment tools, providing kubeadm init and kubeadm join,For rapid deployment Kubernetes Cluster.

Official address: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

Binary package

from github Download the binary package of the distribution and manually deploy each component to form Kubernetes Cluster.

Kubeadm The deployment threshold is lowered, but many details are shielded, so it is difficult to troubleshoot problems. If you want to be more controllable, binary package deployment is recommended Kubernetes Although the manual deployment of clusters is troublesome, you can learn a lot of working principles during the period, which is also conducive to later maintenance.

1.2 installation requirements
Before starting, the deployment of Kubernetes cluster machines needs to meet the following conditions:
1.2.1 one or more machines, operating system centos7 x-86_ x64
1.2.2 hardware configuration: 2GB or more RAM, 2 CPUs or more CPUs, 30GB or more hard disk
1.2.3 network interworking between all machines in the cluster
1.2.4. You can access the external network and need to pull the image. If the server cannot access the Internet, you need to download the image in advance and import it into the node
1.2.5 prohibit swap partition
1. And 3 Preparation Environment
1.3.1 software environment:

Software edition
operating system CentOS7.6_x64
Docker 19-ce
Kubernetes 1.18

Note: considering that some friends have low computer configuration and so many virtual machines can't run, this set of high availability cluster is implemented in two parts. First deploy a single Master architecture (192.168.31.71 / 72 / 73), and then expand to multi Master architecture (the above plan). By the way, get familiar with the Master expansion process.
1.3.2. Single Master architecture:

1.3.3. Single Master server planning:

role ip assembly
k8s-master 192.168.31.71 kube-apiserver,kube-controller-manager,kube-scheduler,etcd
k8s-node1 192.168.31.72 kubelet,kube-proxy,docker etcd
k8s-node2 192.168.31.73 kubelet,kube-proxy,docker,etcd

1.4. Operating system initialization configuration

# Turn off the firewall
systemctl stop firewalld
systemctl disable firewalld

# Close selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config  # permanent
setenforce 0  # temporary

# Close swap
swapoff -a  # temporary
sed -ri 's/.*swap.*/#&/' /etc/fstab    # permanent

# Set the host name according to the plan
hostnamectl set-hostname <hostname>

# Add hosts in master
cat >> /etc/hosts << EOF
192.168.1.71 k8s-master
192.168.1.72 k8s-node1
192.168.1.73 k8s-node2
EOF

# The chain that passes bridged IPv4 traffic to iptables
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # take effect

# time synchronization 
yum install ntpdate -y
ntpdate time.windows.com

Chapter 2 deployment of Etcd cluster
Cluster can be used to set up a cluster. Of course, you can use etbercd to set up a cluster. You can also use etbercd to store data for a single machine. Of course, you can use etbercd to set up a cluster. You can also use etbercd to store data for a single machine.

Node name IP
etcd-1 192.168.31.71
etcd-2 192.168.31.72
etcd-3 192.168.31.73

Note: in order to save machines, it is reused with k8s node machines. It can also be deployed independently of the k8s cluster, as long as the apiserver can connect to it.
2.1. Prepare cfssl certificate generation tool
cfssl is an open source certificate management tool. It uses json files to generate certificates, which is more convenient to use than openssl.
Find any server to operate. Here, use the Master node.

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.2. Generate Etcd certificate
2.2.1 self signed certification authority (CA)
2.2.1.1. Create working directory:

mkdir -p ~/TLS/{etcd,k8s}
cd TLS/etcd

2.2.1.2 self signed CA:

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

2.2.1.3 certificate generation:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls *pem
ca-key.pem  ca.pem

2.2.2 issue Etcd HTTPS certificate with self signed CA
2.2.2.1. Create certificate application documents:

cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.31.71",
    "192.168.31.72",
    "192.168.31.73"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

Note: the IP in the hosts field of the above file is the internal communication IP of the cluster of all etcd nodes, and none of them can be less! In order to facilitate the later expansion, you can write more reserved IP addresses.
2.2.2.2. Generate certificate:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

ls server*pem
server-key.pem  server.pem

2.3. Download binary files from Github

Download address: https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

2.4. Deploy Etcd cluster
Note: the following operations are performed on node 1. To simplify the operation, all files generated by node 1 will be copied to node 2 and node 3 later
2.4.1. Create a working directory and unzip the binary package

mkdir /opt/etcd/{bin,cfg,ssl} -p
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

2.4.2. Create etcd configuration file

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.31.71:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.31.71:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.71:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.71:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.31.71:2380,etcd-2=https://192.168.31.72:2380,etcd-3=https://192.168.31.73:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME: node name, unique in the cluster
ETCD_DATA_DIR: Data Directory
ETCD_LISTEN_PEER_URLS: cluster communication listening address
ETCD_LISTEN_CLIENT_URLS: client access listening address
ETCD_INITIAL_ADVERTISE_PEER_URLS: cluster notification address
ETCD_ADVERTISE_CLIENT_URLS: client notification address
ETCD_INITIAL_CLUSTER: cluster node address
ETCD_INITIAL_CLUSTER_TOKEN: cluster token
ETCD_INITIAL_CLUSTER_STATE: the current state of joining a cluster. New is a new cluster, and existing means joining an existing cluster
2.4.3 systemd management etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

2.4.4 copy the certificate just generated
Copy the certificate just generated to the path in the configuration file

cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

2.4.5 start and set startup

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

2.4.6 copy all files generated by node 1 above to node 2 and node 3

scp -r /opt/etcd/ root@192.168.1.72:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.1.72:/usr/lib/systemd/system/
scp -r /opt/etcd/ root@192.168.1.73:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.1.73:/usr/lib/systemd/system/

Note: modify etcd at node 2 and node 3 respectively Node name and current server IP in conf configuration file:

vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1"   # Modify here to change node 2 to etcd-2 and node 3 to etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.71:2380 "# modify the IP address of the current server
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.71:2379 "# modify the IP address of the current server

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.71:2380 "# modify the IP address of the current server
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.71:2379 "# modify the IP address of the current server
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.71:2380,etcd-2=https://192.168.1.72:2380,etcd-3=https://192.168.1.73:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

Finally, start etcd and set startup, as above.
2.4.7. View cluster status

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.71:2379,https://192.168.1.72:2379,https://192.168.1.73:2379" endpoint health

https://192.168.31.71:2379 is healthy: successfully committed proposal: took = 8.154404ms
https://192.168.31.73:2379 is healthy: successfully committed proposal: took = 9.044117ms
https://192.168.31.72:2379 is healthy: successfully committed proposal: took = 10.000825ms

If the above information is output, the cluster deployment is successful. If there is a problem, the first step is to look at the log: / var/log/message or journalctl -u etcd
Chapter 3 installing Docker
Download address: https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
The following operations are performed on all nodes. Binary installation is adopted here, and so is yum installation.
3.1. Unzip binary package

tar zxvf docker-19.03.9.tgz
mv docker/* /usr/bin

3.2. systemd management docker

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

3.3. Create configuration file

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
registry-mirrors Alibaba cloud image accelerator

3.4. Start and set startup

systemctl daemon-reload
systemctl start docker
systemctl enable docker

Chapter 4 deployment of Master Node
4.1. Generate Kube apiserver certificate
4.1.1 self signed certification authority (CA)

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

4.1.1.1. Generate certificate:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

ls *pem
ca-key.pem  ca.pem

4.1.2 issue Kube apiserver HTTPS certificate with self signed CA
4.1.2.1 create certificate application documents:

cd TLS/k8s
cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.1.71",
      "192.168.1.72",
      "192.168.1.73",
      "192.168.1.74",
      "192.168.1.81",
      "192.168.1.82",
      "192.168.1.88",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

Note: the IP in the hosts field of the above file is all Master/LB/VIP IP, and none of them can be less! In order to facilitate the later expansion, you can write more reserved IP addresses.
4.1.2.2 certificate generation:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

ls server*pem
server-key.pem  server.pem

4.2. Download binary files from Github

Download address: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183

Note: open the link and you will find many packages in it. It is enough to download a server package, which contains the binary files of Master and Worker Node.
4.3 decompress binary package

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

4.4. Deploy Kube apiserver
4.4.1. Create configuration file

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.1.71:2379,https://192.168.1.72:2379,https://192.168.1.73:2379 \\
--bind-address=192.168.1.71 \\
--secure-port=6443 \\
--advertise-address=192.168.1.71 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

Note: the above two \ \, the first is the escape character and the second is the line feed character. The escape character is used to reserve the line feed character using EOF.

–logtostderr: log enable
—v: Log level
–log-dir: Log directory
–etcd-servers: etcd Cluster address
–bind-address: Listening address
–secure-port: https Secure port
–advertise-address: Cluster notification address
–allow-privileged: Enable authorization
–service-cluster-ip-range: Service fictitious IP Address segment
–enable-admission-plugins: Access control module
–authorization-mode: Authentication authorization, enabling RBAC Authorization and node self-management
–enable-bootstrap-token-auth: Enable TLS bootstrap mechanism
–token-auth-file: bootstrap token file
–service-node-port-range: Service nodeport Type default assignment port range
–kubelet-client-xxx: apiserver visit kubelet Client certificate
–tls-xxx-file: apiserver https certificate
–etcd-xxxfile: connect Etcd Cluster certificate
–audit-log-xxx: Audit log

4.4.2 copy the certificate just generated
4.4.2.1 copy the certificate just generated to the path in the configuration file:

cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

4.4.3 enable TLS Bootstrapping mechanism
TLS bootstrapping: after the Master apiserver enables TLS authentication, Node nodes kubelet and Kube proxy must use valid certificates issued by CA to communicate with Kube apiserver. When there are many Node nodes, this kind of client certificate issuance requires a lot of work, which will also increase the complexity of cluster expansion. In order to simplify the process, Kubernetes introduces TLS bootstrapping mechanism to automatically issue client certificates. Kubelet will automatically apply for certificates from apiserver as a low authority user, and kubelet's certificates are dynamically signed by apiserver. Therefore, it is strongly recommended to use this method on the Node. At present, it is mainly used for kubelet. Kube proxy is still issued by us.
TLS bootstrapping workflow:

4.4.3.1. Create the token file in the above configuration file:

cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

Format: token, user name, UID, user group
token can also be generated and replaced by itself:

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

4.4.4. systemd management apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

4.4.5 start and set startup

systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver

4.4.6 authorized kubelet bootstrap users are allowed to request certificates

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

4.5. Deploy Kube Controller Manager
4.5.1. Create configuration file

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF

– master: connect to apiserver through local non secure local port 8080.
– leader select: when the component starts multiple, automatic election (HA)
- cluster file signaling – the certificate issued by cluster file signaling is consistent with that issued by CA
4.5.2 systemd Controller Manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

4.5.3 start and set startup

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

4.6 deploy Kube scheduler
4.6.1. Create configuration file

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"
EOF

 – master: connect to apiserver through local non secure local port 8080.
 – leader select: when the component starts multiple, automatic election (HA)
4.6.2. systemd management scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

4.6.3 start and set startup

systemctl daemon-reload
systemctl start scheduler
systemctl enable scheduler

4.6., 4. View cluster status
All components have been started successfully. Check the current cluster component status through kubectl tool:

kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}

The above output indicates that the Master node component operates normally.
Chapter V deploying worker nodes
Note: the next step is to operate on the Master Node, that is, as a Worker Node at the same time
5.1. Create working directory and copy binary files
5.1.1. Create a working directory in all worker node s:

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

Copy from master node:

cd kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin   # Local copy

5.2. Deploy kubelet
5.2.1. Create configuration file

cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=k8s-master \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF

– hostname override: display name, unique in the cluster
– network plugin: enable CNI
– kubeconfig: empty path, which will be generated automatically, and later used to connect to apiserver
– bootstrap kubeconfig: apply for a certificate from apiserver for the first time
– config: configuration parameter file
– cert dir: kubelet certificate generation directory
– Pod infra container image: manage the image of Pod network container
5.2.2 configuration parameter file

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

5.2.3. Generate bootstrap Kubeconfig file

KUBE_APISERVER="https://192.168.1.71:6443" # apiserver IP:PORT
TOKEN="c47ffb939f5ca36231d9e3121a252940" # And token Consistent in CSV

# Generate kubelet bootstrap kubeconfig configuration file
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

Copy to profile path:

cp bootstrap.kubeconfig /opt/kubernetes/cfg

5.2.4 systemd management kubelet

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

5.2.5 start and set startup

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

5.3. Approve kubelet certificate application and join the cluster

# View kubelet certificate request
kubectl get csr
NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A   6m3s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# Approval application
kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A

# View node
kubectl get node
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   <none>   7s    v1.18.3

Note: because the network plug-in has not been deployed, the node will not be ready NotReady
5.4. Deploy Kube proxy
5.4.1. Create configuration file

cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

5.4.2 configuration parameter file

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master
clusterCIDR: 10.0.0.0/24
EOF

5.4.3. Generate Kube proxy Kubeconfig file
5.4.3.1 generate Kube proxy certificate:

# Switch working directory
cd TLS/k8s

# Create certificate request file
cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

# Generate certificate
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

ls kube-proxy*pem
kube-proxy-key.pem  kube-proxy.pem

5.4.3.2 generate kubeconfig file:

KUBE_APISERVER="https://192.168.1.71:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

Copy to the specified path of the configuration file:

cp kube-proxy.kubeconfig /opt/kubernetes/cfg/

5.4.4 systemd management Kube proxy

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

5.4.5 start and set startup

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

5.5. Deploy CNI network
5.5.1 prepare CNI binary files:

Download address: https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

5.5.1.1. Unzip the binary package and move it to the default working directory:

mkdir /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

5.5.1.2. Deploy CNI network:

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

Note: the default image address cannot be accessed. It is modified to docker hub image warehouse.

kubectl apply -f kube-flannel.yml

kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
kube-flannel-ds-amd64-2pc95   1/1     Running   0          72s

kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    <none>   41m   v1.18.3

Deploy the network plug-in and the Node is ready.
5.6. Authorize apiserver to access kubelet

cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f apiserver-to-kubelet-rbac.yaml

5.7. Newly added Worker Node
5.7.1 copy the deployed Node related files to the new Node
5.7.1.1 copy the files involved in the Worker Node to the new node 192.168.1.72/73 at the master node

scp /opt/kubernetes root@192.168.1.72:/opt/

scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.1.72:/usr/lib/systemd/system

scp -r /opt/cni/ root@192.168.1.72:/opt/

scp /opt/kubernetes/ssl/ca.pem root@192.168.1.72:/opt/kubernetes/ssl

5.7.2 delete kubelet certificate and kubeconfig file

rm /opt/kubernetes/cfg/kubelet.kubeconfig 
rm -f /opt/kubernetes/ssl/kubelet*

Note: these files are automatically generated after the certificate application is approved. Each Node is different and must be deleted and regenerated.
5.7.3. Modify host name

vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node1

vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node1

5.7.4 start and set startup

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

5.7.5 approve the application for new Node kubelet certificate on the Master

kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro   89s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

kubectl certificate approve node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro

5.7.6. View Node status

kubectl get node
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   Ready      <none>   65m   v1.18.3
k8s-node1    Ready      <none>   12m   v1.18.3
k8s-node2    Ready      <none>   81s   v1.18.3

Node2 (192.168.1.73) node is the same as above. Remember to modify the host name!
Chapter 6 deploying Dashboard and CoreDNS
6.1. Deploy Dashboard

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

The default Dashboard can only be accessed inside the cluster. Modify the Service to NodePort type and expose it to the outside:

vi recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

kubectl apply -f recommended.yaml
kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS              RESTARTS   AGE
pod/dashboard-metrics-scraper-694557449d-z8gfb   1/1     Running             0          2m18s
pod/kubernetes-dashboard-9774cc786-q2gsx         1/1     Running		     0          2m19s

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.141   <none>        8000/TCP        2m19s
service/kubernetes-dashboard        NodePort    10.0.0.239   <none>        443:30001/TCP   2m19s

Access address: https://NodeIP:30001
Create a service account and bind the default cluster admin administrator cluster role:

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

Log in to the Dashboard using the output token


6.2. Deploy CoreDNS
6.2.1. Download and rename coredns file

wget https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/coredns.yaml.sed
mv coredns.yaml.sed coredns.yaml
 Description: will coredns.yaml In the file $DNS_DOMAIN Replace with cluster.local  take $DNS_SERVER_IP Replace with 10.0.0.222

6.2.2. CoreDNS is used for Service name resolution within the cluster

kubectl apply -f coredns.yaml

kubectl get pods -n kube-system 
NAME                          READY   STATUS    RESTARTS   AGE
coredns-5ffbfd976d-j6shb      1/1     Running   0          32s
kube-flannel-ds-amd64-2pc95   1/1     Running   0          38m
kube-flannel-ds-amd64-7qhdx   1/1     Running   0          15m
kube-flannel-ds-amd64-99cr8   1/1     Running   0          26m

6.2.3 DNS resolution test:

kubectl run -it --rm dns-test --image=busybox:1.28.4 sh
If you don't see a command prompt, try pressing enter.

/ # nslookup kubernetes
Server:    10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local

No problem parsing.
So far, the deployment of a single Master cluster has been completed.

Posted by radman08 on Wed, 18 May 2022 05:32:23 +0300