Infrastructure Prerequisites

Verify the following requirements across all nodes before deployment. All machines must reside within the same broadcast domain with full connectivity.

• Minimum two host machines
• Unique Hostnames, MAC addresses, and product UUIDs per host
• CentOS 7.9 or equivalent RHEL-based OS
• Minimum 2GB RAM per node
• Control-plane nodes require at least 2 CPU cores
• Swap disabled
• SELinux set to permissive or disabled
• Firewalld disabled or appropriately configured

Control-plane Port Requirements

Worker Node Port Requirements

System Configuration

Execute the following commands on all nodes to prepare the OS environment:

systemctl disable --now firewalld
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Configure time synchronization:

timedatectl set-ntp true

Container Runtime Setup

Install Docker CE on every node. Configure the systemd cgroup driver to align with Kubernetes requirements and set registry mirrors for faster image pulls.

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce-20.10.24 docker-ce-cli-20.10.24 containerd.io
systemctl enable --now docker

Deploy the Docker daemon configuration file:

cat > /etc/docker/daemon. <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  },
  "storage-driver": "overlay2",
  "registry-mirrors": [
    "https://mirror.gcr.io",
    "https://docker.mirrors.ustc.edu.cn"
  ]
}
EOF

systemctl daemon-reload
systemctl restart docker
</code>

Kubernetes Components Installation

Add the Kubernetes repository and install kubeadm, kubelet, and kubectl on all nodes.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubelet-1.22.17 kubeadm-1.22.17 kubectl-1.22.17 --disableexcludes=kubernetes
systemctl enable --now kubelet

Adjust kernel parameters required by Kubernetes:

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

Cluster Initialization

On the primary control-plane node, define the cluster configuration using a YAML file instead of command-line arguments for better maintainability. Ensure the endpoint DNS resolves correctly across all nodes.

export LOADBALANCER_IP=10.0.1.100
export API_ENDPOINT=k8s-api.mycorp.io
echo "${LOADBALANCER_IP} ${API_ENDPOINT}" >> /etc/hosts

cat <<EOF > kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.22.17
controlPlaneEndpoint: "${API_ENDPOINT}:6443"
networking:
  podSubnet: "172.16.0.0/12"
  serviceSubnet: "10.96.0.0/12"
imageRepository: registry.aliyuncs.com/google_containers
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
EOF

kubeadm init --config kubeadm-init.yaml --upload-certs

Configure kubectl access for the administrator:

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Pod Network Deployment

Deploy the Calico network plugin. Modify the default IP pool to match the podSubnet defined in the init configuration.

mkdir calico-manifests && cd calico-manifests
wget https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/calico.yaml
sed -i 's/192.168.0.0/172.16.0.0/g' calico.yaml
kubectl apply -f calico.yaml

Node Management

Adding Worker Nodes

On each worker node, configure the API endpoint resolution and execute the join command generated during initialization. If the token has expired, generate a new one on the control-plane node:

kubeadm token create --print-join-command

Execute the outputted join command on the worker nodes.

Adding Control-Plane Nodes

To achieve high availability, join additional control-plane nodes using the certificate key provided during initialization. At least three control-plane nodes are recommended to prevent etcd split-brain scenarios.

kubeadm join k8s-api.mycorp.io:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

Removing Nodes

Reset the node state and remove it from the cluster:

# On the departing node:
kubeadm reset -f

# On the control plane:
kubectl delete node 

KubeSphere Platform Deployment

Helm Setup

Install Helm 3 to manage Kubernetes package deployments:

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

NFS Storage Provisioning

Configure an NFS client provisioner to dynamically create Persistent Volumes. Create the deployment manifests, adjusting the NFS server IP and export path accordingly.

mkdir nfs-provisioner && cd nfs-provisioner
for f in class.yaml deployment.yaml rbac.yaml; do wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/$f; done

Update deployment.yaml with the target NFS server details:

env:
  - name: PROVISIONER_NAME
    value: nfs-provisioner.local
  - name: NFS_SERVER
    value: 10.0.1.104
  - name: NFS_PATH
    value: /srv/k8s-volumes
volumes:
  - name: nfs-client-root
    nfs:
      server: 10.0.1.104
      path: /srv/k8s-volumes

Apply the manifests and set the NFS storage class as the default:

kubectl apply -f rbac.yaml
kubectl apply -f class.yaml
kubectl apply -f deployment.yaml

kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

KubeSphere Installation

Deploy the full KubeSphere platform using the official installer manifest:

kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/kubesphere-complete-setup.yaml

Monitor the installation progress via the installer logs:

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o path='{.items[0].metadata.name}') -f

Upon successful deployment, the console outputs the access credentials:

#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://10.0.1.100:30880
Account: admin
Password: P@88w0rd