Building Kubernetes Clusters from Scratch
Infrastructure Planning
Prerequisites
-
Familiarity with Kubernetes concepts is essential. Setting up a temporary cluster using existing tutorials helps understand kubectl commands and core concepts like pods and services.
-
Install kubectl following the appropriate installation guide if not already present.
Infrastructure Components
Cloud Provider Integration
Kubernetes defines a Cloud Provider interface for managing TCP load balancers, instances, and routing. This interface is detailed in pkg/cloudprovider/cloud.go. Custom clusters can operate without implementing this interface, particularly for bare-metal deployments.
Node Requirements
- Virtual or physical machines均可 used
- Minimum of 4 nodes recommended for examples and testing
- Master and worker node distinction is optional
- x86_64 Linux systems required
- For clusters exceeding 10 nodes, API server and etcd can run on 1 CPU with 1GB RAM
- Heterogeneous node configurations are supported
Networking Architecture
Kubernetes assigns unique IP addresses to each pod. Network configuration involves selecting an IP address range for pod allocation. Two primary approaches exist:
Route-based Configuration:
- More complex implementation
- Direct IP route configuration for pod connectivity
- Optimal network performance
Overlay Networks:
- Simpler deployment
- Encapsulated traffic enables routable pod IPs
- Examples include Flannel, Weave, and Open vSwitch
- Performance varies based on encapsulation overhead
Select an appropriate IP range for pod addressing. For GCE, projects use "10.0.0.0/8" with clusters allocated "/16" subnets. AWS environments utilize VPCs for cluster isolation.
Configure CIDR ranges for each node's pod network. A "/24" provides 254 addresses, while smaller networks like "/26" (62 addresses) or "/27" (30 addresses) work for limited deployments.
Services receive IP addresses through the "SERVICE_CLUSTER_IP_RANGE" parameter. For instance, "10.0.0.0/16" supports 65534 concurrent services.
Configure a static master IP address (MASTER_IP), allow access to API server ports (80, 443), and enable IPv4 forwarding via sysctl net.ipv4.ip_forward=1.
Cluster Identification
Choose a unique, concise cluster name for multi-cluster management scenarios. This identifier (CLUSTERNAME) helps distinguish resources across different clusters.
Software Dependencies
Required packages include:
- etcd
- Container runtime (Docker or rkt)
- Kubernetes components: kubelet, kube-proxy, kube-apiserver, kube-controller-manager, kube-scheduler
Installation Process
Download the official Kubernetes release package containing all binaries and associated etcd. Extract kubernetes-server-linux-amd64.tar.gz and locate the binary folder at ./kubernetes/server/bin.
Container Image Strategy
Run Docker, kubelet, and kube-proxy as system services using direct binaries. Deploy etcd, kube-apiserver, kube-controller-manager, and kube-scheduler in containers using appropriate images.
Available image sources:
- Google Container Registry:
gcr.io/google_containers/hyperkube:$TAG - Private registries
- Self-built images using provided tar files
For etcd, options include GCR images, Docker Hub, Quay.io, OS packages, or custom builds. Use the version specified in kubernetes/cluster/images/etcd/Makefile under ETCD_VERSION.
Set environment variables:
HYPERKUBE_IMAGE=gcr.io/google_containers/hyperkube:$TAG
ETCD_IMAGE=gcr.io/google_containers/etcd:$ETCD_VERSION
Security Configuration
Two primary security models exist:
HTTP Access: Simpler implementation using firewall restrictions.
HTTPS Access: Recommended approach using certificates and authentication.
Certificate Generation
Prepare multiple certificates:
- Master server certificate
- Kubelet certificates (optional)
Use a root CA to sign certificates for master, kubelets, and kubectl clients. Store certificates at /srv/kubernetes/:
ca.crt(CA certificate)server.crt(master certificate)server.key(master private key)
Authentication Setup
Generate authentication tokens or passwords. Create a token file at /var/lib/kubeapiserver/known_tokens.csv following authentication documentation formats.
Configure kubeconfig files for administrators, kubelets, and kube-proxy. Different authentication strategies include:
- Shared credentials for all components
- Separate tokans for kubelets, kube-proxy, and administrators
- Individual accounts per component (not yet supported)
Node Configuraton
Each node runs three primary processes:
- Container runtime (Docker/rkt)
- kubelet
- kube-proxy
Docker Configuration
Ensure compatibility with minimum Docker versions. Clear existing Docker bridges and iptables rules:
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
Recommended Docker settings:
- Bridge configuration:
--bridge=cbr0 - Disable iptables:
--iptables=false - IP masquerading:
--ip-masq=false(for routable Pod IPs) - MTU setting (for Flannel):
--mtu= - Insecure registry:
--insecure-registry $CLUSTER_SUBNET
rkt Alternative
Supports rkt v0.5.6+ with systemd v215+. Enable rkt metadata service:
sudo systemd-run rkt metadata-service
Configure kubelet with --container-runtime=rkt.
Kubelet Configuration
Essential parameters include:
- API server connection (
--api-servers) - Configuration directory (
--config) - DNS settings (
--cluster-dns,--cluster-domain) - Network bridge configuration (
--configure-cbr0) - Node registration (
--register-node)
Kube-proxy Configuration
Configure similar to kubelet with appropriate API server endpoints and kubeconfig files.
Network Implementation
Assign CIDR ranges to each node (NODE_X_POD_CIDR). Create cbr0 bridges with the first IP as bridge address (NODE_X_BRIDGE_ADDR).
Automated approach uses --configure-cbr0=true with kubelet waiting for node controller assignment.
Manual approach involves:
- Create bridge:
brctl addbr cbr0 - Set MTU:
ip link set dev cbr0 mtu 1460 - Assign address:
ip addr add $NODE_X_BRIDGE_ADDR dev cbr0 - Activate:
ip link set dev cbr0 up
Configure IP masquerading for external traffic:
iptables -t nat -A POSTROUTING ! -d ${CLUSTER_SUBNET} -m addrtype ! --dst-type LOCAL -j MASQUERADE
Bootstrap Process
System services (kubelet, kube-proxy, container runtime) configured traditionally. Kubernetes components managed via pod specifications instead of init scripts.
etcd Deployment
Options include single instance with persistent storage or clustered configuration (3-5 nodes). Deploy using etcd manifest files in kubelet's manifest directory.
Core Services
Deploy apiserver, controller-manager, and scheduler as pods using hyperkube images. Configure appropriate command-line arguments and volume mounts.
Apiserver pod template:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-apiserver"
},
"spec": {
"hostNetwork": true,
"containers": [{
"name": "kube-apiserver",
"image": "${HYPERKUBE_IMAGE}",
"command": ["/hyperkube", "apiserver", "$ARG1", "$ARG2"],
"ports": [
{"name": "https", "hostPort": 443, "containerPort": 443},
{"name": "local", "hostPort": 8080, "containerPort": 8080}
],
"volumeMounts": [
{"name": "srvkube", "mountPath": "/srv/kubernetes", "readOnly": true},
{"name": "etcssl", "mountPath": "/etc/ssl", "readOnly": true}
]
}],
"volumes": [
{"name": "srvkube", "hostPath": {}},
{"name": "etcssl", "hostPath": {"path": "/etc/ssl"}}
]
}
}
Scheduler pod template:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {"name": "kube-scheduler"},
"spec": {
"hostNetwork": true,
"containers": [{
"name": "kube-scheduler",
"image": "$HYBERKUBE_IMAGE",
"command": ["/hyperkube", "scheduler", "--master=127.0.0.1:8080"]
}]
}
}
Controller Manager template:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {"name": "kube-controller-manager"},
"spec": {
"hostNetwork": true,
"containers": [{
"name": "kube-controller-manager",
"image": "$HYBERKUBE_IMAGE",
"command": ["/hyperkube", "controller-manager", "$CNTRLMNGR_FLAG1"],
"volumeMounts": [
{"name": "srvkube", "mountPath": "/srv/kubernetes", "readOnly": true},
{"name": "etcssl", "mountPath": "/etc/ssl", "readOnly": true}
]
}],
"volumes": [
{"name": "srvkube", "hostPath": {"path": "/srv/kubernetes"}},
{"name": "etcssl", "hostPath": {"path": "/etc/ssl"}}
]
}
}
Place completed manifests in kubelet's configuration directory (typically /etc/kubernetes/manifests). Verify pod status using docker ps or process monitoring tools.
Test API server connectivity:
curl -s http://localhost:8080/healthz
# Should return: ok
curl -s http://localhost:8080/api
# Should return: {"versions": ["v1"]}
Nodes automatical register with the API server when --register-node=true is enabled.
Troubleshooting
Verify cluster functionality using validation scripts. Check pod and service status. Ensure node-to-node connectivity using private IP addresses.
