Managing Topological State in Kubernetes with StatefulSet
Notes
1
Deployment controllers operate under the assumption that all replicas of an application are identical and interchangeable. This model works well for stateless workloads where any Pod can be terminated and replaced without consequence. However, distributed systems and data storage applications often require specific startup sequences, stable network identities, and persistent bindings to storage resources. These requirements define what Kubernetes calls "stateful applications."
StatefulSet addresses these challenges by abstracting application state into two categories:
1. Topological State: Application instances have defined relationships and dependencies. A master node must initialize before its replicas. If both Pods are deleted and recreated, the startup order must remain consistent, and the new Pods must retain their original network identifiers.
2. Storage State: Each instance maintains a unique binding to its storage data. A database Pod accessing data at one point must access the same data later, even if the Pod was recreated during that interval.
StatefulSet records these states and restores them when Pods are recreated. The mechanism relies heavily on Headless Services.
Headless Service Configuration
A Headless Service differs from a standard Service by omitting a cluster IP address:
apiVersion: v1
kind: Service
metadata:
name: cache-service
labels:
component: redis
spec:
ports:
- port: 6379
name: redis-port
clusterIP: None
selector:
component: redisSetting
clusterIP: None instructs Kubernetes to skip VIP allocation. Instead, the service exposes individual Pod endpoints through DNS records. Each Pod matching the selector receives a DNS entry following this pattern:<pod-name>.<service-name>.<namespace>.svc.cluster.localThis DNS record provides a stable, resolvable identity for each Pod.
StatefulSet Definition
The StatefulSet controller uses the Headless Service to maintain Pod identity:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
spec:
serviceName: cache-service
replicas: 3
selector:
matchLabels:
component: redis
template:
metadata:
labels:
component: redis
spec:
containers:
- name: redis-node
image: redis:6.2
ports:
- containerPort: 6379
name: redisThe
serviceName field links the StatefulSet to the Headless Service, ensuring each Pod receives a predictable DNS name.Ordered Pod Creation
StatefulSet assigns sequential indices to Pods, starting from zero. The naming convention follows
<statefulset-name>-<ordinal-index>:redis-cluster-0
redis-cluster-1
redis-cluster-2Pod creation follows strict ordering.
redis-cluster-1 remains in Pending state until redis-cluster-0 reports Ready status. This guarantees the master node initializes before replicas.Verify hostname assignments within containers:
kubectl exec redis-cluster-0 -- hostname
redis-cluster-0
kubectl exec redis-cluster-1 -- hostname
redis-cluster-1DNS Resolution Testing
Deploy a diagnostic container to test DNS resolution:
kubectl run dns-utils --image=busybox:1.28.4 --restart=Never --rm -it -- nslookup redis-cluster-0.cache-serviceThe query returns the specific IP address of
redis-cluster-0. Each Pod's DNS entry resolves independently.Pod Replacement Behavior
Delete the managed Pods to observe recreation behavior:
kubectl delete pod -l component=redisWatch the recreation process:
kubectl get pods -l component=redis -w
NAME READY STATUS RESTARTS AGE
redis-cluster-0 0/1 ContainerCreating 0 1s
redis-cluster-0 1/1 Running 0 3s
redis-cluster-1 0/1 Pending 0 0s
redis-cluster-1 0/1 ContainerCreating 0 1s
redis-cluster-1 1/1 Running 0 4s
redis-cluster-2 0/1 Pending 0 0s
redis-cluster-2 0/1 ContainerCreating 0 1s
redis-cluster-2 1/1 Running 0 5sStatefulSet recreates Pods with identical names and startup order. DNS records automatically update to point to the new Pod IPs. The network identity remains stable across Pod lifecycles.
Re-run DNS queries after recreation:
kubectl run dns-utils --image=busybox:1.28.4 --restart=Never --rm -it -- nslookup redis-cluster-0.cache-serviceThe DNS name
redis-cluster-0.cache-service resolves correctly to the replacement Pod's IP. This stability enables reliable service discovery for stateful workloads where master-slave relationships must persist.Applications should always reference Pods through their DNS names rather than direct IP addresses. While IPs change during Pod replacement, DNS names remain constant throughout the StatefulSet lifecycle.
