Prometheus natively supports scraping metrics from services that expose a /metrics HTTP endpoint—typically in Prometheus exposition format. Etcd is one such service, exposnig its telemetry over HTTPS on port 2379. Integrating it into a Kubernetes-based Prometheus stack (e.g., using the Prometheus Operator) requires careful configuration of endpoints, service abstraction, TLS credentials, and discovery rules.

Core Integration Steps

1. Verify Etcd’s metrics endpoint is reachable and secured with mutual TLS.
2. Expose Etcd as a Kubernetes Endpoints object to decouple IP management from static definitions.
3. Create a corresponding Service to enible DNS-based resolution inside the cluster.
4. Define a ServiceMonitor resource with proper TLS settings and scrape interval.
5. Mount Etcd’s CA and peer certificates into Prometheus pods via a Secret, referanced in both the ServiceMonitor and Prometheus CR.
6. Ensure the Prometheus custom resource includes the secret in its spec.secrets list to trigger volume mounts.

Step-by-Step Implementation

1. Validate Etcd Metrics Endpoint

Confirm Etcd serves metrics over HTTPS and accepts client certificate authentication:

curl -k \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  https://192.168.15.31:2379/metrics | head -n 10

2. Define Static Endpoints

Create an Endpoints object in kube-system to reference the Etcd node(s):

apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-targets
  namespace: kube-system
  labels:
    app: etcd
subsets:
- addresses:
  - ip: 192.168.15.31
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP

3. Expose via Service

Bind the endpoints to a stable ClusterIP service:

apiVersion: v1
kind: Service
metadata:
  name: etcd-targets
  namespace: kube-system
  labels:
    app: etcd
spec:
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP
    targetPort: 2379
  clusterIP: None

4. Configure ServiceMonitor with TLS

Define how Prometheus should scrape the service—including certificate paths and insecure fallback (if needed):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-scrape-config
  namespace: monitoring
  labels:
    team: infra
spec:
  selector:
    matchLabels:
      app: etcd
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: https-metrics
    interval: 15s
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-tls/ca.crt
      certFile: /etc/prometheus/secrets/etcd-tls/peer.crt
      keyFile: /etc/prometheus/secrets/etcd-tls/peer.key
      insecureSkipVerify: false

5. Provision Certificates as Kubernetes Secret

Bundle Etcd’s TLS assets into a namespaced secret:

kubectl create secret generic etcd-tls \
  --namespace=monitoring \
  --from-file=/etc/kubernetes/pki/etcd/ca.crt \
  --from-file=/etc/kubernetes/pki/etcd/peer.crt \
  --from-file=/etc/kubernetes/pki/etcd/peer.key

6. Update Prometheus Custom Resource

Reference the secret so Prometheus mounts it into its pods:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: main
  namespace: monitoring
spec:
  # ... other fields omitted for brevity
  secrets:
  - etcd-tls
  serviceMonitorSelector:
    matchLabels:
      team: infra

Verification

After applying all resources:

• Check ServiceMonitor status: kubectl get servicemonitor -n monitoring
• Inspect Prometheus targets UI (http://<prometheus-url>/targets) for the etcd-targets/kube-system/0 job.
• Query metrics like etcd_disk_wal_fsync_duration_seconds_count or etcd_server_is_leader.

Grafana Visualization

Import dashboard ID 3070 ("Etcd Metrics") from Grafana Labs into your Grafana instance. Ensure the Prometheus data source is correctly selected and variables (e.g., job, instance) align with your ServiceMonitor labels and target labels.