The Prometheus Operator simplifies deployment and management of Prometheus monitoring stacks within Kubernetes environments. This implementation enables automated service discovery, persistent storage configuration, and integration with alerting systems like Alertmanager.

Core Architecture Components

Prometheus Server Functions

• Metric collection through HTTP-based pull mechanisms from defined targets
• Local time-series data base storage using TSDB format for efficient data handling
• PromQL query engine for real-time data analysis and retrieval

Data Collection Agents

Various exporters facilitate metric gathering:

• Node Exporter: System-level metrics (CPU, memory, disk)
• Blackbox Exporter: Network service availability testing
• MySQL/Redis/MongoDB Exporters: Database-specific performance data

Push Gateway

Handles metric submission from short-lived jobs that cannot be scraped directly.

Alert Processing

Alertmanager provides grouping, inhibition, silencing, and multi-channel notification capabilities.

Service Discovery Mechanisms

Supports dynamic target identification through:

• Static configuration files
• DNS record parsing
• Kubernetes API integration
• Consul registry

Kubernetes Deployment Process

Using the kube-prometheus project aligned with Kubernetes version 0.13:

git clone -b release-0.13 https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus/manifests/
kubectl create -f setup/
kubectl create -f .

Expose interfaces externally:

kubectl patch svc grafana -n monitoring -p '{"spec": {"type": "NodePort"}}'
kubectl patch svc prometheus-k8s -n monitoring -p '{"spec": {"type": "NodePort"}}'

Storage Persistence Configuration

Persistent volumes ensure data retention across pod restarts:

Prometheus Storage

storage:
  volumeClaimTemplate:
    spec:
      storageClassName: rook-ceph-block
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Alertmanager Persistence

storage:
  volumeClaimTemplate:
    spec:
      storageClassName: rook-ceph-block
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Grafana Configuration Storage

volumes:
- name: grafana-storage
  persistentVolumeClaim:
    claimName: grafana-pvc

Monitoring Target Registration

ServiceMonitor custom resources define scraping configurations:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: example
  endpoints:
  - port: metrics
    interval: 30s

External Service Integration Example

For monitoring external MySQL databases:

Exporter Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-exporter
  template:
    metadata:
      labels:
        app: mysql-exporter
    spec:
      containers:
      - name: exporter
        image: prom/mysqld-exporter
        env:
        - name: DATA_SOURCE_NAME
          value: "user:password@(host:port)/database"
        ports:
        - containerPort: 9104

Service Monitor Definition

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mysql-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: mysql-exporter
  endpoints:
  - port: metrics
    interval: 30s

Static Configuration Approach

Alternative to dynamic discovery using Secrets:

additionalScrapeConfigs:
  name: prometheus-static-config
  key: static-config.yaml

Static configuration content:

- job_name: 'static-targets'
  static_configs:
  - targets: ['192.168.1.100:9100']

Alert Routing Configuration

Alertmanager routing to external notification systems:

receivers:
- name: "notification-channel"
  webhook_configs:
  - url: "http://alert-gateway:8080/receive?type=slack"
route:
  receiver: "notification-channel"
  repeat_interval: 5m

Validation Checklist

Confirm successful monitoring setup:

1. Verify ServiceMonitor CRD creation
2. Check label matching between ServiceMonitors and Services
3. Validate generated Prometheus configuration
4. Test metric endpoint accessibility via Service IPs
5. Confirm port and scheme consistency

This configuration establishes comprehensive infrastructure and application monitoring with persistent data storage and flexible alerting pathways.