Configuring Alertmanager Web Access

Create an Ingress resource for Alertmanager web interface access:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: alertmanager-ingress
  namespace: monitoring
spec:
  rules:
  - host: alertmanager.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-operator-alertmanager
            port:
              number: 9093

Apply the configuration:

kubectl apply -f alertmanager-ingress.yaml

To verify the current Alertmanager configuration:

kubectl get secret -n monitoring alertmanager-prometheus-operator-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 --decode

Setting Up DingTalk Integration

1. Create DingTalk Webhook Bot

Follow the official DingTalk documentation to create a custom bot and obtain the access tokan.

2. Deploy DingTalk Webhook Service

Create a deployment and service for the DingTalk webhook integration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dingtalk-webhook
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dingtalk-webhook
  template:
    metadata:
      labels:
        app: dingtalk-webhook
    spec:
      containers:
      - name: webhook
        image: timonwong/prometheus-webhook-dingtalk:v0.3.0
        args:
          - --ding.profile=dingtalk-alerts=https://oapi.dingtalk.com/robot/send?access_token=YOUR_ACTUAL_TOKEN_HERE
        ports:
        - containerPort: 8060
---
apiVersion: v1
kind: Service
metadata:
  name: dingtalk-webhook
  namespace: monitoring
spec:
  ports:
  - port: 8060
    targetPort: 8060
  selector:
    app: dingtalk-webhook

Deploy the DingTalk webhook service:

kubectl apply -f dingtalk-webhook.yaml

3. Configure Alertmanager Receivers

Update the Alertmanager configuration to include DingTalk as a receiver:

alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ["alertname"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: dingtalk-receiver
      routes:
      - match:
          severity: critical
        receiver: dingtalk-receiver
    receivers:
    - name: dingtalk-receiver
      webhook_configs:
      - url: 'http://dingtalk-webhook.monitoring.svc.cluster.local:8060/dingtalk/dingtalk-alerts/send'
        send_resolved: true

4. Udpate Alertmanager Secret

Update the Alertmanager configuration secret using Kubernetes commands:

kubectl delete secret alertmanager-prometheus-operator-alertmanager -n monitoring
kubectl create secret generic alertmanager-prometheus-operator-alertmanager --from-file=alertmanager.yaml -n monitoring

Alternatively, update using Helm:

helm upgrade prometheus-operator prometheus-community/kube-prometheus-stack --values=alertmanager-config.yaml -n monitoring

Verify the configuration by accessing the Alertmanager web interface and checking that DingTalk alerts are properly configured.