Fading Coder

An Old Coder’s Final Dance

You are here: Home > Tech > Content

Designing Alertmanager Templates for Prometheus Notifications

Tech 4

This guide explains how to craft Alertmanager templates to format alert messages, improving clarity and presentation. Alertmanager uses Go’s text/template engine with additional helper functions.

  • Alerting rules reference: https://www.dogfei.cn/archives/alertmanager-rule
  • Prometheus configuration overview: https://www.dogfei.cn/archives/prometheus-config

Go template essentials used by Alertmanager

Alertmanager renders notifications with Go templates. The core syntax is documented here: https://golang.org/pkg/text/template/

The dot (.) data context

The root data passed to the template is acessible as .. You can traverse nested fields via chaining:

{{ .User.Profile.Age }}

Template-scoped variables

You can bind values to variables and reuse them:

{{ $who := "alex" }}
Hello, {{ $who }}

Defining named templates

{{ define "wechat.message.default" }}
  ...
{{ end }}

Conditionals (if / else if / else)

Falsy values include: false, 0, empty string, nil, empty list/map.

{{ if .Name }}
Hello {{ .Name }}
{{ else if .BackupName }}
Hello {{ .BackupName }}
{{ else }}
Hello, stranger
{{ end }}

Iteration with range

Assume a slice of structs:

type Person struct {
  Name string
  Age  int
}

Template iteration:

{{ range .People }}
- name: {{ .Name }}
  age: {{ .Age }}
{{ end }}

Comparisons

  • eq: equal
  • ne: not equal
  • lt: less than
  • le: less than or equal
  • gt: greater than
  • ge: greater than or equal
{{ if ge .CPU 80 }}
High CPU: {{ .CPU }}%
{{ end }}

Logical operators

  • and: all operands true
  • or: any operand true
  • not: logical negation
{{ if and .Username (gt (len .Password) 8) }}
credentials look valid
{{ else }}
invalid credentials
{{ end }}

{{ if not .Authenticated }}
access denied
{{ end }}

Common helper functions

Useful functions supplied by Alertmanager include:

  • title: capitalize words
  • toUpper: upper-case letters
  • toLower: lower-case letters
  • join: concatenate with delimiter
  • safeHtml: mark string as safe HTML (disable auto-escaping)
  • len: length of strings, lists, maps

Examples:

{{ "alertmanager" | title }}
{{ "WARN" | toLower }}
{{ join "," (list "a" "b" "c") }}
{{ len .Alerts }}

Controlling whitespace

Trim whitespace using - inside delimiters:

{{- /* trim left */ -}}
{{- /* trim both sides */ -}}

Alert data available in templates

Alertmanager passes a full data object to templates. Commonly used fields:

  • .Receiver: receiver name
  • .Status: overall status (firing or resolved)
  • .Alerts: list of alert objects
  • .Alerts.Firing: subset of firing alerts
  • .Alerts.Resolved: subset of resolved alerts
  • .GroupLabels: labels used for grouping
  • .CommonLabels: labels common to all alerts
  • .CommonAnnotations: annotations common to all alerts
  • .ExternalURL: Alertmanager UI address

Example payload shape

{
  "receiver": "webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "NodeMemoryHigh",
        "instance": "10.127.92.100:9100",
        "job": "node",
        "severity": "warning",
        "team": "ops"
      },
      "annotations": {
        "summary": "High memory usage",
        "description": "Memory utilization above 55% (current: 58%)"
      },
      "startsAt": "2020-12-30T07:20:08.775177336Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus:9090/graph?...",
      "fingerprint": "09f94bd1aa7da54f"
    }
  ],
  "groupLabels": { "alertname": "NodeMemoryHigh" },
  "commonLabels": {
    "alertname": "NodeMemoryHigh",
    "job": "node",
    "severity": "warning",
    "team": "ops"
  },
  "commonAnnotations": { "summary": "High memory usage" },
  "externalURL": "http://alertmanager:9093",
  "version": "4",
  "groupKey": "{}:{alertname=\"NodeMemoryHigh\"}",
  "truncatedAlerts": 0
}

Template example for Enterprise WeChat

Reference (default template): https://raw.githubusercontent.com/prometheus/alertmanager/master/template/default.tmpl

The following template renders separate sections for firing and resolved alerts, with timestamps adjusted to Biejing time (UTC+8). The offset 28800e9 is 28,800 seconds in nanoseconds.

{{ define "wechat.message.default" }}
{{- $tz := 28800e9 -}}

{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $i, $a := .Alerts.Firing -}}
========== Alert Triggered ==========
Type:     {{ $a.Labels.alertname }}
Severity: {{ $a.Labels.severity }}
Summary:  {{ $a.Annotations.summary }}
Details:  {{ $a.Annotations.description }}
Started:  {{ ($a.StartsAt.Add $tz).Format "2006-01-02 15:04:05" }}
{{- if $a.Labels.instance }}
Instance: {{ $a.Labels.instance }}
{{- end }}
================= END ===============
{{ end -}}
{{- end -}}

{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $j, $r := .Alerts.Resolved -}}
======== Alert Recovered =========
Type:     {{ $r.Labels.alertname }}
Severity: {{ $r.Labels.severity }}
Summary:  {{ $r.Annotations.summary }}
Details:  {{ $r.Annotations.description }}
Started:  {{ ($r.StartsAt.Add $tz).Format "2006-01-02 15:04:05" }}
Ended:    {{ ($r.EndsAt.Add $tz).Format   "2006-01-02 15:04:05" }}
{{- if $r.Labels.instance }}
Instance: {{ $r.Labels.instance }}
{{- end }}
============== END ==============
{{ end -}}
{{- end -}}
{{ end }}

Using the template in Alertmanager

Reference the named template in the receiver configuration. Ensure your template file is loaded via the templates directive in Alertmanager’s configuration.

# Example: alertmanager.yml (snippet)
receivers:
  - name: wechat
    wechat_configs:
      - send_resolved: true
        message: '{{ template "wechat.message.default" . }}'
        to_party: "<party-id>"
        agent_id: "<agent-id>"
        to_user: "<user-id>"
        api_secret: "<corp-secret>"

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.