Designing Alertmanager Templates for Prometheus Notifications
This guide explains how to craft Alertmanager templates to format alert messages, improving clarity and presentation. Alertmanager uses Go’s text/template engine with additional helper functions.
- Alerting rules reference: https://www.dogfei.cn/archives/alertmanager-rule
- Prometheus configuration overview: https://www.dogfei.cn/archives/prometheus-config
Go template essentials used by Alertmanager
Alertmanager renders notifications with Go templates. The core syntax is documented here: https://golang.org/pkg/text/template/
The dot (.) data context
The root data passed to the template is acessible as .. You can traverse nested fields via chaining:
{{ .User.Profile.Age }}
Template-scoped variables
You can bind values to variables and reuse them:
{{ $who := "alex" }}
Hello, {{ $who }}
Defining named templates
{{ define "wechat.message.default" }}
...
{{ end }}
Conditionals (if / else if / else)
Falsy values include: false, 0, empty string, nil, empty list/map.
{{ if .Name }}
Hello {{ .Name }}
{{ else if .BackupName }}
Hello {{ .BackupName }}
{{ else }}
Hello, stranger
{{ end }}
Iteration with range
Assume a slice of structs:
type Person struct {
Name string
Age int
}
Template iteration:
{{ range .People }}
- name: {{ .Name }}
age: {{ .Age }}
{{ end }}
Comparisons
- eq: equal
- ne: not equal
- lt: less than
- le: less than or equal
- gt: greater than
- ge: greater than or equal
{{ if ge .CPU 80 }}
High CPU: {{ .CPU }}%
{{ end }}
Logical operators
- and: all operands true
- or: any operand true
- not: logical negation
{{ if and .Username (gt (len .Password) 8) }}
credentials look valid
{{ else }}
invalid credentials
{{ end }}
{{ if not .Authenticated }}
access denied
{{ end }}
Common helper functions
Useful functions supplied by Alertmanager include:
- title: capitalize words
- toUpper: upper-case letters
- toLower: lower-case letters
- join: concatenate with delimiter
- safeHtml: mark string as safe HTML (disable auto-escaping)
- len: length of strings, lists, maps
Examples:
{{ "alertmanager" | title }}
{{ "WARN" | toLower }}
{{ join "," (list "a" "b" "c") }}
{{ len .Alerts }}
Controlling whitespace
Trim whitespace using - inside delimiters:
{{- /* trim left */ -}}
{{- /* trim both sides */ -}}
Alert data available in templates
Alertmanager passes a full data object to templates. Commonly used fields:
- .Receiver: receiver name
- .Status: overall status (firing or resolved)
- .Alerts: list of alert objects
- .Alerts.Firing: subset of firing alerts
- .Alerts.Resolved: subset of resolved alerts
- .GroupLabels: labels used for grouping
- .CommonLabels: labels common to all alerts
- .CommonAnnotations: annotations common to all alerts
- .ExternalURL: Alertmanager UI address
Example payload shape
{
"receiver": "webhook",
"status": "firing",
"alerts": [
{
"status": "firing",
"labels": {
"alertname": "NodeMemoryHigh",
"instance": "10.127.92.100:9100",
"job": "node",
"severity": "warning",
"team": "ops"
},
"annotations": {
"summary": "High memory usage",
"description": "Memory utilization above 55% (current: 58%)"
},
"startsAt": "2020-12-30T07:20:08.775177336Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://prometheus:9090/graph?...",
"fingerprint": "09f94bd1aa7da54f"
}
],
"groupLabels": { "alertname": "NodeMemoryHigh" },
"commonLabels": {
"alertname": "NodeMemoryHigh",
"job": "node",
"severity": "warning",
"team": "ops"
},
"commonAnnotations": { "summary": "High memory usage" },
"externalURL": "http://alertmanager:9093",
"version": "4",
"groupKey": "{}:{alertname=\"NodeMemoryHigh\"}",
"truncatedAlerts": 0
}
Template example for Enterprise WeChat
Reference (default template): https://raw.githubusercontent.com/prometheus/alertmanager/master/template/default.tmpl
The following template renders separate sections for firing and resolved alerts, with timestamps adjusted to Biejing time (UTC+8). The offset 28800e9 is 28,800 seconds in nanoseconds.
{{ define "wechat.message.default" }}
{{- $tz := 28800e9 -}}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $i, $a := .Alerts.Firing -}}
========== Alert Triggered ==========
Type: {{ $a.Labels.alertname }}
Severity: {{ $a.Labels.severity }}
Summary: {{ $a.Annotations.summary }}
Details: {{ $a.Annotations.description }}
Started: {{ ($a.StartsAt.Add $tz).Format "2006-01-02 15:04:05" }}
{{- if $a.Labels.instance }}
Instance: {{ $a.Labels.instance }}
{{- end }}
================= END ===============
{{ end -}}
{{- end -}}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $j, $r := .Alerts.Resolved -}}
======== Alert Recovered =========
Type: {{ $r.Labels.alertname }}
Severity: {{ $r.Labels.severity }}
Summary: {{ $r.Annotations.summary }}
Details: {{ $r.Annotations.description }}
Started: {{ ($r.StartsAt.Add $tz).Format "2006-01-02 15:04:05" }}
Ended: {{ ($r.EndsAt.Add $tz).Format "2006-01-02 15:04:05" }}
{{- if $r.Labels.instance }}
Instance: {{ $r.Labels.instance }}
{{- end }}
============== END ==============
{{ end -}}
{{- end -}}
{{ end }}
Using the template in Alertmanager
Reference the named template in the receiver configuration. Ensure your template file is loaded via the templates directive in Alertmanager’s configuration.
# Example: alertmanager.yml (snippet)
receivers:
- name: wechat
wechat_configs:
- send_resolved: true
message: '{{ template "wechat.message.default" . }}'
to_party: "<party-id>"
agent_id: "<agent-id>"
to_user: "<user-id>"
api_secret: "<corp-secret>"