Notification webhooks
Each destination is an HTTPS URL. On every Alertmanager event the notifier POSTs a body and optional headers. Custom destinations (the default) send the raw Alertmanager JSON with HMAC signatures — the same contract POST /api/notifier/webhooks { url, description } has always used.
Named templates (PagerDuty, Slack, Microsoft Teams, Opsgenie) render a vendor-shaped JSON body instead and turn HMAC off so extra X-Webhook-* headers do not confuse the vendor.
$VARIABLE substitution
Use $NAME in the JSON body, URL, or header values. Names are A-Z, digits, and underscore. Unknown names are left intact. $$ becomes a literal $.
When Content-Type is JSON, interpolated values are JSON-string-escaped so quotes and newlines in $TITLE cannot break the document.
| Variable | Meaning |
|---|---|
$STATUS | firing or resolved |
$TITLE | first alert summary or alertname |
$ALERT_NAME | group / common / first alertname |
$SEVERITY | first/common severity (default warning) |
$SUMMARY / $DESCRIPTION | first annotations |
$MESSAGE | [FIRING] / [RESOLVED] + name + counts |
$COUNT / $FIRING_COUNT / $RESOLVED_COUNT | counts |
$STARTS_AT / $ENDS_AT | first alert timestamps |
$GENERATOR_URL / $EXTERNAL_URL | first / payload |
$GROUP_KEY / $RECEIVER / $FINGERPRINT | payload / first |
$INSTANCE | first labels.instance |
$DEDUP_KEY | fingerprint or groupKey (max 255 chars) |
$EVENT_ACTION | trigger if firing else resolve |
$PD_SEVERITY | PagerDuty severity enum |
$COMMON_LABELS / $COMMON_ANNOTATIONS / $ALERTS | JSON text |
$RAW | original Alertmanager body |
$TIMESTAMP | unix seconds |
HMAC and secret headers
Custom destinations default to HMAC on. The notifier signs the rendered body:
X-Webhook-Signature: sha256=<hex>X-Webhook-Timestamp: <unix seconds>User-Agent: elku-notifier/1
Named templates default HMAC off. You can still toggle it in Advanced.
Secret headers (API keys, routing keys) are write-only: list/GET returns { name, secret: true, valueSet: true } with no value. PATCH that omits value keeps the stored secret. Sending value: "" or dropping the header from the array removes it.
Header names follow RFC 7230 tokens. Host, Content-Length, Transfer-Encoding, and Connection are rejected. Max 20 headers.
Custom JSON example
Create a custom webhook with body template:
{"text":"$TITLE","status":"$STATUS","count":"$COUNT"}A firing alert whose summary is Disk full POSTs:
{"text":"Disk full","status":"firing","count":"1"}Leave the body template empty to send the original Alertmanager JSON (the default, including make synth-e2e-full-tenant).
PagerDuty Events API v2
URL default: https://events.pagerduty.com/v2/enqueue.
Store the routing key as a secret header X-PD-Routing-Key (never forwarded). Firing maps to event_action=trigger with summary=$TITLE, source=$INSTANCE (or elk-utilities), severity=$PD_SEVERITY, dedup_key=$DEDUP_KEY. Resolved maps to event_action=resolve with the same dedup_key.
Slack incoming webhook
Paste the Incoming Webhook URL (the URL is the credential). Body:
{"text":"$MESSAGE"}HMAC off. No extra headers.
Microsoft Teams MessageCard
Paste the Incoming webhook / Workflow URL. The notifier POSTs a MessageCard: summary/title = $TITLE, text = $DESCRIPTION, themeColor FF0000 while firing and 00AA00 when resolved. HMAC off.
Opsgenie alerts
Firing POSTs https://api.opsgenie.com/v2/alerts (EU: https://api.eu.opsgenie.com/v2/alerts) with { "message": "$TITLE", "alias": "$DEDUP_KEY", "description": "$DESCRIPTION", "priority": "P3", "source": "elk-utilities" }.
Store Authorization: GenieKey <api-key> as a secret header.
Resolved POSTs {url-base}/$DEDUP_KEY/close?identifierType=alias with { "source": "elk-utilities" }.