Skip to content

Notifications

Kerf can tell you when something happens, so you don't have to watch a page: a rollback, a change blocked by a guardrail, a cluster that lost readiness, or a weekly summary of savings. Messages go to Slack, Microsoft Teams or a generic webhook (signed, for PagerDuty, Opsgenie, a chat bot or your own automation). Set it up under Admin → Notifications (admins edit, everyone else reads) or define channels in the chart values.

Channels

Type What you need Message
Slack An incoming webhook URL A Block Kit message with an Open in Kerf button
Microsoft Teams A Workflows webhook URL (the Post to a channel when a webhook request is received template; Office 365 connectors are being retired) An Adaptive Card with an Open in Kerf action
Webhook Any HTTPS URL, and optionally a signing secret (at least 16 characters) A versioned JSON body, signed with HMAC-SHA256: the contract

The URL of a Slack or Teams webhook is a credential. Kerf stores it encrypted with the same key as the credentials of metrics sources (onboarding.secretsKey, generated by the chart) and shows it masked (scheme and host only); it is never returned by the API, written to a log, or put into an error message. Without that key the page says so and channels cannot be created.

Send test posts a test message to the channel right away and shows the result, including the error when the destination refused it (for example HTTP 404 Not Found: no_service for a revoked Slack webhook). The last delivery of every channel (sent, retrying or failed, and why) is on the channel list; the history of all deliveries is below it and is kept for 30 days.

Rules

A rule says which events go to which channels:

Field Meaning
Event types applied, rolled_back, failed, blocked, readiness_degraded (a cluster's readiness state went down), automation_paused, automation_resumed
Clusters Glob patterns such as prod-*; empty means every cluster
Namespaces Glob patterns such as payments-*; empty means every namespace. Events that belong to no namespace (readiness, automation) never match a rule that lists namespaces
Channels One or more

Example: rollbacks and failures in prod-* to a platform-alerts Slack channel and a PagerDuty webhook; everything else only in the weekly digest. An event matching several rules reaches a channel once.

The events are the entries of the audit log, nothing else is invented. Kerf reads them as the hub stores them and sends a rollback within seconds. It applies the same de-duplication as the log: a blocked event is sent once per workload and reason per channel per day (the recommender hitting the same guardrail every run is one message, not one per run), a failed event once per workload and reason per hour. Nothing is sent for events from before the first rule existed.

Weekly digest

On the day and at the time you choose (in your time zone), the digest goes to the channels you choose:

  • Estimated savings per month, and the change since last week;
  • Realized savings of the week, labelled "Realized since \<date>" like the Overview and shown apart from the estimate (see Realized savings); left out until a change has released requests;
  • the top five opportunities;
  • changes applied, rolled back and failed;
  • clusters that need attention: a silent agent, readiness below Automation ready, paused automation;
  • savings by team, when teams are defined: each team's estimated monthly savings, with Unassigned.

Send digest now queues one at once. A digest the hub was down for is sent late (up to 12 hours), then skipped.

Delivery

  • The hub leader sends, so a message leaves once however many replicas run; a failover loses nothing (the queue is in Postgres).
  • Retries: a failed attempt is retried with exponential backoff (30 s, 1 min, 2 min, … at most 15 min apart) for one hour, then the delivery is marked failed and stays in the history. Errors that waiting cannot fix (a revoked webhook answers 404) fail at once. Retry-After is honored.
  • Rate limit: at most 20 messages per minute per channel (notifications.rateLimitPerMinute); the surplus waits.
  • Metrics and alert: kerf_notification_deliveries_total{channel_type,result} (success, retry, failed) and kerf_notification_last_success_timestamp_seconds{channel} (metrics). The chart's alert KerfNotificationDeliveryFailing fires when deliveries fail.
  • Links: messages link to the workload (or cluster) page of the web UI when the hub knows its address (notifications.baseURL). Without it messages carry no links.

Chart values

hub:
  notifications:
    baseURL: https://kerf.example.com        # where the web UI is reachable; used for the links
    rateLimitPerMinute: 20
    # Channels from values (read-only in the UI). Each reads its URL, and a webhook channel its signing secret,
    # from a Secret you create (or sync with External Secrets):
    channels:
      - name: platform-alerts
        type: slack                          # slack | teams | webhook
        existingSecret: kerf-slack-platform  # a Secret in the hub's namespace
        urlKey: url                          # default: url
      - name: pagerduty
        type: webhook
        existingSecret: kerf-pagerduty
        urlKey: url
        secretKey: signing-secret            # optional: sign the body with this

Rules and the digest are set in the UI (they are stored in the database and changes are recorded in the audit log as NotificationRule… and NotificationDigestUpdated). allowInsecureHTTP: true accepts http:// URLs for a receiver inside your cluster; the default is HTTPS only. With the umbrella chart the values live under hub.notifications.

The hub makes the requests: allow egress from its pods to Slack (hooks.slack.com), Teams (the Workflows host of your tenant) or your webhook (hub.networkPolicy.egress).

Troubleshooting

Symptom Likely cause
Send test fails with HTTP 404 or invalid_token The webhook was revoked or the URL is wrong: create a new one and edit the channel.
Test works but no rollback arrives No rule matches: check event types, cluster and namespace patterns, and that the rule and the channel are enabled.
Deliveries stay pending The destination is failing (see the error on the channel), or egress to it is blocked.
secrets_key_missing The hub has no secrets encryption key; see onboarding.secretsKey in the hub chart.
A rollback came twice It didn't: two rules to two channels, or an applied and a rolled back message. Retries carry the same X-Kerf-Delivery.