Pulse

6 Sep 2026 · Alerts

Alert noise is a design choice

Most teams we sit with do not have a dashboard problem first. They have a page problem. Grafana looks busy. Alertmanager is full. On-call still learns which pages they can ignore.

That is not a tooling gap. Someone accepted a rule that fires on a condition the business can live with for an hour. Then another. Then a scrape target that flaps. After a quarter the list is a product, and nobody owns the catalog.

A useful pass looks like this:

1. Export every firing rule from the last 30 days. Count pages per rule. Name an owner or delete the rule.

2. Keep pages tied to user pain or SLO burn. Disk at 70% on a node that autoscales is a ticket, not a page.

3. Write the reason on the rule. If you cannot write one sentence, it is not ready to wake someone.

4. Put inhibition and routing next. A child page should not stack on the parent outage.

Dashboards come after. A board cannot teach a team to trust a pager they already learned to mute.

If you want this pass on a stack, say what paged last week.

All notes