Minimal Observability: The Five Alerts You Actually Need

Early observability fails in one of two directions: nothing at all, or a dashboard with 40 panels nobody opens and alerts everyone has muted. Five alerts is the right number before you have a team. Each one has to mean a human should act now, and everything else is a query you run when you are already investigating.

Alert 1: Error Rate Above 1 Percent for 5 Minutes

Count 5xx responses over total responses, evaluated in a rolling five-minute window. A percentage rather than a raw count is what makes this survive traffic growth without retuning.

Five minutes is deliberate. Shorter windows fire on a single bad deploy request; longer ones let a broken release run for a quarter of an hour. Exclude 4xx entirely, because a client sending malformed requests is not your outage.

Alert 2: p95 Latency Above 2 Seconds for 10 Minutes

Track the 95th percentile, not the average, because the average hides the tail where users actually live. Two seconds is not a performance goal, it is a “something is wrong” threshold; a healthy early API sits at 100 to 300 milliseconds and a jump to 2 seconds means a missing index, a slow provider, or an exhausted connection pool.

Ten minutes filters out cold starts and one-off slow queries. If this fires and error rate is clean, the cause is almost always the database or a third-party call in a request path where it does not belong.

Alert 3: Oldest Unprocessed Job Older Than 5 Minutes

Alert on the age of the oldest item in the queue, not on queue depth. Depth is meaningless without throughput: 10,000 items draining in 30 seconds is fine, and 3 items stuck for an hour is an outage.

This single alert covers a dead worker, a poison message in a retry loop, and a provider limit you are now hitting. For most early SaaS it catches more real problems than the error-rate alert does.

Alert 4: Any Dead-Letter or Failed Webhook Above Zero

The threshold is one. Not ten, not a rate. A failed inbound billing event or a permanently failed job means a customer’s state is wrong in a way that will not repair itself, and those are the incidents that end in a refund.

Make the alert include the event id and the last error string, so triage is reading one line instead of opening a log search.

Alert 5: The Daily Money Reconciliation Mismatch

Once a day, sum your own ledger per customer and compare it to the payment provider’s totals for the same period. Alert on any non-zero difference.

This is the alert nobody sets up and the one that finds the expensive bugs: a webhook silently dropped, a refund applied twice, a plan change that never wrote entitlement. It is a scheduled job and a comparison, and it is the difference between finding a billing divergence yourself and having a customer find it.

Why Everything Else Is Noise Now

CPU, memory, and disk alerts belong to whoever operates the machine; on a managed platform that is not you. Uptime pings from three regions tell you what alert 1 already told you. Per-endpoint latency alerts multiply into dozens of pages for problems the p95 alert catches once. Business metrics like signups are dashboards, not pages, because no engineer can fix a slow signup day at 2 a.m.

One more rule: every alert must route to a place you will see at night, and every alert that fires twice without action gets deleted or retuned. A muted alert is worse than no alert, because it creates the belief that you are covered.

Setup Checklist

• 5xx rate above 1 percent over a rolling 5-minute window, 4xx excluded.
• p95 request latency above 2 seconds sustained for 10 minutes.
• Age of oldest unprocessed job above 5 minutes, measured on age and not depth.
• Dead-letter or permanently failed event count above zero, with event id and error in the alert body.
• Daily ledger versus provider reconciliation, alerting on any non-zero difference.
• All five route to a channel you receive at night, and you have tested each one by triggering it deliberately.
• Structured request logs with a request id, tenant id, route, status, and duration exist, so triage does not depend on alerts.
• Every alert has a one-line runbook naming the first thing to check.