When something breaks, our first priority is restoring service, the second is keeping you informed, and the third is making sure it doesn’t happen the same way again.
Severity levels
| Level | Definition | Examples |
|---|---|---|
| SEV1 | Production is down or data is at risk for all users. | Site unreachable, database corruption, suspected breach. |
| SEV2 | Significant degradation, or a key feature broken for many users. | Checkout failing for some customers, very slow responses. |
| SEV3 | Limited impact, and a workaround exists. | A background job delayed, an admin page broken. |
| SEV4 | No user impact. | Questions, cosmetic issues, housekeeping. |
Response times for each level are in Service levels. When in doubt we classify higher and lower it later.
Roles
- Incident lead — owns decisions until the incident is closed. Exactly one person at a time; handovers are announced.
- Communications — sends updates to you and, for shared services, to the status page.
On smaller incidents one engineer holds both roles. On a SEV1 we always split them, so the person fixing the problem isn’t also writing updates.
What happens during an incident
- Acknowledge. The alert or your report is picked up and an incident lead is named.
- Assess. Who is affected, since when, and how badly. Severity is set.
- Stabilise. Restore service first — roll back, fail over, add capacity — and look for the root cause second.
- Communicate. Updates at fixed intervals, even when the update is “still investigating”.
- Resolve and verify. We confirm recovery from the outside, not just from the server’s point of view.
- Review. A written post-incident review, described below.
How we communicate
Updates follow four states — Investigating, Identified, Monitoring, Resolved — and each one says what we know, what we are doing and when you will hear from us next. Updates go to your shared channel and by email to your named contacts. During a SEV1 we update at least every 30 minutes.
After the incident
For every SEV1 and SEV2 we write a blameless post-incident review within five business days: timeline, impact, root cause, what went well, what didn’t, and action items with an owner and a date. We go through it with you on a short call. Action items are tracked in your monthly report until they are closed.
How to report an incident to us
- With the 24/7 on-call add-on: call the on-call number in your runbook.
- Otherwise: post in your shared channel or email your engineer with URGENT in the subject.
Tell us what you see, since when, and who is affected. Screenshots help; passwords don’t belong in the message.