Postmortem Recovery Playbook: How to solidify recovery actions after recovery

Why you need a postmortem recovery playbook

After the fault review (postmortem) is completed, the team usually produces a cause analysis and a list of improvement items. But here’s the thing: the list is posted in the Wiki, and no one remembers it months later. When the next accident occurs, I will still be scrambling through the chat history and asking my colleagues "How did I recover last time?"

The Postmortem recovery playbook extracts the repeatable recovery actions from the backup and formats them into a step manual that anyone (including newcomers on duty) can follow. It does not replace root cause analysis, but solves the problem of "slow recovery" - Google's SRE practice shows that with playbook services, MTTR (mean recovery time) can be reduced by more than 50%.

But many people fall into a trap when writing playbooks: they copy the full text of the review report without extracting the truly executable steps. As a result, the playbook becomes just another document that no one reads or uses.

Review input: What content is worthy of settling into recovery actions?

Not all review conclusions can become playbooks. Only actions that meet the following conditions are worth writing:

• Verifiable: The effect can be seen immediately after execution (such as restarting services, switching traffic, rolling back versions).
• Deterministic: If multiple people execute it under the same conditions, the results will be consistent.
• Bounded: Trigger conditions are specified (e.g. "When P99 delay exceeds 500ms for 3 minutes").

For example: The review found that "the database connection pool is exhausted causing requests to be queued". The root cause is that the connection is not closed in the code. The root cause itself is a fix task, left to development iterations. But the recovery action can be: "When the connection pool usage is >90% for 2 minutes, automatically expand the upper limit of the connection pool to 200, and trigger an alarm to notify the DBA to intervene." This action can be written into the playbook.

A common failure scenario is that the team regards long-term improvement items such as "strengthening code review" and "increasing unit test coverage" as recovery actions. These are important, but they are not part of the playbook - they are part of the improvement backlog.

Recovery action precipitation: how to convert the review report into steps

When extracting recovery actions, I recommend organizing them in an "operation flow" manner rather than a list. For example:

步骤 1：确认影响范围
  - 登录 Grafana 查看全局错误率面板
  - 如果错误率 > 5%，转步骤 2；否则继续监控 10 分钟
步骤 2：执行回滚
  - 切换到上一次稳定版本标签（v2.3.1）
  - 确认部署后错误率下降至 < 1%
  - 如果回滚失败（如版本冲突），转步骤 3
步骤 3：切换流量到冷备集群
  - 修改 DNS 记录指向备用集群 IP
  - 等待 TTL 过期后验证可用性

重点在于：写清楚判断条件和备选路径。很多团队只写了“执行回滚”，但回滚本身可能失败，没有 fallback 计划就会卡住。

另一个容易失败的地方是：步骤中缺少“负责人”。比如“重启服务”这个动作，谁有权限操作？值班 SRE 还是开发工程师？在 playbook 里明确“由 on-call 工程师发起，需要安全审计授权”可以减少推诿。

告警与负责人：当 playbook 和监控联动

Playbook 不能孤立存在，它必须与告警体系绑定。做法是：在 playbook 头部标明触发此 playbook 的告警规则。例如：

告警名称：HighErrorRate
告警条件：服务错误率 > 2% 持续 5 分钟
负责人（Role）：On-Call SRE（主）, 后端服务负责人（备）
切换条件：On-Call SRE 10 分钟内未响应，自动升级给后端负责人

Clearly write down the person in charge and the upgrade conditions to avoid delays caused by "waiting for confirmation". A real case: A team's playbook wrote "Contact DBA" but did not indicate who the DBA on duty was. As a result, it took 20 minutes for the call to get through. After improvement, directly write "Enterprise WeChat Robot @DBA Duty Group" in the playbook, and embed a rule for automatically dialing calls if there is no reply for 5 minutes.

In addition to people, you also need to write down the tool entrance clearly. For example, for the action "Execute rollback", you cannot just write "Use CI/CD platform", but must be specific to "Log in to Jenkins (Address: xxx) and select 'Rollback to Last Stable' parameterized build".

Next walkthrough: How to prevent the playbook from becoming a zombie document

A Playbook’s worst enemy is not being used. Regular drills are the only solution.

The frequency of drills depends on the criticality of the service and the frequency of changes. My suggestion is:

• Core Payment Services: monthly walkthrough
• General API Service: Quarterly
• Internal Tools: Semi-annually

The drill does not have to be a full-flow stress test, but can be a tabletop drill - team members gather around a playbook and verbally walk through the steps to find any missing or unclear areas.

Failure Scenarios: Many team drills are just a “read-through” without simulating abnormal conditions. A more effective approach is to deliberately create fault points that are not written in the playbook during the drill (such as the alarm channel itself being disconnected, the rollback version number being written incorrectly), and observing whether the practitioners know how to bypass it.

After each exercise, the playbook version number must be updated and the changes must be announced in the team group. Git can be used for version management. Each submission record states "This exercise found that a certain step lacks judgment conditions and has been supplemented."

Common misunderstandings and boundaries

1. **The more detailed the Playbook, the better? ** no. Playbooks longer than 20 steps tend to lose focus while executing. It is recommended to control it within 10 steps and split it into sub-playbooks if necessary (such as "rollback sub-process", "expansion sub-process").
2. Once the **Playbook is written, it will not be changed? ** impossible. It must be revised after each major accident review and after quarterly drills.
3. **Playbook is only suitable for large teams? **Small teams are also needed. When working alone, the playbook is your insurance policy against forgetting.

Scenario case: playbookization of one-time payment timeout recovery process

Suppose a review concludes: the payment order interface causes a cache avalanche due to Redis hot keys, and the recovery action is to manually clear the hot keys and limit the flow.

Extracted playbook fragment:

告警条件：支付下单接口 P99 延迟 > 3s
步骤：
1. 登录 Redis-cli，执行 `MEMORY USAGE payment:hot_order` 确认热 key 内存占用
2. 如果占用 > 100 MB，执行 `DEL payment:hot_order` 清除缓存（注意：会短暂增加 DB 压力）
3. 同步在网关层对 /payment/order 接口限流 50%（跳转 Nginx 限流配置页）
4. 观察 2 分钟，如果 P99 恢复 < 1s，结束；否则转步骤 5（回滚版本）
责任人：On-Call SRE

This playbook avoids the step of "contacting the development first for confirmation" and can be executed by the person on duty.