Rollback Recovery Playbook: How to safely bring the system back to a usable state after rollback

System recovery after rollback: more than just clicking a button

If something goes wrong with a deployment change in a production environment, the most common response is to roll it back. However, rollback only restores the code or configuration to the previous version and does not guarantee that automatic system recovery is available. Incompatibilities caused by database migration, dirty data in the cache, and status changes in downstream dependencies may cause the system to remain in a sub-healthy or unavailable state after rollback. What is really needed is to execute a complete recovery playbook to bring the system security back to stability and verify that it is available.

The most likely failure scenario: unrecovered variables and dirty data remaining

Scenario: Database migration rollback is inconsistent with data

Consider a common failure scenario: you perform a deployment that includes a database migration, adding new tables and modifying the old table structure. After going online, it was found that the performance was degraded, so the execution code was rolled back to the old version. However, the migration operation is not rolled back, the new table still exists, and the old table structure has been modified. When old code attempts to write to the old structure, it may directly report an error or produce inconsistencies. More insidiously, new data written during migration conflicts with old business logic, causing user-visible errors.

Failure point: The rollback script does not include the reverse operation of data migration, or the reverse operation is performed in the wrong order (for example, the new table is deleted first, but there are already records in the old table that depend on the new structure).

Another common scenario: Configuration center and rollback are out of sync

Your configuration items are delivered through the configuration center. This deployment changes both the code and configuration. After the code was rolled back, the configuration was not rolled back, causing the old code to use the new configuration (e.g. the connection string pointed to the wrong cluster).

Rollback recovery steps: 7 key actions from stop to verify

The following are the core steps of a production-proven recovery playbook. Each step has clear judgment conditions and failure responses.

1. Stop change diffusion immediately: If the changes are still being released in grayscale or in batches, stop subsequent traffic access. Do not continue to expand the fault area before confirming the rollback.
2. Execute code/configuration rollback: Use git revert to roll back to the previous release commit, or roll back to the previous build through CI/CD. For configuration, ensure that the configuration center is also rolled back synchronously (rolling back according to historical versions).
3. Execute data migration reverse operation: If this change includes database migration, perform reverse migration (such as using Flyway or Liquibase's undo script). If reverse migration is not prepared, you need to manually execute SQL to restore the old structure (restoring from backup should be prioritized at this time).
4. Clean cache: Incompatible format data written by old code may remain in the cache after rollback. Clear the relevant caching layer (Redis, Memcached or CDN cache). Note: If the cache is completely cleared, it may cause an avalanche. It is recommended to phase out or warm up by business dimensions.
5. Check dependency status: Confirm whether downstream services (API, message queue, database) are in the expected version. For example, you roll back service A, but service B that depends on A may have sent requests based on the new format that A can no longer handle correctly. At this time, B’s consumption needs to be suspended or downgraded.
6. Gradual Volume Verification: Restore traffic to a small group of users (e.g. 1%) and monitor error rates, latency, and key business metrics. Compare baseline data before and after rollback.
7. Continuous observation: Observe for at least 30 minutes (or a complete business cycle), and gradually restore full capacity after confirming that there are no abnormalities.

The easy step to make mistakes is step 3: Many teams only focus on code rollback and ignore data rollback, resulting in an inconsistent state of "old code, new data". It must be agreed in advance that all database changes must be accompanied by reversible scripts and verified in the pre-release environment.

Permission boundaries: Who has the authority to perform rollback recovery?

Rollback recovery is a high-stress operation, and improper permission control can introduce additional risks.

• Execution rollback decision: Usually judged by the on-call engineer or SRE based on the runbook, but major changes (such as affecting the entire site) need to be notified to the team lead for confirmation.
• Code Rollback: Only CI/CD services or authorized administrators can trigger the rollback pipeline. Avoid having each developer rollback directly into the production environment.
• Data Rollback: Database rollback scripts should only be executed by DBA or a role with backup recovery permissions. The automatic rollback script should be added to the "dry-run" mode, and the SQL to be executed should be output first and then be manually confirmed.
• Configuration Rollback: The configuration center should support change auditing and fast rollback, and the rollback operation should be recorded in the audit log.

Boundary case: When a failure seriously affects the user's core operations, the permission policy should set up an "emergency bypass mechanism" to allow designated roles to skip certain approval steps and record the reasons afterwards.

Fallback path when rollback fails: when rollback itself is unreliable

Even with a complete recovery playbook, rollback sometimes fails. For example:

• Service cannot start after rollback (old code relies on removed libraries or expired certificates).
• Data reverse migration resulted in more inconsistencies.
• Network interruption during the rollback process caused some nodes to be in a mixed state.

Alternative 1: Restore from Snapshot

Restore from the most recent full backup (full snapshot). This is typically more thorough than rollback, but has a longer recovery time (RTO). Suitable for scenarios where the data layer fails and cannot be repaired through reverse migration. Be sure to rehearse the snapshot recovery process regularly.

Alternative 2: Parallel switch to alternate environment

If the architecture supports blue-green deployment, traffic can be directly switched to the green environment (old version) when rollback fails. Pay attention to whether the green environment still maintains the latest data synchronization. If not, you need to perform data migration first or accept partial data loss.

Alternative 3: Downgrade and circuit breaker

If full recovery is not possible, you can enable the downgrade feature: turn off non-core functions to ensure that core links are available. For example, temporarily turn off the recommendation algorithm and only return popular data; change the write operation to an asynchronous queue to avoid direct failure. After downgrading, continue to fix underlying issues until a successful recovery can be performed.

Key Decision: Do not repeatedly try to roll back after a failed rollback, as this may lead to more serious data corruption. You should first evaluate whether it is appropriate to switch to an alternative solution and notify the team to enter the emergency response process.

Next step: Transform recovery practices into automation and team capabilities

The playbook provided in this article is a basic framework, and each team should customize it according to its own system characteristics. The important thing is not to memorize the steps, but to ensure that the steps are rehearsed and internalized into the deployment process.

If you want to learn more systematically how to design and write this type of recovery mechanism with the assistance of AI, and how to transform from an ordinary developer to an Agent engineer with engineering judgment, you can consider further exploring related courses.