The Rollback Trap: Why Your Database Migration Cannot Be Undone Like Code

A single column change reveals why "redeploy previous image" fails. Reversibility depends on schema state, not the migration tool.

7 min read

Failure Modes are written from published postmortems, incident write-ups and engineering documentation, which are listed at the foot of every piece. Nothing in this section has been run, measured or operated here — where a number, a threshold or a result appears, it belongs to the source it is credited to.

The Illusion of Symmetric Reversal

The standard operating procedure for a failed release is to redeploy the previous container image. This practice rests on a fundamental assumption: that the application state is isolated from the infrastructure state. For code, this is largely true. A binary or a bundle of scripts does not alter the environment it runs in; it simply executes within it. However, a database migration is not a deployment artefact in this sense. It is a state mutation. When a migration runs, it changes the physical structure of the storage layer in a way that the previous version of the application was not designed to anticipate. The claim that a migration can be rolled back like any other deployment ignores the asymmetry between code execution and schema definition. Code is passive until invoked; schema is active in how it constrains and presents data. If the previous version of the application expects a column to exist, and the migration has dropped it, the previous version will not merely fail; it will encounter a structural mismatch that no amount of container orchestration can fix. The error is not in the deployment pipeline, but in the mental model that treats the database as a static fixture rather than a dynamic participant in the release process.

This misconception is pervasive because migration tools are designed to look like deployment tools. They have versions, they have up and down commands, and they integrate with CI/CD systems. This interface suggests that the process is reversible in the same way that a git revert is reversible. But a git revert restores a file to a previous state; it does not alter the filesystem permissions or the disk geometry. A database migration, by contrast, alters the contract between the application and the data. The "down" migration is not a true inverse in the mathematical sense; it is a best-effort reconstruction. If the data has changed during the period the new schema was live, the reconstruction is impossible. The data in the new columns has no corresponding representation in the old schema. Therefore, the reversibility of a migration is not a property of the tool, but a property of the temporal window during which both schemas are compatible. Once that window closes, the rollback is no longer a software operation; it becomes a data engineering problem.

Step One: Adding the Column

Consider a specific, common scenario: splitting a single `inventory_code` field into three distinct fields: `location_code`, `batch_number`, and `serial_number`. The first step in the evolutionary database design approach, as described by Fowler and Sadalage, is to add the new columns to the existing table. This step is often considered safe because it is additive. The previous version of the application, which only knows about `inventory_code`, continues to function. It reads and writes to the old column, unaware that new, empty columns now sit alongside it. This is the only phase where the "redeploy previous image" strategy works without modification. The schema has expanded, but the contract for the old version has not been broken. The old code does not query the new columns, so their presence is invisible to it. This phase is the "expand" part of the expand-contract pattern, and it is the only state where the system is truly backwards compatible.

However, this safety is fragile. It relies on the assumption that the new columns are optional and that no other part of the system has been updated to depend on them yet. If a view, a stored procedure, or a trigger has been modified in the same release to reference the new columns, the old application version might still work, but the database layer is now in a mixed state. The PostgreSQL documentation notes that adding a column with a constant default value does not require updating every row immediately, which makes the operation fast. But if the default is volatile, or if constraints are added simultaneously, the operation can lock the table or require a full rewrite. The risk here is not logical incompatibility, but operational disruption. If the `ALTER TABLE` command takes too long, it can block reads and writes, causing the application to time out. In this case, the "rollback" is not a code redeployment; it is a database recovery operation. The previous image is irrelevant if the database is locked or corrupted by the migration attempt.

Step Two: Migrating the Data

The second step is the data migration itself. The application code is changed to write to the new columns, and a script is run to backfill the new columns from the old one. This is the critical juncture. Once the data migration script runs, the new columns contain valid data. But the old `inventory_code` column still exists. At this point, the system is in a dual-write state. The new version of the application writes to both the old and new columns, or it writes only to the new ones while reading from the old ones, depending on the implementation. The previous version of the application, however, still writes only to the old column. If you deploy the new version, then realize it is broken, and redeploy the previous image, the previous image will start writing only to the old column. The data in the new columns will become stale. Worse, if the new version had already written some data to the new columns, that data is now orphaned. The previous version does not know how to read it.

This is where the claim of reversibility collapses. The migration tool can drop the new columns, but it cannot un-migrate the data that was written to them by the new version. If the new version was live for even a few minutes, it has generated data that has no home in the old schema. The "down" migration would have to delete the new columns, effectively destroying that data. This is not a rollback; it is a data loss event. The expand-contract pattern exists to avoid this state. It requires that the data migration be idempotent and that the application code be updated in a way that allows both versions to coexist. But if the application code is not updated to handle the dual state, the data migration creates a dependency that cannot be undone. The previous version is not just incompatible; it is actively harmful, because it will continue to write to the old column while ignoring the new one, leading to data divergence.

Step Three: Contracting the Schema

The final step is the contraction: removing the old column. This is the point of no return. Once the old `inventory_code` column is dropped, the previous version of the application cannot run. It will throw a "column not found" error on every read and write. This is the intended outcome of the evolution, but it is also the moment where the rollback plan becomes impossible. If you have reached this stage, you cannot redeploy the previous image. The schema has changed in a way that is incompatible with the old code. The only way to "rollback" is to re-add the column and re-migrate the data back into it. This is a complex, error-prone process that is rarely tested. It requires a reverse data migration script, which is just as likely to fail as the forward one. The expand-contract pattern buys you time, but it does not buy you reversibility. It buys you compatibility. It allows you to deploy the new version, verify it, and then contract the schema. But once you contract, you are committed. The previous version is dead. The only way back is to build a new version that can handle the old schema, which is effectively a new release. The claim that a migration can be rolled back is therefore false in any practical sense. It is a one-way door.

What this is built on