Drift Happens in Silence

No one schedules infrastructure drift. It doesn't arrive with a deployment notification or a calendar invite. It accumulates in the gap between what the code says and what the cluster is actually doing, and by the time anyone looks at that gap directly, it has stopped being a configuration problem and become an archaeology problem.

The uncomfortable part is that drift is not caused by carelessness. It is caused by the system working — by people fixing things under pressure, using exactly the access they were given for exactly that purpose, and then going home.

Where the gap opens

There are only a few ways declared state and running state come apart, and they are worth separating because they need different answers.

The first is the emergency change. Something is on fire at two in the morning, the fix is one field on one resource, and going through a pull request would add twenty minutes to an outage. Nobody sensible argues that the pull request should win. The failure is not the change; it is that nothing afterwards forces the change back into the repository, so the fix survives and the record of it does not.

The second is the thing outside your control. A managed service adds a default. A cloud provider changes what an unset field means. A controller you installed mutates the object you created, for good reasons, and now the object in the cluster is not the object you wrote. This kind of drift is not anybody's mistake and cannot be prevented — only detected and then either adopted or ignored deliberately.

The third is orphaned state: resources that exist and that nothing in the repository claims. A test environment somebody built by hand. A load balancer left behind when a service was renamed. A security group that was attached to something that no longer exists. These do not show up as drift, because drift is a comparison and there is nothing to compare against.

The plan that is never empty

The most reliable early symptom is a plan that always shows changes and a team that has learned to read past it.

This starts as a small annoyance and becomes something worse: it destroys the signal. Once a plan routinely shows a dozen diffs that everyone knows are fine, nobody can distinguish the thirteenth — the one that is about to delete a volume — from the noise. The tool is still working perfectly. It is telling you the truth, in full, and the truth has become unreadable.

What makes this dangerous rather than merely untidy is that the tolerance is asymmetric. The habit of scrolling past a diff is trained by hundreds of harmless diffs and is then applied, once, to a harmful one. Nobody decides to ignore the destructive change. They just do the thing they have done every day for a year.

Detect on a schedule, not at apply time

Most teams discover drift at the worst possible moment: when they are trying to ship something else. You go to deploy a small change, the plan shows nine unrelated modifications, and now the deployment is blocked behind a question nobody has time to answer. So the drift gets absorbed into the apply, which is to say it gets accepted without being read.

Running the comparison on its own schedule, decoupled from any deployment, changes what it is. It becomes a report rather than an obstacle. Someone can look at nine differences on a Tuesday morning with no deadline attached, work out which are legitimate provider behaviour, which are last week's emergency fix, and which are genuinely unexplained. Only the last category needs alarm, and it is usually small.

The other half is making the emergency path produce evidence. If the break-glass procedure ends with a diff — even a rough one, even just the command that was run and the resource it touched — then folding the change back into the repository is a task with a definition, and it can be assigned. Without that, the reconciliation depends on somebody remembering, several days later, what they did while adrenaline was involved.

The recovery assumption

Here is what makes drift more than an aesthetic complaint. Almost every disaster recovery plan rests on a sentence like: we can rebuild the environment from the repository. That sentence is either true or it is not, and drift is precisely the measure of how untrue it has become.

An environment carrying a year of undocumented changes cannot be rebuilt from its declaration, because the declaration does not describe it. It describes an environment that would probably not work. Nobody discovers this until they need to rebuild, which is by definition the moment when discovering it is most expensive.

This is also why "we'll clean it up later" is a more serious commitment than it sounds. Later is when you need the thing to work.

Practical positions

A few positions hold up well.

Treat a non-empty plan as a defect with an owner rather than as weather. It does not have to be fixed today, but it should be explained, and the explanation should live somewhere other than in one person's memory.

Where a provider mutates your objects for legitimate reasons, tell your tooling to ignore those fields explicitly. An ignore rule that is written down is a decision; scrolling past the same diff every day is not.

Find the unowned resources on purpose, because nothing else will surface them. Then delete them deliberately, in daylight, rather than leaving them for someone to discover during an incident and be afraid to touch.

And keep the environments comparable. Once staging and production have diverged in ways nobody can date, staging has stopped being a test of anything. That is not drift causing an outage directly. It is drift removing the thing that was supposed to catch the outage.