Articles

Detecting configuration drift

When reality diverges from the declaration: detecting, classifying and closing the gap.

Reading: 4 minServer & Virtualization

Article cover: Detecting configuration drift

An environment is described twice: in the declarations meant to produce it, and in what is actually running. Drift is the distance between the two. It is less a fault in the tooling than a fact of operating — people type into a console, an upgrade moves a default, an incident is fixed first and folded back later. This sheet is about spotting that gap and closing it deliberately.

   declared ──┐
              ├─► compare (plan / --check) ─► difference ─► classify
   actual ────┘                                                 │
                                             ┌─────────────────┴─────────────────┐
                                         expected                          unexpected
                                   (approved deviation,                  (hand edit, changed
                                    transient window)                     default, leftover)
                                         │                                     │
                                         ▼                                     ▼
                                   record, no action                 who may close it?
                                                                     change control element
                                                                     decides:
                                                                       ├─ apply the declaration
                                                                       └─ approve a deviation
                                         ▲                                     │
                                         └──────── verify ◄────────────────────┘
                                              (re-run: the difference is gone,
                                               or has become expected)

What drift is, and why it is normal

Drift is any difference between the declared configuration and the actual state. Three origins account for most of it. First, a hand edit: a Proxmox VE host’s network may be set “either via the GUI, or by manually editing the file /etc/network/interfaces”, never passing through the declaration. Second, an upgrade that moves a default: the ZFS ARC limit used to match “ZFS’ default value which was either 62.5% starting on ZFS 2.3.0 or 50%”, and older installations needed “manual steps” to converge — nothing was edited, yet declared and actual disagree. Third, the emergency: a declarative tool documents refreshing state as reconciling changes made “outside of the usual workflow”. A manual fix during an outage is drift by design. (That this drift is normal, not exceptional, is this sheet’s reading.)

The plan is a difference, recomputed

Terraform states the detection step plainly: “The terraform plan command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.” A plan “Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date”, “Compares the current configuration to the prior state and noting any differences”, and “Proposes a set of change actions that should, if applied, make the remote objects match the configuration”. “The plan command alone does not actually carry out the proposed changes”, and when nothing differs the tool “will report that no actions need to be taken”. Refresh-only mode isolates the comparison: it “creates a plan whose goal is only to update the Terraform state and any root module output values to match changes made to remote objects outside of Terraform”.

Comparing without writing

Ansible frames detection as an execution mode. “In check mode, Ansible runs without making any changes on remote systems. Modules that support check mode report the changes they would have made. Modules that do not support check mode report nothing and do nothing.” That last sentence is the caveat the later classification rests on: silence can mean agreement, or a blind spot. The mode admits its limits — “Check mode is just a simulation. It will not generate output for tasks that use conditionals based on registered variables (results of prior tasks)” — and with diff mode it adds the comparison itself: “In diff mode, Ansible provides before-and-after comparisons.” Plan and check mode ask one question without touching the target: what, if anything, differs?

Policy as a rule you can re-run

Not every target comes from one tool, so not every difference is caught by one diff. Some divergence is better written as a rule. Open Policy Agent is a “general-purpose policy engine that unifies policy enforcement across the stack” and “decouples policy decision-making from policy enforcement”: a system “queries OPA and supplies structured data (e.g., JSON) as input”, and “OPA generates policy decisions by evaluating the query input against policies and data”. Its compliance use is drift work — a policy enforced at provisioning and re-checked to “periodically audit the system to find servers that violate the policy”. Written as rules “called violation that generate a set of servers that are in violation”, the result is a list “handed back to the infrastructure as code system to present to the user”.

Noise and the divergence that matters

Most differences are not drift, and treating them all as drift is its own failure. The formal dividing line is approval: the configuration-settings control requires an organization to “Identify, document, and approve any deviations from established configuration settings”. An approved deviation is a named, recorded, intentional difference; an unapproved one is drift. The tools show why a clean run proves less than it seems: a check run that reaches a module without check-mode support will “report nothing and do nothing”, so the absence of a reported change is not evidence of the absence of change. Classify by ownership and approval, not by size. (That test is this sheet’s reading.)

The fix that makes it worse

The instinct when something has drifted is to correct the target. That is how drift propagates: the correction exists only on the machine. Proxmox VE’s interface-pinning tool updates several files automatically, yet “interface names contained in the Firewall Datacenter configuration ( /etc/pve/firewall/cluster.fw ) are not automatically updated” — a partial fix leaves one site out of step. The same trap has a tool form: targeting a single resource is documented as leading “to undetected configuration drift and confusion about how the true state of resources relates to configuration”. Repairing the one object a plan complained about hides the rest; the durable move is to fix the declaration that produced it.

Reconciling, and who may close the gap

Closing the gap is a loop, not an event: refresh the state so it reflects reality, compute the difference, classify it, and dispose of it — by applying the declaration or by recording an approved deviation. Proxmox VE makes the controlled path concrete: a GUI apply “will move changes from the staging interfaces.new file to /etc/network/interfaces and apply them live”, and even a hand edit goes through an explicit command, ifreload -a. Who decides? Not whoever noticed the difference. Change control requires an organization to “Monitor and review activities associated with configuration-controlled changes to the system” and to “Coordinate and provide oversight for configuration change control activities through” a named element; a configuration changed “in an unauthorized manner” triggers a response, and personnel are told “when there is an unauthorized modification of a configuration item”. The authority that closes a gap is the one that owns the declaration.

Level and prerequisites

L5 — detecting and closing the gap between a declared configuration and the running system: why drift is normal, how a comparison that writes nothing reveals it, how a policy turns a rule into a repeatable check, how approval separates noise from real divergence, and who may reconcile. It sits above the declarative mechanics — idempotence, state and the plan/apply cycle belong to the companion infrastructure-automation sheet, used here as the detector — and above the base hardening of a single host, owned by the baseline hardening sheet. It is L5 because it reasons about a fleet’s declaration-to-reality relationship over time, not one machine’s settings.

Where to go next

  • Server & Virtualization — the area this sheet belongs to.
  • The companion sheet that declares infrastructure: idempotence, state and the plan/apply cycle.
  • The sheet that hardens a single host’s baseline.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.