Articles

Health checks, restart policies and logging

When a container is unhealthy, what restarts it, and where the logs went.

Reading: 4 minServer & Virtualization

Article cover: Health checks, restart policies and logging

A running container is a process, and a process can be alive while the service it provides is not. Three jobs — configured separately — decide what happens next: a health check declares how to test the service, a restart policy decides whether a container that exits comes back, and a logging driver decides where its output lands. Confusing them is how a container ends up watched by nobody, restarted into a corrupted directory, or writing to a disk that fills.

Alive is not the same as serving

The health check exists because process liveness is not service liveness. The HEALTHCHECK instruction “tells Docker how to test a container to check that it’s still working”, and the failure it catches is the one a supervisor watching only the process cannot see: “This can detect cases such as a web server stuck in an infinite loop and unable to handle new connections, even though the server process is still running.” The process is present; the service is not. A health check makes that difference actionable.

   who watches the process                     where the output goes
   ───────────────────────                     ──────────────────────
   image:  HEALTHCHECK CMD ...                 stdout / stderr
        │                                           │
        ▼                                           ▼
   runtime: probe every --interval             logging driver
        │  timeout → SIGKILL; --retries         ├── json-file / local  → a file on this host
        ▼                                       ├── syslog / journald /
   state: starting → healthy → unhealthy        │   fluentd           → a daemon on this host
        │                                       └── awslogs / splunk / … → a remote service
        ▼
   restart policy: no · on-failure · always · unless-stopped

Declaring the check: the image and the runtime

A check can be declared in the image or supplied at run time. In a Dockerfile, HEALTHCHECK [OPTIONS] CMD command is the declaration; the options fix cadence and tolerance — interval, timeout, start period and retries. A probe that overruns its timeout is not merely counted as failed: “The process performing the check is abruptly stopped with a SIGKILL”. A checked container “has a health status in addition to its normal status”, beginning as starting and, “After a certain number of consecutive failures, it becomes unhealthy” — “It takes retries consecutive failures of the health check for the container to be considered unhealthy”. A slow boot deserves slack: “start period provides initialization time for containers that need time to bootstrap.”

The runtime can override this, and the override is blunt: set on Podman’s command line, “The default values are used even if healthcheck is configured in the image”, so an image that tuned its interval does not carry that tuning into a run-time command. Compose exposes the same check as healthcheck, which “declares a check that’s run to determine whether or not the service containers are” healthy.

What a restart policy actually promises

A restart policy decides what happens once the container exits. Docker provides restart policies “to control whether your containers start automatically when they exit”. no: “Don’t automatically restart the container.” on-failure[:max-retries]: “Restart the container if it exits due to an error” — but “The on-failure policy only prompts a restart if the container exits with a failure. It doesn’t restart the container if the daemon restarts.” always: “Always restart the container if it stops.” unless-stopped: like always “except that when the container is stopped (manually or otherwise), it isn’t restarted even after Docker daemon restarts”.

A policy is not a promise to restart everything. “A restart policy only takes effect after a container starts successfully”, and success means “the container is up for at least 10 seconds and Docker has started monitoring it” — a container that crashes instantly is deliberately not hammered in a loop. “If you manually stop a container, the restart policy is ignored until the Docker daemon restarts or the container is manually restarted”. Docker also warns against a second supervisor: “Don’t combine Docker restart policies with host-level process managers, as this creates conflicts.”

Order of startup: depends_on and what it does not cover

Restart acts on one container; Compose adds ordering between them. depends_on “expresses startup and shutdown dependencies between services”: “Compose creates services in dependency order” and “Compose guarantees dependency services have been started before starting a dependent service.” Started, though, is not ready: a web service can be created after its database and still fail to reach it. The long form adds a condition — service_started, service_healthy or service_completed_successfully — and with the healthy condition “Compose waits for healthchecks to pass on dependencies”, the health check used again as a startup gate. It sequences Compose’s own operations; it does not by itself make the application inside a started container ready.

Where the logs go: the logging driver

Docker keeps a per-daemon default: “As a default, Docker uses the json-file logging driver” — “The logs are formatted as JSON. The default logging driver for Docker.” The safer default is the local driver, where “Logs are stored in a custom format designed for minimal overhead.” The problem with the json-file default is its lifetime, not its format: “By default, no log-rotation is performed.” A container that logs heavily therefore “can lead to disk space exhaustion”; the remedy is the local driver, since “Use the local logging driver to prevent disk-exhaustion.”, or bounding json-file with max-size and max-file. Other drivers hand the log to a daemon on the host — syslog, journald, fluentd — or to a remote service — awslogs, splunk — so “Writes log messages to journald” instead of a file under the data directory.

Moving the output changes the failure mode. Delivery has two modes: the default “direct, blocking delivery from container to driver”, and “non-blocking delivery that stores log messages in an intermediate per-container buffer”, which “prevents applications from blocking due to logging back pressure” — at a cost: “When the buffer is full, new messages will not be enqueued.”

The chain: an unverified process, a restart with no plan

A container without a health check is a process nobody is verifying: it may be up while serving nothing, and if the policy is no, nothing will ever say so. A restart policy without data coherence is a way to corrupt better — always on a stateful service that exits on a failed write restarts it into the same broken state faster, and the json-file driver with no rotation fills the disk the restarted process then cannot write to. A health check makes “alive” mean “serving”; a restart policy, matched to how the service stores its data, decides whether a failure is recovered or repeated; a logging driver decides whether the record survives. Chosen together they are supervision; chosen separately they are three ways to lose the same container.

Level and prerequisites

L4 — operational supervision: what decides whether a container is working, what brings it back, and where the output is kept.

Prerequisites: the container runtime and its tools (the sibling sheet Docker, Podman and Compose); the image that carries a declared health check (the sibling sheet Container images, registries, volumes and networks); and the operating-system process model beneath every container (the L1 and L2 sheets).

Where to go next

  • Server & Virtualization — the area this sheet belongs to; its Operating systems node is operating systems.
  • The sibling sheet Docker, Podman and Compose — the tools whose flags this sheet uses.
  • The sibling sheet Container images, registries, volumes and networks — where the declared health check is stored.
  • The sibling sheet Rootless containers, capabilities and isolation — who the process runs as.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.