Docker for Machine Learning
The 2026 Skills Guide

ML engineers need to understand Docker at the level of writing and optimising Dockerfiles, not just running pre-built images. Three core concepts:

• Images and layers — A Docker image is a stack of read-only layers, each layer being the result of a Dockerfile instruction (RUN, COPY, ADD). Layers are cached: if a layer and everything before it is unchanged, Docker reuses the cached layer rather than re-executing the instruction. Layer caching is the primary tool for making Docker builds fast.
• The Union File System — Docker uses a union file system (OverlayFS on Linux) to present the stack of layers as a single coherent filesystem. When a container runs, a writable layer is added on top; writes go to this layer and do not affect the image. This is why containers are ephemeral — the writable layer is discarded when the container stops, unless you use volumes.
• Build context — docker build . -t my-image sends the entire current directory to the Docker daemon as the build context. Large build contexts (model weights, datasets, node_modules) significantly slow builds. Use .dockerignore to exclude files and directories from the context.

Layer caching strategy — Order Dockerfile instructions from least to most frequently changing:

1. Base image (FROM pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime)
2. System packages (RUN apt-get update && apt-get install -y libgl1 && rm -rf /var/lib/apt/lists/*) — clean apt cache in the same RUN command to avoid bloating the layer
3. Python requirements (COPY requirements.txt . then RUN pip install --no-cache-dir -r requirements.txt) — separate from source code so it's only re-run when requirements change
4. Source code (COPY . .) — changes on every code edit; must be last

Non-root user — Run the container as a non-root user: RUN useradd -m appuser && chown -R appuser /app, USER appuser. Running as root inside a container is a security risk; most production environments require non-root containers.

Environment variables — Set ML-specific env vars in the Dockerfile: ENV PYTHONUNBUFFERED=1 (stream Python output immediately, not buffered — important for logging), ENV PYTHONDONTWRITEBYTECODE=1 (don't write .pyc files — saves disk space in containers), ENV TRANSFORMERS_CACHE=/model-cache (control HuggingFace model cache location for volume mounts).

The standard pattern for serving an ML model in production is a FastAPI application containerised with Docker. Key considerations:

• Model loading at startup — Load the model once at startup using a FastAPI lifespan context manager or the @app.on_event("startup") handler. Store the model in the app's state (app.state.model). Never load the model inside a request handler — this re-loads on every request, making inference 10–100× slower.
• Concurrency — FastAPI is async; model inference is typically synchronous and CPU/GPU-bound. Run inference in a thread pool: await asyncio.get_event_loop().run_in_executor(None, model.predict, data) to avoid blocking the event loop and degrading throughput for other requests.
• Health checks — Implement /health and /ready endpoints. /health returns 200 immediately (liveness). /ready returns 200 only after the model is loaded and ready to serve (readiness). Kubernetes uses these to manage pod lifecycle.
• Uvicorn/Gunicorn — Start FastAPI with uvicorn app:app --host 0.0.0.0 --port 8080 in the container. For CPU-bound applications, use gunicorn -w 4 -k uvicorn.workers.UvicornWorker to utilise multiple CPU cores. For GPU inference, typically one worker per GPU.