Docker GPU in 2026: Ship AI Apps with Compose

1. The Problem: It Works on My GPU (and Nowhere Else)

Shipping a GPU-powered AI app has three moving parts that must agree with each other: the NVIDIA driver on the host, the container runtime that exposes the GPU, and the CUDA libraries inside the image. When any one of them drifts — a tutorial pins an old nvidia-docker2 flag, the Compose file uses a 2021-era runtime key, the Dockerfile ships a devel image to production — you get cryptic errors at 2am and images measured in gigabytes.

The goal of this guide is a reproducible stack you can commit to git: one verified host setup, one compose.yaml with explicit GPU reservations, and one multi-stage Dockerfile that builds in devel and ships in runtime. Every command below was verified against the official NVIDIA and Docker docs.

2. Minimal Concepts You Actually Need

Only four ideas matter. Understand these and every GPU error message suddenly makes sense.

3. Tutorial: GPU AI App with Compose, Step by Step

We will go from a bare Linux host with an NVIDIA GPU to a running Python inference API with GPU access, in six steps. Tested flow, official commands only.

nvidia-smi

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu22.04 nvidia-smi

services:
  api:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - model-cache:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  model-cache:

# Stage 1: build native extensions (only if you compile anything)
FROM nvidia/cuda:12.8.0-devel-ubuntu22.04 AS builder
WORKDIR /build
COPY requirements.txt .
RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip \
    && rm -rf /var/lib/apt/lists/* \
    && pip3 install --no-cache-dir --prefix=/install -r requirements.txt

# Stage 2: ship lean runtime
FROM nvidia/cuda:12.8.0-runtime-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*
COPY --from=builder /install /usr/local
COPY . /app
WORKDIR /app
CMD ["python3", "main.py"]

docker compose up --build -d
docker compose exec api nvidia-smi
docker compose exec api python3 -c "import torch; print(torch.cuda.is_available())"

4. Common Errors (and the Exact Fix)

These five failures cover roughly 90% of Docker GPU pain. Each has one deterministic fix — no reinstall-the-world needed.

Conclusion

GPU containers feel fragile only when the three layers are improvised separately. With the system from this guide — driver plus toolkit on the host, explicit device reservations in Compose, pinned CUDA flavors, multi-stage images — your AI app becomes boring infrastructure: one repo, one compose up, same GPU behavior everywhere.

Start from the acceptance checklist above on your own machine, then pin your tags and commit. If you are deploying open models on that stack next, my GPT-OSS and Qwen3 guides show exactly what to run on top of it.

Sources

• NVIDIA Container Toolkit — Install Guide
• Docker Docs — Run Compose Services with GPU Access
• Docker Docs — Compose Deploy Specification (devices)
• nvidia/cuda — Docker Hub
• CUDA Containers — NVIDIA NGC Catalog