Running a self-hosted Ollama Open-WebUI Docker stack gives infrastructure engineers and developers complete privacy over local Large Language Models (LLMs) without relying on external cloud APIs. However, moving beyond basic 7B-parameter chat demos into production coding, log analysis, or Retrieval-Augmented Generation (RAG) workflows quickly exposes severe memory bottlenecks. When model weights exceed discrete GPU Video RAM (VRAM) or when long conversation histories expand the Key-Value (KV) cache uncontrollably, Linux hosts suffer severe PCIe bus thrashing and abrupt container terminations triggered by the kernel Out-Of-Memory (OOM) killer.
- Architecting a Production Ollama Open-WebUI Docker Compose Stack
- Memory Budgeting: Model Weights vs. KV-Cache Growth
- Tuning GPU VRAM Layer Offloading and Context Windows
- Preventing Linux OOM-Killer Terminations and Swap Thrashing
- Hardware Selection Criteria for Local LLM Workstations
- Frequently Asked Questions
Achieving deterministic token generation rates requires treating local LLM inference as a strict memory-bandwidth and capacity budgeting problem. Below is an engineering blueprint for deploying containerized Ollama and Open-WebUI services, calculating exact VRAM/RAM footprints across context lengths, and isolating container memory ceilings.
Architecting a Production Ollama Open-WebUI Docker Compose Stack
Deploying both the inference engine and the web frontend inside a unified Docker Compose topology isolates Python dependencies, simplifies volume persistence for multi-gigabyte GGUF model weights, and enforces deterministic restart policies. Rather than exposing the raw Ollama REST API on port 11434 to external network interfaces, binding both containers to an internal bridge network ensures only the authenticated Open-WebUI frontend accepts client traffic.
services:
ollama:
image: ollama/ollama:latest
container_name: ollama-engine
restart: unless-stopped
volumes:
- ollama_models:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=15m
- OLLAMA_MAX_LOADED_MODELS=1
- OLLAMA_NUM_PARALLEL=2
- OLLAMA_FLASH_ATTENTION=1
- OLLAMA_KV_CACHE_TYPE=q8_0
mem_limit: 24g
memswap_limit: 24g
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
networks:
- llm_internal
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
ports:
- "127.0.0.1:3000:8080"
volumes:
- open_webui_data:/app/backend/data
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_AUTH=true
- ENABLE_RAG_WEB_SEARCH=false
depends_on:
- ollama
mem_limit: 4g
networks:
- llm_internal
volumes:
ollama_models:
open_webui_data:
networks:
llm_internal:
driver: bridge
Two environment variables in this stack prevent silent memory exhaustion during multi-turn sessions. Setting OLLAMA_FLASH_ATTENTION=1 enables Flash Attention kernels on supported NVIDIA (Turing/Ampere/Ada/Blackwell) and AMD ROCm GPUs, cutting quadratic attention memory growth. Pairing it with OLLAMA_KV_CACHE_TYPE=q8_0 quantizes the runtime KV cache from 16-bit floating point (FP16) down to 8-bit integers, slashing context memory consumption by nearly 50% with negligible perplexity impact. Additional environment flags and container parameters are documented in the official Open-WebUI Docker deployment documentation.
Memory Budgeting: Model Weights vs. KV-Cache Growth
A common capacity-planning mistake is sizing hardware solely around the static file size of a 4-bit quantized (Q4_K_M) model. During active inference, total memory consumption equals static model weights plus the CUDA/ROCm compute scratchpad (~0.8 GB to 1.5 GB) plus the dynamic KV cache, which scales linearly with the configured context window (num_ctx).
| Model Class (Q4_K_M) | Base Weights | Total RAM/VRAM (4K Context) | Total RAM/VRAM (32K Context, FP16 KV) | Total RAM/VRAM (32K Context, Q8_0 KV) | Target Hardware Tier |
|---|---|---|---|---|---|
| 7B – 8B (Llama / Qwen) | ~4.9 GB | ~6.2 GB | ~10.8 GB | ~8.4 GB | 8GB–12GB GPU or 16GB Mini PC |
| 14B (Qwen / DeepSeek) | ~9.0 GB | ~10.6 GB | ~19.4 GB | ~14.9 GB | 16GB GPU or 32GB Dual-Channel DDR5 |
| 32B (Qwen Coder / DeepSeek) | ~19.8 GB | ~21.8 GB | ~36.5 GB | ~29.2 GB | 24GB GPU + 64GB System RAM Split |
By default, Ollama initializes models with a conservative num_ctx of 2048 or 4096 tokens. When an engineer increases the context window to 32768 tokens inside Open-WebUI to paste full source files or system logs, the KV cache on a 14B or 32B model expands by 8 to 16 gigabytes instantaneously. If that surge exceeds free VRAM, Ollama falls back to partial CPU offloading.
Tuning GPU VRAM Layer Offloading and Context Windows
When a quantized model exceeds physical GPU capacity—such as loading a 32B parameter model onto a 12GB or 16GB graphics card—llama.cpp (Ollama’s underlying inference backend) partitions transformer layers across discrete GPU VRAM and host system DDR4/DDR5 memory. Because PCIe 4.0 x16 bandwidth (~32 GB/s bidirectional) is an order of magnitude slower than GDDR6X or HBM VRAM (500–1,000+ GB/s), every offloaded layer introduces measurable latency per generated token.
To prevent unstable automatic layer calculations or sudden OOM crashes during context expansion, compile a dedicated Modelfile with explicit hardware ceilings:
# Create a custom Modelfile with deterministic context and GPU layer caps
cat << 'EOF' > Modelfile.qwen32k
FROM qwen2.5-coder:14b-instruct-q4_K_M
# Cap context window to prevent unbounded KV-cache allocation
PARAMETER num_ctx 16384
# Explicitly pin transformer layers to GPU VRAM (adjust based on nvidia-smi)
PARAMETER num_gpu 38
# Match physical performance cores (exclude hyperthreads/E-cores)
PARAMETER num_thread 8
EOF
# Build the tuned model variant inside the running container
docker exec -i ollama-engine ollama create qwen-prod-16k -f /dev/stdin < Modelfile.qwen32k
Setting num_thread strictly to the physical core count—rather than total logical threads—prevents thread contention across hyperthreaded or heterogeneous P/E-core CPU architectures during CPU-offloaded matrix multiplication. All valid runtime directives and quantization parameters are detailed in the official Ollama Modelfile reference.
Preventing Linux OOM-Killer Terminations and Swap Thrashing
On Linux hosts running CPU-only inference—such as compact home servers covered in our Intel N100 homelab server setup guide—or hybrid GPU/RAM workstations, loading a model larger than available physical memory forces the kernel to swap pages to NVMe storage or invoke the OOM killer (SIGKILL). Notice in the Compose manifest above that mem_limit and memswap_limit are set to identical values (24g), disabling container swap usage entirely so disk I/O stalls never freeze the host SSH daemon.
Use the following telemetry commands on the Linux host to verify layer placement, real-time container RSS usage, and historical OOM events:
# Monitor active GPU VRAM consumption and compute utilization
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv -l 2
# Inspect live Docker container memory ceilings and resident set size
docker stats --no-stream ollama-engine open-webui
# Audit kernel ring buffer for past OOM-killer container terminations
sudo dmesg -T | grep -i -E 'oom-killer|killed process|out of memory'
Hardware Selection Criteria for Local LLM Workstations
When building or upgrading a workstation for local containerized inference, memory bandwidth dictates token-per-second throughput whenever models spill out of GPU VRAM:
- High-Capacity Dual-Channel DDR5 Kits: CPU-offloaded inference is strictly memory-bandwidth bound. Populating two matched modules (such as a 64GB DDR5-6000 CL30 kit in slots A2 and B2) maximizes dual-channel throughput while providing ample headroom for 32B models and host hypervisors. If your workstation exhibits prolonged cold-boot delays after upgrading to high-capacity modules, follow our guide to fix DDR5 slow boot memory training loops.
- High-Endurance PCIe 4.0 NVMe Storage: Cold-loading a 20GB to 40GB GGUF model archive into RAM or VRAM saturates sustained sequential read bandwidth. Utilizing a dedicated 2TB PCIe 4.0 NVMe SSD with onboard DRAM caching eliminates cold-start bottlenecks when switching between specialized coding and reasoning models.
For additional architectural guides on neural network deployment and edge inference optimization, explore our AI & Neural Networks engineering archive.
Frequently Asked Questions
Why does Open-WebUI ignore the context window size configured in Ollama?
Open-WebUI passes runtime inference parameters via API request payloads on every chat completion call, overriding default Modelfile values unless explicitly configured. Navigate to Admin Panel > Settings > Models (or per-chat Controls > Advanced Params) in Open-WebUI and set num_ctx explicitly so the frontend requests your target context length.
Can I run Ollama and Open-WebUI in Docker without a discrete GPU?
Yes. Remove the deploy.resources GPU reservation block from docker-compose.yml, and Ollama will automatically execute quantized GGUF models using AVX2 or AVX-512 CPU instructions. On CPU-only hosts, restrict models to 7B–8B parameters (Q4_K_M) with at least 16GB of dual-channel DDR4/DDR5 RAM for usable interactive speeds.
How do I calculate exact KV-cache memory overhead before loading a model?
In FP16 precision, KV-cache size scales approximately as 2 × layers × hidden_size × context_length × 2 bytes. Practically, an 8B model consumes roughly 128 MB per 1,000 context tokens in FP16 (~4.1 GB at 32K context), whereas enabling OLLAMA_KV_CACHE_TYPE=q8_0 cuts that footprint in half to ~2.0 GB.
For homelabs, remote testing, and staging environments, we run NVMe VDS instances on Erahost. Fast provisioning, European routing, and predictable I/O.
View NVMe Plans →