SPM
SPM Thor

Configure SPM Thor

Weights, KV cache and runtime memory — on one GPU, split across several, or beyond one system.

Parameters
Architecture
Precision

Near-lossless for most models, and supported in hardware on current RTX PRO GPUs.

GPU memory per blade

As modelled: 16 × 3 GB GDDR7. Both are concept figures.

KV cache precision

Estimated GPU memory

82 GB

70B dense model at 8.2 bits per weight, 32k tokens × 1 session.

  • Weights 67 GB
  • KV cache 10 GB
  • Runtime 5.3 GB
One GPU · 48 GBNeeds about 82 GB; one GPU has 47 GB availableNeeds more than one GPU

Across SPM Thor · 48 GB per GPU

  • 2 GPUsSplit across 2 GPUs, tight
  • 4 GPUsSplit across 2 GPUs, tight · 2 copies
  • 6 GPUsSplit across 2 GPUs, tight · 3 copies
  • 8 GPUsSplit across 2 GPUs, tight · 4 copies

Split across 2 GPUs with tensor or pipeline parallelism — tight; a group of 4 leaves room for longer context and more sessions. Each GPU keeps its own share of the model, some memory is duplicated on every one, and activations cross PCIe Gen5 between them — the concept has no dedicated GPU-to-GPU link.

Configure 2 GPUs × 48 GB →Ask Sarja to check it

Assumptions

  • Attention: assumed 80 layers × 8 KV heads × 128 dimensions, typical of 70B models such as Llama 3.3 70B.
  • Every session can fill its whole context at once — real traffic usually needs less.
  • 1.5 GB of each GPU is kept for the driver and runtime.
  • Runtime overhead: 1 GB, plus 6% of the weights, plus 0.25 GB per session.
  • A split model rounds up to a group of 2, 4 or 8 GPUs and allows 10% for what parallelism duplicates.
  • Speed is not estimated. Nothing here is a benchmark.

What changes the real figure

  • The model's architecture — layer count and attention design set the size of the KV cache
  • The runtime and its settings (vLLM, llama.cpp, TensorRT-LLM and others allocate memory differently)
  • KV cache precision and how the runtime pages it
  • Context length actually used, not just the maximum
  • How many requests are served at once
  • Quantisation format and its block size
  • Framework overhead and CUDA context
  • Tensor or pipeline parallelism, which duplicates some memory on every GPU
  • Multimodal components such as vision encoders