Nemotron 3.5 Lightning local requirements
Short answer: the official Q4_0 GGUF is 18.9GB and needs about 20.5GB at 4K context before the operating system's reserve. A 24GB GPU is a tight full-offload tier; 32GB GPU or unified memory is the comfortable local target.
Answer by hardware
These are memory-fit estimates, not measured speed claims. Longer context and GPU/CPU splitting need more headroom.
Official GGUF sizes and working memory
Download sizes come from the official ggml-org repository. Working set adds 5% loaded-weight overhead, the published hybrid-attention KV layout at 4K context, and a small runtime allowance.
| Quant | Download | Est. working set @4K | Practical tier |
|---|---|---|---|
| Q4_0 | 18.9 GB | ~20.5 GB | 24GB VRAM / 32GB Mac preferred |
| Q8_0 | 33.6 GB | ~35.9 GB | 48GB+ memory |
What the 1M context claim costs locally
Nemotron interleaves Mamba-2, MoE, and six full-attention layers. That keeps its KV cache far smaller than a conventional 30B transformer, but processing a million-token prompt still needs substantial temporary memory and time.
| Context | Attention KV cache | Q4 working set | Interpretation |
|---|---|---|---|
| 4K | ~0 GB | ~20.5 GB | Consumer-local range |
| 32K | ~0.2 GB | ~20.6 GB | Consumer-local range |
| 256K | ~1.5 GB | ~21.9 GB | High-memory experiment |
| 1M | ~6 GB | ~26.4 GB | Datacenter-class target |
Why “3B active” does not mean “3B memory”
Mixture-of-Experts routing selects roughly 3B parameters for each token, which can make decoding much faster than a dense 30B model. It does not discard the other experts: the complete 30B model still has to be stored and addressable. Active parameters describe compute per token; the GGUF file describes the weight memory.
See every phone and Mac result →Primary sources
FAQ
How much RAM does Nemotron 3.5 Lightning need?
The official Q4_0 GGUF is 18.9GB. Our short-context working-set estimate is about 20.5GB before operating-system reserve, so 24GB is the minimum sensible memory tier and 32GB is the comfortable recommendation.
Can Nemotron 3.5 Lightning run on a 24GB GPU?
Yes at the current Q4_0 quant, whose weights are 18.9GB. The estimated 4K working set is about 20.5GB before OS reserve, so close other GPU workloads and begin with a short context.
Can a 32GB Mac run Nemotron 3.5 Lightning?
Yes. The current Q4_0 quant fits at short context with useful headroom. A 24GB Mac is tighter after macOS reserve; 32GB is the safer target.
Does 3B active mean it uses only 3B-model memory?
No. The model routes each token through roughly 3B active parameters, which reduces compute, but all 30B parameters still have to live in RAM, VRAM, or a split CPU/GPU working set.
Can a phone run Nemotron 3.5 Lightning?
No current phone has enough app-usable memory for the official 18.9GB Q4_0 file plus runtime memory. The 3B-active label is not a phone-memory requirement.
Can local hardware really use the 1M-token context?
The model supports up to 1M tokens, but that is not the practical default for a consumer device. The attention KV cache alone grows to about 6GB at 1M, before prompt processing buffers and the 18.9GB Q4 weights; NVIDIA validates 1M on B200-class hardware.