Read the article for free here.

Someone ran Qwen3.8-Flash Next last week on a single RTX 4090, using 5.95GB of VRAM. Not a shrunk-down copy, the full download. They asked it for the capital of France, capped the answer at 8 tokens, and it said Paris.

A model is a stack of layers (48 here). Normally all 48 sit in memory at once. This method keeps one: load a layer, use it, drop it, load the next. That is where the 5.95GB comes from. Writing the next word means going through all 48 again, from the disk. The post reports the memory and never the speed.

Guides written for consumer cards say a 4090 cannot run this model at all. One person is getting 21 tokens a second out of a single 24GB 4090, with a 250K context. Another is running it on a pair of desktop RTX 3080s at about 16 tokens a second, faster than you can read. The reason is one piece of this model that is not weights at all.

The Setup That Works

The baseline here is two RTX 3080s and 64GB of DDR4, the cheapest setup anyone has reported working.

The 4-bit version of Qwen3.8-Flash Next is a 93GB file. That is more than the memory on that machine, and it runs because a chunk of the file never has to be loaded. That chunk is a lookup table. The model reads one line of it for each word it writes, so it can stay on the SSD and get fetched a row at a time. Only the rest has to sit in fast memory, meaning RAM or VRAM, and that comes to about 55GB.

There are several of these files at different compression levels, and picking one is not about whether it fits. It is about how much has to stay in memory, and how much of the model's judgment you give up to shrink it. KL divergence puts a number on the second part: how far the compressed answers drift from the original, where lower is closer. Take the 4.27 bits per weight (bpw) Q4_K_M build from the AtomicChat/Qwen3.8-Flash-Next-GGUF repository. Its divergence is 0.0842, which is close.

Launch it like this:

llama-server -hf AtomicChat/Qwen3.8-Flash-Next-GGUF:Q4_K_M --jinja -c 32768 -ngl 99 -fit off --mmproj mmproj-Qwen3.8-Flash-Next-F16.gguf

-ngl 99 puts every layer it can on the GPU. This works on 10GB cards because the engine will automatically spill what does not fit into system RAM. --jinja is needed for the tool-calling template. The GGUF conversion keeps image input working and --mmproj loads the vision adapter for it.

By default llama.cpp checks how much VRAM is free at launch and grabs more than the run will use. Pinning the context with -c and passing -fit off stops that. The 32768 value sets a 32K context window, which costs about 0.8GB of memory.

The lookup table, called the n-gram layer, wants "4-bit minimum", because its reads are random and quantizing it harder damages the model.

The table below uses Unsloth's dynamic (UD) quants, which are a different build family, so treat the numbers as indicative. The 2x RTX 3080 row is our 64GB baseline.

Generation speed lands in low twenties on standard consumer machines and pushes higher on workstation cards.

Why Every Requirement You've Read Disagrees

The official answer is datacenter hardware: two GB300s at minimum, four recommended. A 4090 or a 5090 has no supported path at all, and neither does a single A100 or H100.

One person is getting 21 tokens a second out of a single 24GB 4090 at a 250K context (@analogalok on X). He did not say how much system RAM sat behind it, which is the part you would need to match. 12 more reports of it running on desktop hardware sit in the community speed thread.

They are answering different questions. A serving cluster has to keep hundreds of people supplied at once, so every part of the model stays in the fastest memory available and the hardware gets sized for that. One person at a desk needs only whatever a single word touches, and can leave the rest on the disk.

That is why the "6B active parameters" line on the model card is a trap. It counts what the model computes with, not what has to be loaded.

Part of This Model Is a Dictionary

Most of a model is arithmetic. To write one word it multiplies its input against a block of numbers, and it uses all of them, every time. This is the reason those numbers have to sit somewhere the card can reach in nanoseconds.

The n-gram table is not that. It is a list of 20 million word pairs, and the model looks up one line per word it writes. That line is about 2.7KB, so at the speeds people are getting the table is read at roughly 1MB a second, and any NVMe drive does thousands of MB a second.

So 51B of this model's parameters can sit on the disk, and the hardware requirements in circulation count them as though they had to be in memory. That is most of the gap between 93GB and what your machine has.

One user is running it on two 10GB RTX 3080s with 64GB of system RAM, with the table on an NVMe drive. The official server does the same, offloading to system RAM instead of disk.

What Has to Be Awake at the Same Time

To work out whether a model fits your machine, take the download size and subtract the parts that only get looked up. Here that is the n-gram table. What is left is the part every word runs through, and on top of that you add room for the conversation history you want to hold. That total is the number your machine has to meet.

The baseline box has 84GB of fast memory, counting its system RAM and its two cards together. The weights need 54.5GB. A full 262K conversation needs another 6.5GB, which leaves about 23GB for the operating system and everything else. At 32K, which is what the launch command above sets, the history costs 0.8GB.

This only works on a model that keeps its lookups separate from its arithmetic. A dense model still needs every weight in fast memory, because every word multiplies against all of them.

The Long Context Comes Cheap Here

To keep a conversation going a model has to hold on to every word it has already seen, and that store grows with each new word. It is why a long session eats memory even though the model itself never changes size.

Three of every four layers here keep a running summary of fixed size instead, so those layers stop growing however long the conversation runs. Only the fourth keeps the growing kind.

The result is that the full 262K window costs about 6.5GB, and 32K costs 0.8GB. The dense 27B in the same family needs about 67GB for that same 262K window, on a model a fifth the size. You can put a whole book in the prompt here without running out of room for the model itself.

Speed is the part that does decay. Generation drops from 60 tok/s to 30 tok/s by 128K on an RTX 6000 Pro paired with an RTX 5090. Four 5090s running vLLM held 188–206 tok/s flat out to 224K, using a feature the first GGUF builds could not export.

Why the 6GB Version Crawls

The 5.95GB run used AirLLM, which streams layers from the disk to the GPU one at a time instead of holding the weights in memory. It reads the full BF16 checkpoint, so you need 335GiB of free disk and a transformers build from git before you start.

Nobody has published a speed for this model on that setup, and AirLLM does not publish throughput figures at all. The closest measurement is a smaller model: a 34B, 68GB across 59 layers, on a 16GB Mac, one prompt at a time, at 35.4 seconds per token. Two tokens a minute.

The layer read costs the same whether one prompt is waiting on it or a hundred are. Batching 500 prompts on that same 34B setup brought the cost down to 4.85 seconds per token. It is a tool for queues that run overnight, not for anything you are sitting in front of.

if this helped, clap 👏 so others can find it too — and if you want the shorter, sharper cuts of stuff like this, I also post notes on substack. Further reading:

Qwen 3.8 27B Runs Faster on a 3090 Than a 4090. The GPU Isn't Why: Same model family, previous generation, where the inference stack beat the card.

Kimi K3 Is the Biggest Open Source Model Ever. Almost No One Can Run It.: The inverse case: open weights nobody can run, at 2.8 trillion parameters.

Running Local LLMs in 2026: 5 Things That Actually Matter: The five ordinary decisions that decide whether local inference is usable.

The 2026 Memory Crisis Killed Most Local LLM Builds. Here's the Map That Still Works.: What to buy at every budget after DRAM prices broke the rankings.

Why vLLM and SGLang Are Replacing Ollama for Agentic Workflows: When to leave llama.cpp behind for a real serving stack.