MoE models on consumer GPUs: the A3B lane, worked in the open

A 35B-A3B model stores about 35 billion parameters in memory but computes through only about 3 billion per token. Memory bills the total. Speed bills the active slice. That split is why this class of sparse model runs on one 24 GB consumer card. A dense model of similar reach cannot match its speed on the same silicon. Measured receipts put the class at 104 tokens per second on a 32 GB AMD card in our August 2026 runs. A tuned RTX 3090 reached 157.66 in an owner-run benchmark updated October 2026. A September 2026 one-card study prices the lane's break-even at 4.3 percent of a card's day.

All amounts on this page are United States dollars written as bare numerals, and every number carries its source and date. Every table is static. There is no calculator widget and no script running in your browser.

Last verified: 2026-10-26

The split that makes the lane work

A mixture-of-experts model, or MoE, keeps its feed-forward layers as banks of small specialists instead of one big block. A router picks a few of them for each token. The rest sit idle in memory. The public ancestor of this class is Qwen3-30B-A3B. The Qwen team documents it on its model card: 30.5 billion parameters in total, 3.3 billion activated, 128 experts with 8 active per token. The 2026 generation of the class carries about 36 billion total at the same roughly 3 billion active. Its members, Qwen 3.6 35B A3B and Ornith 1.5 35B A3B, hold 256 experts. Eight route per token, plus one shared expert.

Two formulas govern everything on this page.

The first formula prices memory, and memory only cares about the total.

weights in GB  =  total parameters  x  bits per weight  /  8

A 4-bit copy, the Q4_K_M kind most people run, stores about 4.84 bits per weight. Work it for the class: 36 billion parameters times 4.84 divided by 8 is about 21.8 GB. A measured run on a 24 GB card peaked at 21,652 MiB with 8,000 tokens of context already cached. That is the same number within the overhead. The dense contrast class, a 27B model at about 27.8 billion weights, works out to about 16.8 GB at the same bits. Figure about 19.9 GB with a working context on the fit-database basis. Both fit a 24 GB card. Memory does not separate them.

The second formula prices speed, and speed only cares about the active slice. Generating one token means reading the weights that token touches from video memory. Decode is bandwidth-bound: the card reads, the chip waits, and the token appears.

tokens per second  =  effective bandwidth  /  bytes read per token

The dense 27B reads nearly all of its roughly 16.8 GB of 4-bit weights for every token. The A3B model reads only the small expert set the router picks, roughly a tenth of the weight bytes, plus its attention state. On the same card, in the same software, the A3B class wins. It generates roughly four to five times the tokens per second of the dense model. That ratio is the whole consumer case for the class. The receipts section below shows it on real cards.

One honest wrinkle. The active ratio is about 9 to 1 in favor of the MoE. The measured speed ratio runs 3.8 to 5.4 to 1 under matched conditions. Expert routing, the gather of chosen experts, and the shared attention layers eat the difference. Anyone promising the full 9 to 1 in tokens per second is quoting math, not a machine.

What fits where: memory by card tier

The tier table below works the first formula for the classes people actually ask about. Quants are compressed copies: Q4 means about 4 bits per weight, Q8 about 8, and the lower the bits the smaller and slightly rougher the copy.

Three readings fall out.

First, 24 GB is the boundary that matters. The 35B-A3B class at Q4 lands at 21 to 22 GB of weights. A 24 GB card holds the whole model with room for a working context. OpenAI built the smaller sibling of its open pair for exactly this shape. The gpt-oss cards document gpt-oss-20b at 21 billion parameters with 3.6 billion active, running within 16 GB. The bigger gpt-oss-120b, at 117 billion and 5.1 billion active, targets a single 80 GB part.

Second, below 24 GB you are not fitting the class, you are streaming it. A 12 GB card runs the 35B-A3B at about 38.9 tokens per second with its experts in system RAM. That is a measured figure. The CPU-only floor for the class sits near 12 to 15 tokens per second. It works. It is a different machine.

Third, above 24 GB the extra memory buys context and heavier quants, not a bigger model class. That holds until about 60 GB, where gpt-oss-120b at 4-bit finally fits. The bandwidth does the speed. The memory just holds the payload.

One trap deserves its own line. A desktop session on the same card eats about 4.4 GB. At 24 GB that is the difference between the whole model resident and a load failure. Run the model headless, or give the display its own cheap card, before you blame the card.

Measured speed on real cards

Every row below is a receipt, not an estimate. Our own rows come from two dated hardware studies, August and September 2026. The rest are named forum or publisher-side benchmarks with their conditions stated. Speeds are single-user output in tokens per second at 4-bit-class quants unless the row says otherwise.

Read the dense column against the A3B column under matched conditions. On the R9700, tuned against tuned, it is 147.8 to 156.3 against 29.0 to 32.8, or 4.5 to 5.4 to 1. On the B70 it is 76 against 20, or 3.8 to 1. On the 3090 it is 157.66 against 34, or 4.6 to 1. The class turns a memory card into a fast one without changing the silicon.

Two more receipts sharpen the picture.

The tuned-versus-default gap is real and large. Default installs, with a full-precision KV cache and no flash attention, leave the class at 40 to 70 tokens per second on 24 GB cards. The same cards reach 100 to 158 once the cache is quantized and the flags are set. Half the posted speed of this class is settings, not hardware.

The offload path is a straight line, not a cliff with a safe edge. A measured sweep in the owner-run benchmark offloaded expert layers one at a time on a 24 GB card. Every layer cost about 0.53 milliseconds per token, from the first to the fortieth. At the bottom of that line, with every expert in system RAM, the 24 GB card made 36.1 tokens per second. A 12 GB card set up the same way did 38.9. The expensive card lost. Once the experts stream from RAM, the RAM sets the speed. The lesson is blunt: buy the bigger card to keep the model on it, not to stream faster.

The economics: when one card pays for itself

Speed receipts answer whether the class runs. The money question is when the card that runs it earns its price back. A September 2026 one-card study works the math on two lanes at matched rates. It prices the A3B class against the dense 27B, both from this platform's model rows below.

The break-even works in three named variables, then numbers.

revenue per day at full use   R  =  out-tokens per day x P_out  +  in-tokens per day x P_in
cost per day                  C  =  card price / amortization days  +  power  +  share of the host
break-even use                u* =  C / R

On the study's card, a 32 GB part at 1,299 list amortized over 24 months with power and a host share, C is about 2.33 per day. The A3B lane at its measured 176 tokens per second aggregate can emit 15.2 million output tokens a day. On the ornith row's fence rates, with the study's 3-to-1 input blend, that is 54.7 per day. So u* is 2.33 divided by 54.7, or about 4.3 percent. The dense 27B lane on the same card, at its measured and estimated 3.5 million output tokens a day, prices at 12.4 per day on the same rows and blend, and needs about 19 percent.

That pair of numbers is the study's headline. The A3B lane is the only one of the two where a single consumer card breaks even inside the low-use band a real project starts in. Payback at a quarter use lands near 3.3 months for the A3B lane. The dense lane needs about 16 months. One sensitivity line, as the law of this page: halve the output rate and the break-even doubles, to roughly 8.6 percent. That still sits far under the dense lane's bar.

Cheaper cards move the bar the same direction. The study's value menu puts a 24 GB card at 599 with an estimated 3.4 percent break-even on the same lane, because the memory holds the class and the revenue does not care what the card cost.

Now the hosted side, as named rates read today, October 26, 2026. One multi-provider router's catalog lists the Qwen 3.6 35B A3B class at 0.10 per million input tokens and 0.95 per million output. DeepInfra's own page for the same model lists the same standard tier, 0.10 and 0.95. Its slower flex tier runs 0.08 and 0.76, its faster tier 0.15 and 1.425. The smaller gpt-oss-20b sits at 0.018 and 0.09 on the same router. A local 24 GB card at 101 to 158 tokens per second can emit on the order of 9 to 13 million tokens a day at full tilt. Priced against the hosted 0.95 output rate, that output is worth thousands per day from a card that costs about 2.33 per day to own on the study's 32 GB card basis. That is the same break-even math stated from the other side. For bursts of a million tokens a month, hosted wins on simplicity and costs a few dollars. For steady millions a day, the card wins. The receipt tables above tell you exactly how fast yours will emit.

The rented middle path still exists. Community clouds list card-hours from 0.015, with a 4090-class hour at 0.160 at the lowest tier on Salad's own pricing page today. Renting fits the burst shape and dodges the idle card bill, at the price of preemption and setup. The robot side of the own-versus-rent call sits in the robot-hour cost guide. The entry-level budget version lives in the hobbyist compute budget.

What the forums keep asking

The demand for this page is visible in the threads. On r/LocalLLaMA and its neighbors, the same three questions recur through 2025 and 2026.

The first is the fit question, phrased as hardware anxiety: will this model run on my card. The 24 GB fit threads benchmark everything that fits. They keep finding the same boundary this page derives. The second is the speed question, phrased as disappointment: why do I get half the posted tokens per second. The answer in nearly every thread is the same three settings: cache precision, flash attention, and offload depth. The tuned-versus-default gap above is the receipt. The third is the CPU question, asked with hope: can I run the A3B class without a card at all. Threads through 2025 report 12 to 15 tokens per second on pure CPU. The offload sweep explains why the ceiling sits there. The class streams from RAM at RAM speed.

None of these threads carry a worked cost answer. The economics section above is the part the forums keep asking for and never get.

Decision rules

Run it well: the three settings that matter

The software side is settled enough to state plainly. The class runs on llama.cpp and its packaging in Ollama and LM Studio, on vLLM for multi-stream serving, and on MLX on Apple silicon. The model files come from the publisher pages and the standard quant libraries. The publisher card for the class ancestor is the reference for its shape and its license. Both families, Qwen and gpt-oss, ship Apache 2.0.

Setting one: quantize the KV cache. The full-precision default is the tax. A q8_0 cache is reported 12 to 38 percent faster while using less memory, one of the rare free wins. Setting two: turn flash attention on. Setting three: touch expert offload only when the card cannot hold the weights. The sweep says every layer offloaded bills the same 0.53 milliseconds forever. A card that holds the model should hold all of it.

If you would rather not own the hardware at all, the hosted routes are on the models index and the pricing page. You can open an account at signup. The 3090 rental lane and the 3060 budget lane each have their own guide: the 3090 economics and the 3060 budget lane. For the speech side of the local-or-hosted call, see the TTS break-even guide.

Frequently asked questions

What does A3B actually mean?

It is two numbers in one name. The part before the letter A is the total weight count in billions, 35B in this class. The part after is the active count, about 3 billion computed per token. The Qwen card for the class ancestor documents the pattern: 30.5 billion total, 3.3 billion active, 128 experts with 8 firing. Memory bills the first number. Speed bills the second.

Can I run a 35B-A3B model on a 24 GB card?

Yes, fully. At the standard 4-bit quant the weights run 21 to 22 GB. A measured run peaked at 21,652 MiB with 8,000 tokens of context cached. Two conditions matter. Run the card headless, or free about 4.4 GB if a desktop sits on it. And keep the experts on the card: every layer pushed to system RAM costs about 0.53 milliseconds per token.

Is a mixture-of-experts model faster than a dense model on the same card?

Yes, by a measured 3.8 to 4.6 times on the same hardware in the receipts above. The honest ceiling sits lower than the math suggests. The active ratio is about 9 to 1, but routing and expert gathering eat part of it. Anyone quoting the full 9 to 1 as a speed promise is quoting math, not a machine.

How fast is the class on an AMD card?

The receipts: 104 tokens per second stock on a 32 GB Radeon AI Pro R9700 in our August 2026 runs. A tuned Vulkan setup reached 147.8 to 156.3. The dense 27B ran 17 to 20 single-card stock, 23.4 on two cards, and 29.0 to 32.8 tuned, on the same silicon. AMD is not the second-class path this question assumes. The same records show the class running the standard llama.cpp routes.

Does the MoE class save money versus an API?

It depends on volume, and the break-even section works it. At hosted rates read on October 26, 2026, two catalogs price the class at 0.10 per million input and 0.95 per million output. A burst of a million tokens a month costs about a dollar, and hosted wins there. A card emitting millions of tokens a day at 104 to 158 tokens per second prices out far below hosted. The one-card study puts its break-even at about 4.3 percent of the card's day.

Why not just run gpt-oss-120b, which is bigger?

Bigger total does not mean a card can run it. At 4-bit the 120B model carries 60.4 GB of weights, and its own card names the target as a single 80 GB part. Its active count, 5.1 billion, sits only a little above this class. Its smaller sibling, gpt-oss-20b at 21 billion total and 3.6 billion active, is the one built for 16 GB machines. The A3B lane is the fit point between them on consumer cards.

The lane, in one line: memory bills the total, speed bills the active slice. A 24 GB card holds the whole 35B-A3B class at 101 to 158 tokens per second. The receipts, the fit table, and the break-even sit above, re-runnable with a pencil. Rates for every hosted lane sit on the pricing page. Sign up, create a key, and the only variable left is your own card.

Last verified: 2026-10-26