Beyond AI · 02 · Carved in silicon
Act I · The fetch · 01/20
Scroll
Clikiti Atelier
Tokens per second, one user0writing as you read this
Beyond AI · the tech that enables the new frontier · episode 2
Act I · 01 · The fastest writer

Seventeen thousand tokens a second.

This chip, the Taalas HC1, writes about 17,000 tokens a second for a single user. That's roughly a novel in under ten seconds. The best graphics processors, serving one person, manage a few hundred.

To see how, we start with a problem we met in episode 1.

02 · Previously

The math is cheap. The trip is not.

In episode 1 we followed one number from memory to math. Reading 32 bits from DRAM costs about 640 pJ; multiplying them costs about 3.7 pJ. Fetching is well over a hundred times dearer than thinking.

DRAM read 640 pJfloat multiply 3.7 pJHorowitz, ISSCC 2014
03 · Every word, every number

Sixteen gigabytes per word.

A small model like Llama 3.1 8B is 8 billion numbers. At 16 bits each, that's 16 GB. Writing one token means multiplying through all of them, so all 16 GB have to stream past the math units, again, for every token.

04 · The pipe

The limit is the pipe, not the math.

For one conversation, speed ≈ memory bandwidth ÷ bytes per token. An H100 moves 3.35 TB/s, so its ceiling is about 210 tokens a second on this model. A B200 at 8 TB/s: about 500. Real services land below that.

H100 ≈ 210 tok/s ceilingH200 ≈ 300B200 ≈ 50016-bit weights
05 · Sharing the trip

One read, a hundred conversations.

Data centres hide the wall with batching: read the weights once, use them for many users at the same moment. Total throughput soars and cost per token falls. But each person still gets their words one trickle at a time.

06 · Move the numbers closer

Hundreds of chips, one model.

Groq keeps weights in fast on-chip SRAM instead of HBM. But its original chip held only about 230 MB, so a single model is spread across many chips wired together. Fast, at about 600 tokens a second on this model, but it takes a room.

07 · Go big

A chip the size of a dinner plate.

Cerebras builds one chip from an entire silicon wafer, with vast on-chip memory, and reaches about 2,000 tokens a second on Llama 3.1 8B. Astonishing engineering. But the numbers still live in memory and still have to be fetched.

Act II · 08 · The question

What if there were
no memory at all?

09 · Carve it in

The numbers are the circuit.

Toronto's Taalas, founded by Ljubisa Bajic (a co-founder of Tenstorrent), took one model, Meta's Llama 3.1 8B, and built its weights into the chip's wiring as read-only circuitry, right beside the math. Nothing is fetched from outside.

no HBMno advanced packagingair cooled
10 · The music box

Same box. A new cylinder for every tune.

In a music box the comb, spring and case never change. The tune lives on the pins of a turning cylinder. Swap the cylinder and you have a new song, without building a new box.

11 · Two layers

Only two layers hold the model.

A chip is built in layers: transistors at the bottom, a dozen-plus layers of metal wiring above. Taalas says only two metal layers change from one model to the next. Those two masks are the cylinder.

TSMC 6 nm815 mm² die2 masks per model
12 · A fixed multiply

Times five is a shift and an add.

When a weight never changes, multiplying by it needs no general-purpose multiplier and no lookup. x × 5 = (x shifted left by 2) + x: two wires and an adder, laid down once. Bajic says one 4-bit parameter is stored and multiplied on a single transistor; the exact circuit isn't public.

x · 5 = (x ≪ 2) + x
13 · Squeezed to fit

Eight possible values.

To fit 8 billion weights on one die, most are stored in a custom 3-bit format (just eight values), with some at 6 bits. Taalas acknowledges "some quality loss". The chip has 53 billion transistors, with SRAM for the conversation's working memory and small fine-tuning add-ons.

3-bit base · some 6-bit53B transistorsLoRA adapters in SRAM
14 · The race

About 50× a B200, for one user.

H200~230
B200~350
Groq~600
SambaNova~900
Cerebras~2,000
Taalas HC1~17,000

Tokens per second for one user on Llama 3.1 8B, as compared by Taalas (GPU figures from Nvidia and Taalas; others from Artificial Analysis). EE Times measured 15,000+ on the public demo.

Company claims also include 10× less power and 20× cheaper to build. How many users one chip serves at once isn't disclosed, and part of the gap comes from the 3-bit compression.

15 · Cool by design

No towers. No plumbing.

Remember the chip from episode 1? An H100 draws up to 700 W through a stack of layers: package substrate, silicon interposer, towers of HBM. Its successors run hotter (a B200 up to about 1,000 W), and racks like Nvidia's GB200 NVL72, at roughly 120 kW, are liquid cooled by design.

HC1 has no HBM and no advanced packaging. Taalas quotes about 200–250 W per card and about 2.5 kW for a ten-card server, cooled by ordinary air. Fetch less, and there's less heat to carry away.

H100 · up to 700 WGB200 NVL72 · ~120 kW · liquidHC1 · ~200 W · air
Act III · 16 · The catch

Frozen forever.

The base model can never change. Small LoRA adapters can tune it, but a new model means a new chip. Llama 3.1 was already 18 months old when HC1 launched. Compression costs quality, no benchmark scores have been published, and room for long conversations is tight.

17 · Print, don't patch

Two months from weights to chips.

Because only two masks change, Taalas says a finished model becomes working cards in about two months, with the model-to-circuit step taking about a week. The bet: when a model is good enough, freeze it, and print the next one when it's better.

18 · We've seen this before

Freeze what's settled.

Bitcoin mining moved from GPUs to custom chips from 2013, once its algorithm stopped changing. 1980s gate arrays were customised only in their top metal layers. The pattern: flexible chips for what's new, fixed silicon for what's settled.

19 · The giants

One chip reads. Another writes.

Nvidia licensed Groq's technology in a deal reported at about $20B and now pairs Groq LPUs with its GPU racks. On 6 August 2026 AMD agreed to buy Taalas, about $2.9B including contingent payments, per AMD's filing. The deal hadn't closed at the time of writing.

GPUs read the promptspecialised chips write the answer
20 · What comes next

Stop treating it as software.

Taalas has promised a frontier-class model on a second chip, HC2, spread over a few dozen chips. As of November 2026 it hasn't been shown publicly. If it works, the fastest way to run a settled AI may be to carve it into silicon.