Cost Per Token Is Set Outside the Chip

cost-per-token-is-set-outside-the-chip · conviction medium · status open · horizon 2027-2028 · as of 2026-08-04

The market prices AI compute on accelerator performance. But the cost of a served token is set by a chain that runs from copper-clad laminate to the interconnection queue, and the accelerator is only one term in it. Every link is measurable, most are not priced, and the binding one moves between them.

Rests on filed figures, not on modelled shares. 15 premises (15 entity); no derived cell is involved, so the undisclosed supply weights that put a range on other pages in this bank cannot move this one.

Exhibits

Exhibit 1Relative performance, indexed to 100How the names in this thesis have traded against SOXX.

Series available as data/cost-per-token-is-set-outside-the-chip.csv

Exhibit 2Who pays CoWoS advanced-packaging capacity, and who keeps the moneyCapturers average 47.1% operating margin against payers' 44.6% — the owners of the scarce thing capture the rent, as expected.

Green/blue = model marks it as CAPTURING the rent (unbound and supplies the scarce good); faded = PAYING it (bound severe or moderate). Operating margin, live.

Exhibit 3What the conviction is actually made ofEach premise and the number it composes to. A conjunction of plausible premises is far weaker than any of them.

Materials and packaging set a floor on how much compute fits in a package at all

42% if the 3 gates are independent, 70% if they move together. They are claims about one industry, so the truth is between and nobody can say where. Treat this as an ordering device rather than a calibrated probability — the ranking of premises is the information, not the level.

Weakest link: Elite Material Co., Ltd. (EMC) at 0.70 — M9-grade CCL with quartz-glass reinforcement is the substrate material for the Kyber midplane at ~1 square metre and ~78 layers, where yield compounds

The package perimeter forces a trade between power and bandwidth that neither side wins outright

31% if the 3 gates are independent, 60% if they move together. They are claims about one industry, so the truth is between and nobody can say where. Treat this as an ordering device rather than a calibrated probability — the ranking of premises is the information, not the level.

Weakest link: Coherent Corp. at 0.60 — Co-packaged optics needs perimeter it does not control, so the optics opportunity is gated by a power-architecture decision made by the accelerator ve

Memory architecture moves the coefficient more than a hardware generation does

48% if the 3 gates are independent, 75% if they move together. They are claims about one industry, so the truth is between and nobody can say where. Treat this as an ordering device rather than a calibrated probability — the ranking of premises is the information, not the level.

Weakest link: Long-context inference (KV-cache growth) at 0.75 — Agentic workloads run 142k input tokens per turn over a median 65 turns, which is ~71x a chat turn on context alone.

Energy per token converts the whole chain into megawatts, and megawatts are queued

39% if the 3 gates are independent, 70% if they move together. They are claims about one industry, so the truth is between and nobody can say where. Treat this as an ordering device rather than a calibrated probability — the ranking of premises is the information, not the level.

Weakest link: Energy per token served (site level) at 0.70 — 0.000199 Wh per token at frontier scope, composite grade medium, every input now sourced: GB200 NVL72 rack power over GPU count, hyperscale PUE from o

Therefore cost per token is a product of six coefficients, and the binding one moves

29% if the 3 gates are independent, 60% if they move together. They are claims about one industry, so the truth is between and nobody can say where. Treat this as an ordering device rather than a calibrated probability — the ranking of premises is the information, not the level.

Weakest link: NVIDIA GB200 (per-GPU in NVL72) at 0.60 — The accelerator is one term. Rack power per GPU is ~1,667W all-in against ~1,000W for the die alone, so even inside the hardware term the chip is 60%

The variant

Consensus

Cost per token falls with each accelerator generation. Buy the compute leader and the deflation accrues to whoever serves the most tokens.

Variant

Accelerator performance is one of at least six terms. Package perimeter, memory architecture, power delivery, grid access and serving software each set the coefficient independently, and the binding term has changed twice in eighteen months. A view on token cost that only models the chip is modelling one variable in six.

Differentiator

Consensus watches FLOPS per dollar. This watches the product of six coefficients, each measurable, and asks which one binds. On 2026-08-04 the answer was serving software, not silicon.

Open questions

Which term binds RIGHT NOW is the whole practical question and it is not stable. On 2026-08-04 the evidence points at serving software and memory architecture rather than silicon, but the Rubin step lands in H2 2026 and could move it back to hardware for a generation.
The six coefficients are held at different grades. Energy per token is medium and sourced; kv-bytes-per-token is computed and upper-bounded; the serving contribution is qualitative. A product of coefficients inherits the worst grade, so the aggregate claim is weaker than any single leg.
No term here is priced as a coefficient by the market. Testing whether that is a mispricing or a correct read of unpredictability requires the scored record this desk does not yet have.

Reasoning chain

Materials and packaging set a floor on how much compute fits in a package at all VALID

premises

Elite Material Co., Ltd. (EMC)0.70 moderate
M9-grade CCL with quartz-glass reinforcement is the substrate material for the Kyber midplane at ~1 square metre and ~78 layers, where yield compounds per layer and a single defect scraps a board serving 144 GPUs.
why
Materials qualification gates the board before any chip ships.
CoWoS (Chip-on-Wafer-on-Substrate)0.85 strong
2.5D packaging capacity has been the binding constraint on accelerator output, held severe in this model.
Hybrid Bonding0.70 moderate
Stack height beyond current HBM requires hybrid bonding, and JEDEC relaxing dimensional specs to avoid forcing it is a revealed statement that the industry cannot yet do it at volume.

The first term is physical assembly. Before an accelerator has a cost per token it has to exist as a package, and the package is gated by laminate qualification, interposer capacity and bonding capability — none of which the accelerator vendor controls end to end.

The package perimeter forces a trade between power and bandwidth that neither side wins outright VALID

premises

Package shoreline (perimeter and cross-section contention between power, memory and data)0.75 strong
Blackwell achieves ~0.4 Tb/s/mm off-package bandwidth against ~10 Tb/s/mm theoretically available.
why
Power escape and optical I/O compete for the same millimetres.
Multilayer ceramic capacitor (high-capacitance, AI server)0.70 moderate
Lateral power delivery requires a perimeter sea of decoupling capacitors because loop inductance prevents the regulator answering AI-workload di/dt.
why
Those capacitors occupy exactly the shoreline the SerDes need.
Coherent Corp.0.60 moderate
Co-packaged optics needs perimeter it does not control, so the optics opportunity is gated by a power-architecture decision made by the accelerator vendor.

The second term is geometric and is usually invisible in cost models. Perimeter is fixed, power and data both need it, and the resolution — power down through the back, data up and out the front — is a packaging decision that determines how much bandwidth a given die can actually reach. Bandwidth per package is a cost-per-token input and it is set by millimetres, not by process node.

Memory architecture moves the coefficient more than a hardware generation does VALID

premises

KV cache bytes per token, by attention architecture0.80 strong
252 KB34.3 KB7.3x
Computed from published configs: GQA at 252 KB/token against MLA at 34.3 KB/token, a 7.3x spread readable from a config file on release day.
HBM40.80 strong
3x
HBM bit-density per die is roughly one third of standalone DRAM because of the TSVs, so a given HBM capacity consumes ~3x the wafers.
Long-context inference (KV-cache growth)0.75 strong
71x
Agentic workloads run 142k input tokens per turn over a median 65 turns, which is ~71x a chat turn on context alone.

The third term is architectural and it dominates the second. Compression bought 7.3x and the agentic workload took 71x, so cache pressure rose despite better attention. This is the term that has embarrassed the desk twice: it holds both the coefficient and the denominator and had never multiplied them.

Energy per token converts the whole chain into megawatts, and megawatts are queued VALID

premises

Energy per token served (site level)0.70 moderate
0.000199 Wh
0.000199 Wh per token at frontier scope, composite grade medium, every input now sourced: GB200 NVL72 rack power over GPU count, hyperscale PUE from operator disclosures, MLPerf-measured throughput discounted for MoE.
Grid interconnection queue position0.80 strong
79.6%
Queue position gates new load, and MISO data shows 79.6% of decided projects withdrawn, so headline queue size overstates incoming supply.
Disclosed token volume (operator-reported)0.70 moderate
330x
Google alone discloses ~107 trillion tokens per day, up ~330x in twenty-five months.

The fourth term is where the chain leaves the building. Tokens become watt-hours become megawatts become a queue position, and the queue is the only term with a multi-year lead time. Efficiency relieves it in STEPS at hardware transitions rather than continuously, so between generations the pass-through is close to one for one.

Therefore cost per token is a product of six coefficients, and the binding one moves VALID

premises

Inference cost per token0.75 strong
3x4x
Serving cost fell from ~$20 to ~$0.40 per million tokens in roughly three years.
why
Decomposed: silicon perf/W 2-3x per generation, numerics ~4x cumulative, architecture and serving uncaptured and faster.
Inference serving stack (vLLM / TensorRT-LLM class)0.65 moderate
Architecture and serving improvements are published and open-weighted within weeks, so they are captured by nobody and diffuse for free.
why
Together they contributed more cumulative reduction than process nodes.
NVIDIA GB200 (per-GPU in NVL72)0.60 moderate
60%
The accelerator is one term.
why
Rack power per GPU is ~1,667W all-in against ~1,000W for the die alone, so even inside the hardware term the chip is 60% of the number.

THE INVESTABLE CLAIM. Consensus buys the compute leader because FLOPS per dollar improves. But the fastest deflation levers — architecture and serving — are published free and captured by nobody, which means a hyperscaler's cost curve improves substantially without buying anything. The terms that ARE capturable sit in materials, packaging, memory and power, which are priced as cyclicals rather than as coefficients on the token economy. THE FALSIFIER IS SHARP: if cost per token tracks accelerator generations more closely than it tracks architecture releases over the next four quarters, the chip is the term and this thesis is wrong. That is checkable against dated model releases and dated silicon launches.

Sources

Composed from measurements and signals already held in this ontology, each cited in a premise acc 2026-08-04

Write-up

Pre-filled skeleton: cost-per-token-is-set-outside-the-chip.md