This interview with Dylan Patel, founder and CEO of SemiAnalysis, was published as part of the Dwarkesh Podcast on Oct 2, 2024. The conversation—hosted by Dwarkesh Patel with guest appearances by Jon (Asianometry)—focuses on the physical limits and supply-chain constraints that determine how quickly AI compute can scale. (dwarkesh.com)
The discussion was released as a Dwarkesh Podcast episode and is available on the podcast page and the program's YouTube feed. (dwarkesh.com)
Hyperscaler CapEx, timelines and what “gigawatt” means
Patel lays out how the published CapEx plans of the major hyperscalers map into staged compute capacity rather than instant deployments. He explains that a portion of the announced spending is for near-term chip and data-center purchases, but a large share is for setup CapEx—land, turbines, down payments and other multi-year commitments—so the actual capacity comes online across multiple years. He says roughly 20 GW of incremental IT capacity is expected to deploy in the U.S. in a given year but that large hyperscaler budgets are spread across longer windows. (dwarkesh.com)
"A portion of this is immediately for compute going online this year: the chips and the other parts of CapEx that get paid this year. But there's a lot of setup CapEx as well."
How labs acquire compute and why contract length matters
Patel describes the trade-off facing fast-growing AI labs: sign long-term, lower-price deals and lock in margin advantages, or acquire last-minute capacity at much higher per-unit rental costs. He points out that long-term five-year deals preserve margins and crowd out shorter-term buyers, whereas spot and short-term deals can be twice (or more) as expensive. He describes OpenAI’s aggressive multi-provider contracts versus Anthropic’s more conservative approach, and explains how last-minute purchases push labs to smaller or lower-quality suppliers.
"The person who committed early has better margins in general."
GPU value, depreciation and the utility lens
Rather than treating GPUs as simple depreciating assets, Patel stresses their value is set by the utility they deliver today. Newer model architectures and inference improvements (for example, moving from GPT-4 to GPT-5.4) can make an older GPU more valuable if it runs higher‑value tokens. He lays out two lenses: (1) a TCO/depreciation model that treats a GPU as an amortized asset over a fixed life, and (2) a utility model where the price is driven by the immediate revenue the chip can produce. Under the latter, an H100 can be worth more today than three years ago because of model improvements and product-market adoption.
Memory: HBM bandwidth versus commodity DRAM
Patel identifies memory—especially HBM bandwidth per shore-line—as the core gating factor for modern model inference and long-context KV caches. He quantifies the difference in shoreline bandwidth: a single HBM4 stack can deliver roughly 2.5 TB/s of bandwidth from a ~13 mm shoreline, whereas DDR alternatives deliver on the order of tens to low hundreds of GB/s for the same shoreline area. As a result, moving from HBM to DDR multiplies bits per wafer but drastically reduces bandwidth per edge, shifting the system-design constraints and typically slowing inference.
"The metric you actually care about is bandwidth per wafer, not bits per wafer."
Wafer math, EUV passes, and the ASML choke point
Patel walks through the arithmetic tying wafers and lithography to deployed AI capacity. He explains that producing a gigawatt of Rubin-class accelerators requires tens of thousands of 3 nm wafers, thousands of 5 nm wafers and hundreds of thousands of DRAM wafers—and that across those wafers you need millions of EUV exposures. He gives a back-of-envelope: roughly 2 million EUV passes for a single gigawatt of modern accelerator capacity, which translates into a need for multiple EUV tools. Because ASML’s EUV tool output is limited and the tool itself is extraordinarily complex, Patel argues tool production and the toolmakers’ supply chains become a structural bottleneck several years out.
"ASML makes the world's most complicated machine...they can make about 70 now, maybe a little over 100 by the end of the decade, and that limits how fast chips can be produced."
Packaging, interconnect and scale-up topology
Beyond transistor counts, Patel highlights packaging and scale-up topology as multiplicative: larger, better interconnected packages reduce rack-to-rack communication losses and enable much higher end-to-end model throughput. He contrasts Nvidia’s all‑to‑all rack-scale approach with Google’s torus-style TPU pods and explains why differences in topology and in‑chip vs inter‑chip bandwidth generate real performance gaps that cannot be solved purely by moving to an older node. Packaging trends (more dies per package, CoWoS and multi‑die integration) will continue, but there are limits and trade-offs in cooling, routing and memory placement.
Power, behind‑the‑meter builds and the limits of energy as the single constraint
Patel argues that while energy and permitting are nontrivial, they present a wider range of engineering and commercial fallbacks than semiconductors do. He outlines the menu of solutions—behind‑the‑meter gas turbines, aeroderivative engines, ship engines, fuel cells, solar-plus-battery, and utility-scale batteries—and says these options make large additions of critical IT power feasible even if more expensive. In short: the grid and turbines are hard but solvable; tooling and wafers are the structural bottleneck.
Space data centers, robots and geopolitics
When asked about Elon Musk’s space‑based data‑center idea, Patel notes that space moves the energy constraint but not the fundamental chip‑production constraint. Because chips and tooling are the contended resource, moving compute to space delays useful deployment and introduces massive new reliability and interconnect challenges. He also touches on robot deployment: local device compute versus cloud batching, and the semiconductor implications if millions of humanoids require leading-edge chips.
China, indigenization and long-run scenarios
Patel explains that China is accelerating at many layers of the stack and may have working domestic DUV/EUV tools in the coming decade, but production quality and mass‑volume manufacturing remain the hard parts. He frames a simple heuristic: fast global takeoff favors current Western + Taiwan + Korea supply concentration; slower multi‑year takeoff gives China time to indigenize capacity and potentially catch up at scale.
Closing: what “the single biggest bottleneck” means
Throughout the conversation Patel repeatedly returns to the same framing: many problems (power, data centers, packaging) have engineering workarounds and multiple suppliers; the semiconductor manufacturing stack—memory fabs, logic wafers and the tooling (EUV and its optics, sources, stages)—is artisanal, has very long lead times, and cannot be scaled instantly. That structural reality, he argues, will determine who can deploy the next waves of AI capacity fastest.
References
Episode page and transcript: @Asianometry & Dylan Patel — How the semiconductor industry actually works (Dwarkesh Patel, Oct 2, 2024). (dwarkesh.com)
Podcast listing (aggregators) and episode metadata: Listen Notes: Oct 2, 2024. (listennotes.com)
Episode video (YouTube): @Asianometry & Dylan Patel — How the Semiconductor Industry Actually Works (YouTube). (neunetz.com)
Explore more exclusive insights at nextfin.ai.
