AI Inference Chip Architecture: From GPU + HBM to ASIC + CXL + DRAM + NMC

Inference is often waiting for memory

The first page sets up the central misunderstanding: inference is not simply a smaller version of training. Training likes large batches and repeated matrix operations, so the accelerator can stay busy once the data is staged. LLM inference, especially the decode phase, is more sequential and repeatedly asks the system to fetch model weights and KV cache data for every new token.

This is why the bottleneck becomes the memory wall. The ALUs or tensor units may be ready to compute, but the useful data is still traveling through a narrow, congested, or expensive path. In a real deployment, the winning metric is not only TOPS. It is tokens per second, latency, tokens per watt, usable memory capacity, and tokens per dollar.

The classroom metaphor is deliberately simple: the chip is not lazy; it is hungry. If the memory system cannot feed it fast enough, adding more compute does not solve the user-visible problem.

GPU + HBM is powerful, but not always economical

HBM is extremely valuable when bandwidth is the first priority. It sits close to the compute die, offers a very wide interface, and reduces energy per transferred bit compared with moving data across a board-level memory channel. That is why GPU + HBM became the natural answer for training and high-end acceleration.

The second page explains why the same answer can be wasteful for inference. HBM is expensive, depends on advanced packaging capacity, and comes in stack capacities that may not match the growing memory footprint of large models and KV cache. If a deployment buys more HBM than the workload really needs, the extra bandwidth may sit underused while the cost remains very real.

For inference, the question is therefore not whether HBM is good. It is where HBM is necessary, where standard DRAM is sufficient, and how the system should balance bandwidth, capacity, availability, power, and total cost of ownership.

A new inference server formula

The third page introduces the system formula behind the series: ASIC + CXL + DRAM + near-memory compute. The ASIC specializes the inference datapath instead of carrying every feature of a general GPU. CXL turns memory expansion into a composable system resource. DRAM supplies economical capacity. Near-memory compute reduces data movement when the work can be handled close to where the data lives.

The important shift is architectural. Instead of treating the accelerator as the only hero, the full server becomes the optimization target. The ASIC, memory controller, CXL fabric, DRAM pool, software scheduler, and model-serving runtime all participate in the final tokens-per-dollar result.

This page also explains why different companies can pursue different products while still following the same underlying direction. Some optimize the accelerator, some optimize the memory fabric, and some optimize rack-level pooling. The common goal is to make memory less of a wall.

CXL turns memory into a scalable resource

The fourth page uses CXL as the bridge between old capacity and new inference demand. CXL does not magically replace local memory, and it is not as low-latency as on-package HBM. Its value is that it allows memory expansion, pooling, and tiering through a standard interconnect that the host and devices can coordinate.

In practical inference systems, this matters because not all data needs the same latency and bandwidth. Hot tensors, active KV cache, and high-frequency data may stay near the compute engine, while colder or larger data structures can live in an expanded DRAM tier. The software stack then decides placement and migration policy.

The Meta-style lesson is especially useful: older DDR capacity does not have to become worthless. If the system can attach and manage it through CXL, cheap memory can extend useful capacity and improve fleet-level economics.

Near-memory compute reduces the moving cost

The fifth page focuses on near-memory compute. The expensive part of inference is often not a multiply operation by itself; it is repeatedly moving data back and forth between memory and compute. If a small operation can be performed near the memory controller or memory-adjacent logic, the system may save bandwidth, latency, and energy.

This does not mean every neural-network layer should move into memory. Near-memory compute is most attractive for operations that are data-movement heavy and compute-light, such as filtering, reduction, simple transformation, prefetch assistance, or memory-side handling of selected cache and embedding traffic.

The restaurant metaphor in the image captures the point: carrying ingredients across town for every step wastes time and energy. Cooking near the pantry is not glamorous, but it can improve throughput and power efficiency when the workflow is dominated by movement.

The conclusion is composition, not a single chip

The final page pulls the pieces together. There is no single universal answer to the memory wall. Meta's route emphasizes CXL memory expansion and reuse of DRAM capacity. Qualcomm's route emphasizes inference-oriented accelerators and high-bandwidth controllers. Marvell and Google-style routes point toward custom ASICs, CXL fabrics, and full-system optimization.

The shared formula is the real takeaway: place the data correctly, use the right memory tier, avoid unnecessary movement, and optimize the whole server or rack for cost and efficiency. HBM remains crucial where extreme bandwidth is required, but commodity DRAM, CXL pooling, and near-memory compute can make inference more scalable.

For product planning, this is a warning against comparing chips only by peak performance. The better question is whether the architecture can deliver enough capacity, bandwidth, latency, and operational flexibility at the right TCO for the model and deployment scale.

References

1. CXL Consortium, Compute Express Link 2.0 overview: Background on CXL switching, memory pooling, and capacity expansion.
2. Qualcomm Dragonfly AI250: Official product reference for Qualcomm's AI inference and HBC positioning.
3. Marvell CXL switch for memory pooling: Official memory pooling announcement tied to the AI memory wall.
4. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory: Research background on page placement in CXL tiered-memory systems.