Start with the word-chain game
Do not start by memorizing token, embedding, K/V, or KV Cache. First picture an LLM as a very patient word-chain machine: it reads the text so far, picks the most likely next text piece, then uses that new piece as part of the next guess.
The six plates slow that loop down. Text is cut into tokens, tokens become numbers, numbers become vectors, attention writes K/V notes, and KV Cache keeps old notes so the model does not rewrite them every step.
On a first pass, follow the metaphors: word chain, building blocks, coordinates, notes, and a filing cabinet. Once those feel natural, the engineering terms are much easier to place.
An LLM is a next-token prediction machine
The first plate starts with the most important simplification: an LLM does not output a whole essay in one magical step. During inference, it repeatedly looks at the current context and predicts the most likely next token. The probability bar in the comic is the model's final decision surface for one step.
After the model emits a token, that new token is appended to the context and becomes part of the next prediction. This is why responses appear to stream out piece by piece. The model is constantly asking: given everything so far, what should come next?
This framing is useful because it connects user experience to hardware cost. Every generated token triggers another pass through the model layers, another round of matrix multiplications, and another set of memory reads. Long answers are not just more text; they are more repeated inference steps.
The core takeaway is that LLMs are powerful because they learned rich statistical structure from training data, but the runtime behavior is still a disciplined engineering loop: read context, compute probabilities, choose a token, append it, and repeat.
Tokens are the model's smallest text units
The second plate explains tokenization. Computers do not manipulate human text directly inside the model. A tokenizer cuts text into pieces, then maps each piece to an integer token id. Those ids are the vocabulary entries the model can actually process.
A token is not always a word. It may be a character, a subword, a punctuation pattern, or a word fragment. Chinese text can often produce more tokens than people expect because the tokenizer's segmentation rules do not match human word boundaries perfectly.
Token count matters operationally. Pricing, context length, latency, and memory pressure are usually tied to tokens rather than visible characters. A prompt that looks short to a human can still become expensive if it tokenizes into many pieces.
Once text becomes token ids, the model can look up embeddings. That lookup is the bridge from symbolic text to numeric vectors, and it is the entry point to the rest of Transformer computation.
Embeddings turn text into vectors and computation
The third plate moves from token ids to vectors. Each token id is mapped to an embedding vector, a long list of learned numbers. These numbers are not hand-written features; they are learned coordinates that help the model place related meanings near each other in a high-dimensional space.
The semantic map in the comic is a simplified picture of this idea. Tokens related to drinks, royalty, or fruits can cluster because the model learned from contexts where those ideas appear together. Meaning becomes geometry.
After embeddings, the heavy work is dominated by linear algebra. Transformer layers repeatedly multiply vectors by weight matrices, add results, apply attention, and feed the representation through neural network blocks. The friendly phrase 'AI understands text' hides a huge amount of multiply-accumulate work.
This is why AI hardware focuses so strongly on matrix throughput, memory bandwidth, data reuse, and numerical formats. The language interface is soft and human; the underlying engine is very much a data-movement and matrix-computation machine.
Attention writes Key and Value notes for context
The fourth plate introduces K/V notes. In self-attention, each token produces Query, Key, and Value vectors. A Query represents what the current position is looking for; Keys are labels for previous information; Values are the content that can be retrieved.
When generating a new token, the model compares the new Query with the Keys from previous tokens. The resulting attention scores decide which old tokens matter most. Then the model combines the corresponding Values to build context-aware information for the next step.
This explains why an LLM can refer back to earlier words. It is not rereading text like a person with eyes on a page; it is matching numeric queries against stored numeric keys and mixing stored values according to attention weights.
The catch is scaling. If every new token required recomputing all old Keys and Values from scratch, generation would get slower and slower. That is the setup for KV Cache.
KV Cache saves recomputation but not new thinking
The fifth plate captures the key tradeoff: KV Cache stores the Keys and Values that were already computed for previous tokens. During the next decoding step, the system reuses those cached K/V tensors instead of recomputing them from the beginning.
This can greatly reduce redundant work in autoregressive generation. Without KV Cache, every new token is like rewriting all the old index cards. With KV Cache, the old cards remain in the drawer and the model only writes the new card.
However, KV Cache does not make inference free. The new token still has to pass through the model, and the attention operation still has to read an ever-growing history of cached Keys and Values. Long contexts therefore increase memory capacity needs and memory bandwidth pressure.
For serving systems, KV Cache is often one of the largest runtime memory consumers. Managing it well affects throughput, latency, batching, and the number of users a system can serve at once.
The next bottleneck is moving data
The final plate summarizes the whole pipeline: tokenize the text, predict the next token, transform token ids into vectors, run massive multiply-accumulate operations, use K/V notes to attend to context, and store old K/V in KV Cache to avoid recomputation.
The important nuance is that KV Cache saves compute but spends memory. As prompts and conversations grow, cached context can occupy gigabytes. The cost is no longer only the math; it is also moving weights, activations, and KV tensors through the memory hierarchy.
This is why AI inference architecture quickly becomes a semiconductor topic. GPU memory, HBM bandwidth, SRAM capacity, cache management, near-memory computing, and compute-in-memory all attack the same question: how do we keep data close enough that the compute engines do not starve?
The next classroom can naturally continue into CIM and near-memory compute: if moving data is expensive, one answer is to move part of the compute closer to where the data lives.
References
- Attention Is All You Need: Original Transformer paper introducing scaled dot-product attention and Query, Key, and Value projections.
- Hugging Face Transformers: KV cache strategies: Practical documentation on KV cache use in generative Transformer inference.
- OpenAI tiktoken: OpenAI tokenizer implementation showing how text is encoded into token ids.
- Efficient Memory Management for Large Language Model Serving with PagedAttention: vLLM paper explaining why KV cache memory management is central to LLM serving throughput.
- FlashAttention-2: Attention acceleration paper focused on reducing memory movement and improving hardware utilization.
Learning guide
AI Technology Classroom
Prerequisites
- Basic probability and matrix multiplication
What I learned
- Explain why LLM output is generated token by token
- Describe how token ids become embedding vectors
- Explain how KV Cache saves recomputation but increases memory pressure