The model
A program whose calculations use values learned from examples. These values are its “parameters”. Text is split into small pieces called “tokens”.
A timeline of
artificial intelligence.
Every innovation addresses a problem. Discover which one.
How models learn, evolve and become usable. Explore ideas, mechanisms and trade-offs, starting with the foundations of the Transformer.
163 documented milestones6 reading paths↗ Primary sources
Three things to know before you start.
Learning
During training, the program adjusts these values to perform tasks better, such as predicting the next piece of text. It does not simply store every answer in a library.
Answering
A text-generating model calculates possibilities for what comes next. It produces successive pieces of text using what precedes them. A plausible continuation can contain errors.
Explore how a model connects words, learns to follow instructions and reduces its computation and memory needs.
Ideas take shape.
Reveal the next part of a sentence and follow the information used at each step.
Continue writing without reading everything again.
A text takes shape, one token at a time.
Thecatsleepsonthesofa.
Already readToken being processedNew token
Information kept in the cache
TheK/V storedcatK/V storedsleepsK/V storedonK/V storedtheK/V stored
The initial text has been read: its 5 K/V entries are in memory. Generate the continuation to see what gets reused.
Scripted example; no model is running. A word or punctuation mark acts as a token here; splitting, weights and routing are illustrative.
Go deeper: diagrams and settings
The same sentence, up close
Thecatsleepsonthe
When the model writes a continuation, it reuses information already computed from the preceding text. This information takes up space: it is the KV cache. Circles represent reading paths, called “attention heads”; the blocks below show retained information.
Try these, in order
Choose the first step to begin.
Go deeper: adjust calculation stages and heads
Assumptions: one sequence, 128 dimensions per K/V head, 2 bytes per element; for MLA, a 512-element latent and a 64-element positional key. Model weights, activations, fragmentation and engine overheads are excluded.
K and V are two sets of computed data: keys (K) help determine which positions to consult; values (V) carry information to combine. A “K/V” box represents these two sets, not two copies of the text.
See the formula
GQA : 2 × G × d × L × T × b
| Architecture | Memory | Share of MHA |
|---|---|---|
| MHA | 2.5 MiB | 100 % |
| MQA | 80 KiB | 3.125 % |
| GQA | 640 KiB | 25 % |
| MLA | 180 KiB | 7.031 % |
A longer bar means more memory. GiB and MiB are memory units: 1 GiB equals 1,024 MiB. These numbers cover only the retained information for the text.
Understand the calculation and its limits
T = tokens, L = layers, H = heads, G = KV groups, d = dimensions per head, b = bytes per element. MHA: 2HdLTb; MQA: 2dLTb; GQA: 2GdLTb; MLA: (512 + 64)LTb. The factor 2 in MHA/MQA/GQA counts K and V; the MLA latent jointly compresses K and V.
MLA is not always the smallest cache: here MQA stores fewer elements. Size alone measures neither quality nor speed. The MLA diagram shows retained elements; projections can be absorbed into weights to compute without reconstructing all K/V.
“She”: which information to attend to?
A text takes shape, one token at a time.
Leaputsanappleonthetable.Shecutstheapple.
Already readToken being processedNew token
What does the purple token attend to?
Attended words turn blue. A stronger color represents a higher weight in this mixture of information.
We process “She”. In this illustrative mixture, “Lea” gets the highest weight.
Scripted example; no model is running. A word or punctuation mark acts as a token here; splitting, weights and routing are illustrative.
Go deeper: diagrams and settings
The same sentence, up close
Leaputsanappleonthetable.She
To process a word, the model combines information from other text positions. Percentages show the share taken from each position for this calculation. The row is the word being processed; the column is the position consulted.
Try these, in order
Choose the first step to begin.
Go deeper: concentrate the weights
How this example is calculated
Invented two-dimensional Q/K vectors; no model is running. The divisor τ is a teaching control for attention scores; it is not token-sampling temperature.
weights = softmax((QKᵀ / √2 + mask) / τ)
| Reads ↓ / attends to → | Lea | puts | an | apple | on | the | table | . | She | cuts | the | apple | . |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Lea | 100% | × | × | × | × | × | × | × | × | × | × | × | × |
| puts | 94% | 6% | × | × | × | × | × | × | × | × | × | × | × |
| an | 89% | 5% | 5% | × | × | × | × | × | × | × | × | × | × |
| apple | 85% | 5% | 5% | 5% | × | × | × | × | × | × | × | × | × |
| on | 81% | 5% | 5% | 5% | 5% | × | × | × | × | × | × | × | × |
| the | 77% | 5% | 5% | 5% | 5% | 5% | × | × | × | × | × | × | × |
| table | 74% | 4% | 4% | 4% | 4% | 4% | 4% | × | × | × | × | × | × |
| . | 71% | 4% | 4% | 4% | 4% | 4% | 4% | 4% | × | × | × | × | × |
| She | 68% | 4% | 4% | 4% | 4% | 4% | 4% | 4% | 4% | × | × | × | × |
| cuts | 4% | 4% | 4% | 65% | 4% | 4% | 4% | 4% | 4% | 4% | × | × | × |
| the | 4% | 4% | 4% | 63% | 4% | 4% | 4% | 4% | 4% | 4% | 4% | × | × |
| apple | 4% | 4% | 4% | 61% | 4% | 4% | 4% | 4% | 4% | 4% | 4% | 4% | × |
| . | 3% | 3% | 3% | 59% | 3% | 3% | 3% | 3% | 3% | 3% | 3% | 3% | 3% |
Go deeper: this row’s numbers
The selected row, in detail
The query “She” combines information from the start of the sentence. The four tokens in “cuts the apple .” are masked.
output = Σ weightᵢ × Vᵢ
Weights describe how this head mixes values. By themselves, they do not explain the model’s reasoning. Transformer paper
One token, selected experts at work.
A text takes shape, one token at a time.
Thecatsleepsonthesofa.
Already readToken being processedNew token
theRouter2 selected experts
E1Active
E2On standby
E3Active
E4On standby
E5On standby
E6On standby
The selected experts’ outputs are combined, then the model continues its calculations to predict the next token.
The token “the” passes through E1 and E3. The other four experts remain stored without performing this computation.
Scripted example; no model is running. A word or punctuation mark acts as a token here; splitting, weights and routing are illustrative.
Go deeper: diagrams and settings
The same sentence, up close
Thecatsleepsonthe
A large model can contain several computation blocks and use only some for each piece of text. These blocks are called “experts”. A selection mechanism, the “router”, chooses which ones to use, then their results are combined.
Try these, in order
Choose the first step to begin.
Fictional example with 6 equally sized experts. Illustrative scores are assigned to the sentence’s tokens; they do not give experts fixed specialties. Shared layers, shared experts and communication costs are not counted.
E142%Active
E28%On standby
E327%Active
E44%On standby
E513%On standby
E66%On standby
Go deeper: combine the results
output = 0.609 × E1(x) + 0.391 × E3(x)
2 of 6 experts active for this token; all 6 remain in memory.
This ratio is neither the fraction of all model parameters activated nor a speed gain. Real methods differ in routing, normalization and load balancing. Sparse MoE paper
Foundations
Attention connects different parts of a text. Optimizers adjust parameters during training; activation and normalization functions organize the network’s calculations.
Attention before the Transformer
Translation can consult useful passages instead of squeezing the whole sentence into one fixed representation.
The problem
A long sentence loses details if everything must fit into one numerical representation.
The idea
For each output word, the model scores positions in the source sentence and combines their information according to those scores.
An example to picture the idea
When translating “the black cat”, it may give “black” more weight when producing the color word.
What this adds
Translation can reuse specific information from the source sentence.
The limit to remember
Processing still proceeds step by step, limiting how much calculation can happen at once.
Go deeper: mechanism, diagram and origins
How it works
A network scores encoder states and combines them into a weighted sum. This soft attention avoids compressing the entire sentence into a single vector.
Limits and trade-offs
The model remains recurrent, so sequential computation limits parallelism.
- Original demonstration
- Neural translation by Bahdanau, Cho and Bengio; a precursor, not a Transformer.
Papers and sources
Direct link ↗Adam: adaptive learning steps
Adjust the size of corrections during learning.
The problem
Changing every learned value by the same step size can make training difficult.
The idea
Adam keeps recent averages of correction directions and sizes. It uses them to adjust the step for each learned value, called a parameter.
An example to picture the idea
A value that repeatedly receives large corrections can be assigned a more cautious step.
What this adds
This adjustment makes many networks easier to train.
The limit to remember
The histories take memory and do not directly make answers faster after training.
Go deeper: mechanism, diagram and origins
Loss gradientTransform the updateAdjust parameters
How it works
It tracks moving averages of gradients and squared gradients to adapt each weight’s update. A correction compensates for initial bias in these estimates.
Limits and trade-offs
Extra optimizer states consume memory, and hyperparameters still matter.
- Original demonstration
- Kingma and Ba: experiments on several networks, without a named first LLM.
Papers and sources
Direct link ↗BPE: representing words with subword units
Split words into reusable text pieces.
The problem
A vocabulary containing only whole words struggles with rare or new words.
The idea
The method starts with characters and repeatedly merges frequent pairs. The model then uses this vocabulary of text pieces, called tokens.
An example to picture the idea
Fictional split: “gardening” becomes “garden” and “ing”, two pieces already in the vocabulary.
What this adds
A limited vocabulary can represent many different words.
The limit to remember
Splitting depends on languages and training data: one word does not always correspond to one piece.
Go deeper: mechanism, diagram and origins
How it works
Starting with characters, BPE merges frequent pairs to build a limited vocabulary. New words can then be split into known units instead of becoming an unknown symbol.
Limits and trade-offs
Token counts depend on the language and the learned vocabulary.
- Original demonstration
- Sennrich et al., neural translation; 2015 preprint, ACL publication in 2016.
Papers and sources
Direct link ↗Recomputing activations to save memory
Repeat some calculations to keep fewer intermediate results in memory.
The problem
During training, retaining every intermediate result can fill the available memory.
The idea
Only selected checkpoint results are kept. When a correction needs a missing result, calculations are repeated from a retained checkpoint.
An example to picture the idea
Fictional example: retain every third stage and recalculate intermediate stages when needed.
What this adds
A larger network can be trained with the same memory.
The limit to remember
Repeating calculations adds work and can lengthen training.
Go deeper: mechanism, diagram and origins
How it works
Instead of storing every forward-pass activation, selected checkpoints are retained. Missing activations are recomputed during backpropagation when they become necessary.
Limits and trade-offs
Memory savings increase computation time according to the chosen partitioning.
- Original demonstration
- Chen et al.: a general method demonstrated on deep networks including vision models.
Papers and sources
Direct link ↗GELU: a smoother activation
Scale internal values gradually instead of cutting them off abruptly.
The problem
A hard cutoff for negative values limits how a network transforms information.
The idea
This function scales each value according to its magnitude, with a smooth transition near zero. It operates in stages that transform each text piece separately.
An example to picture the idea
A small negative value is reduced rather than automatically replaced with zero.
What this adds
This transformation offers another way to shape internal calculations during learning.
The limit to remember
It does not by itself reduce the number of comparisons between text pieces.
Go deeper: mechanism, diagram and origins
How it works
The activation multiplies input x by the standard normal cumulative probability at x. It gives a smooth response near zero in layers that transform each token individually.
Limits and trade-offs
A better activation does not by itself reduce attention cost.
- Original demonstration
- Hendrycks and Gimpel; experiments preceding BERT.
Papers and sources
- Gaussian Error Linear Units (GELUs) ↗
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding ↗
LayerNorm: stabilizing activations
Bring internal values to a more regular scale.
The problem
Internal values that are too large or too spread out can make training difficult.
The idea
For each example, a calculation stage centers its values and adjusts their spread. Learned values then allow this rescaling to be modified.
An example to picture the idea
A stage receiving widely scattered values can bring them to a more regular scale.
What this adds
The network’s different stages become easier to train together.
The limit to remember
Where this operation is placed affects its influence on learning.
Go deeper: mechanism, diagram and origins
How it works
For one example and layer, activations are centered and rescaled using their statistics. Learned parameters can then modify this transformation.
Limits and trade-offs
Its position inside a block affects gradients and stability.
- Original demonstration
- Ba, Kiros and Hinton; evaluations include recurrent networks.
Papers and sources
Direct link ↗MoE: many weights, few active experts
Keep many subnetworks but use only a few for each text piece.
The problem
Activating every learned value in a very large network requires considerable calculation.
The idea
A selection mechanism sends each text piece to a few subnetworks, called experts. Their specialization is learned and does not necessarily match subjects such as medicine.
An example to picture the idea
Fictional example: choose two subnetworks out of eight to process a piece of a sentence.
What this adds
The network can contain more learned values without using them all on every pass.
The limit to remember
All experts must be stored and their workload must be balanced.
Follow the ideas
Later developments
- Improved bySwitch Transformer: one expert per token
Switch retains routing to sparse experts and simplifies it: a single expert is chosen for each token.
Paper: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity ↗ - Built on byMixtral: opening sparse experts
Mixtral uses the sparse-expert principle: each token is sent to two of the eight experts in each MoE layer.
Paper: Mixtral of Experts ↗
Go deeper: mechanism, diagram and origins
MoE: choose experts for each token
On small screens, scroll the diagram horizontally.
How it works
A router selects a few expert networks and weights their outputs. Load-balancing constraints encourage using different experts rather than concentrating tokens on the same ones.
Limits and trade-offs
Experts still require storage, load balancing and communication.
- Original demonstration
- Shazeer et al.: MoE layers embedded in recurrent language and translation models.
Papers and sources
Direct link ↗SiLU, then Swish: a smooth gate
Pass internal values through a smooth transition.
The problem
A sharp cutoff of internal values is not always a good choice for learning.
The idea
The input value is multiplied by a factor between zero and one calculated from that same input. Negative values therefore do not all disappear abruptly.
An example to picture the idea
A slightly negative value can remain present while being reduced.
What this adds
This function provides a smooth transformation for the network’s internal calculations.
The limit to remember
Its usefulness depends on the network and it is not a complete architecture by itself.
Go deeper: mechanism, diagram and origins
How it works
A sigmoid smoothly gates the input without abruptly eliminating every negative value. The October 2017 Swish paper studies x × sigmoid(βx), which becomes SiLU when β = 1.
Limits and trade-offs
Benefits depend on the architecture; this is not yet SwiGLU.
- Original demonstration
- Elfwing et al., reinforcement learning; preceding the Swish paper.
Papers and sources
- Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning ↗
- Searching for Activation Functions ↗
How to read this timeline
Start with the major milestones. Explore all developments to go deeper. Every card explains the mechanism, its contribution and its limits, with links to original publications.
A few concepts to get started
- Token
- A model text unit: a word fragment, punctuation or symbol. A token is not necessarily a word.
- Parameters and activations
- Parameters are learned, stored values. Activations are intermediate values computed for an input.
- Q, K and V
- Queries Q look for information, keys K produce matching scores and values V carry the information to combine.
- Softmax
- A transformation of scores into positive weights that sum to one. In attention, these weights determine how values are combined.
- KV cache
- Keys and values for processed tokens are retained to avoid recomputing them for each new token. This cache is separate from model weights.
- Prefill and decoding
- Prefill processes the prompt and builds the cache. Decoding then generates tokens, usually one at a time in an autoregressive model.
- HBM and bandwidth
- HBM is the GPU’s fast main memory. Bandwidth measures how much data can be moved per second; it can limit speed even when arithmetic is fast.
- FLOPs and quadratic cost
- FLOPs count arithmetic operations. Full attention compares positions pairwise: doubling sequence length roughly quadruples that work, without quadrupling every model cost.
- Pretraining, SFT and RL
- Pretraining learns from large corpora. SFT learns from example responses. RL adjusts the generation policy using rewards.
- Dense and MoE
- A dense model uses its layers for every token. In an MoE, a router selects some experts; active parameter count is lower than total stored parameters.
No results. Try another term or broaden the filters.
2017
Attention Is All You Need: the Transformer
Connect text pieces through comparisons that can be calculated in parallel.
The problem
Having to process a sentence strictly in order limits training speed.
The idea
Each text piece consults others using relevance scores: this is attention. Several reading paths, called heads, perform these calculations across successive stages.
An example to picture the idea
In “the cat eats because it is hungry”, one path may link “it” with “cat”.
What this adds
Much of the calculation over known text can happen in parallel.
The limit to remember
Comparing every pair of pieces becomes expensive as the text gets longer.
Follow the ideas
Later developments
- Improved byMQA: sharing keys and values
MQA retains Transformer attention heads but makes them share a single set of keys and values.
Paper: Fast Transformer Decoding: One Write-Head is All You Need ↗ - Combined inJamba: mixing Mamba and attention
Jamba retains attention layers within a model that also uses Mamba’s recurrent memory.
Paper: Jamba: A Hybrid Transformer-Mamba Language Model ↗
Go deeper: mechanism, diagram and origins
Vectors + positionsAttention → feed-forwardOutput prediction
How it works
Each head scores queries Q against keys K and combines values V. Blocks alternate attention, feed-forward transformations and residual connections; position information restores token order.
Limits and trade-offs
Dense attention compares every pair, making its cost quadratic in sequence length.
- Original demonstration
- Vaswani et al.: an encoder-decoder Transformer for translation.
Papers and sources
Direct link ↗Learning rewards from human preferences
Use human comparisons to define what training should encourage.
The problem
It is difficult to write a numerical rule that captures exactly the desired behavior.
The idea
People choose between two behaviors. A model learns to predict their choices, and that learned score then guides the system’s training.
An example to picture the idea
In a simulation, evaluators may prefer stable movement over falling down.
What this adds
Learning can be guided without programming every detail of the desired behavior.
The limit to remember
The learned score can misrepresent preferences and improve without satisfying their intention.
Follow the ideas
Later developments
- Built on byInstructGPT: following human intent
InstructGPT applies reward learning from human preferences to tune a language model.
Paper: Training language models to follow instructions with human feedback ↗
Go deeper: mechanism, diagram and origins
How it works
People compare two behaviors. A model learns to predict their preferences, then an agent optimizes that learned reward through reinforcement learning. The principle predates its use for large language models.
Limits and trade-offs
The reward can misrepresent preferences and be exploited by the agent.
- Original demonstration
- Christiano et al.: Atari and simulated locomotion, not ChatGPT.
Papers and sources
Direct link ↗PPO: limiting reinforcement-learning updates
Avoid changing behavior too much within one series of updates.
The problem
Reward-guided training can become unstable when choices change too abruptly.
The idea
The method compares action probabilities before and after an update. Its objective limits the benefit of increasing certain probabilities excessively.
An example to picture the idea
A well-scored behavior becomes more likely, but the objective limits the incentive to make it suddenly dominant.
What this adds
The same observations can support several more restrained updates.
The limit to remember
This limit guarantees neither improvement with every update nor easy tuning.
Follow the ideas
Later developments
- Improved byGRPO: comparing a group of responses
GRPO adapts PPO by comparing rewards for several answers to the same question, without a separate value critic.
Paper: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models ↗
Go deeper: mechanism, diagram and origins
How it works
The algorithm compares action probabilities under old and new policies. A clipped objective limits incentives to change that probability excessively across several updates using the same data.
Limits and trade-offs
Clipping does not guarantee improvement, and tuning remains delicate.
- Original demonstration
- Schulman et al.: control tasks and games, before LLM RLHF.
Papers and sources
Direct link ↗Mixed precision: computing with fewer bits
Use smaller numerical formats for part of the calculations.
The problem
Representing every number with high precision consumes memory and computing capacity.
The idea
Some operations use shorter numerical formats. Sensitive values and accumulated results remain more precise, with adjustments to preserve very small corrections.
An example to picture the idea
Multiplications can use a short format while the main learned values are retained in a more precise one.
What this adds
Training can use less memory and run faster on suitable hardware.
The limit to remember
Rounding and numbers that are too small or too large must be controlled.
Go deeper: mechanism, diagram and origins
How it works
Operations use formats such as FP16 while master weights and accumulations remain FP32. Loss scaling helps preserve small gradients during conversion.
Limits and trade-offs
Gains depend on hardware, and underflow and overflow require control.
- Original demonstration
- Micikevicius et al.: vision, speech and language networks.
Papers and sources
Direct link ↗2018
ULMFiT: pretrain, then adapt
Reuse an already trained model instead of starting from scratch for every task.
The problem
A specialized task often has too few examples to train a complete network from scratch.
The idea
The network first learns from general text, then from the target domain. Its calculation stages are adapted gradually to the task to limit loss of previous learning.
An example to picture the idea
A model trained on general writing is adapted to customer reviews, then to classifying their sentiment.
What this adds
General text becomes useful even when specialized examples are scarce.
The limit to remember
This method targets tasks such as text classification, rather than an assistant freely following instructions.
Go deeper: mechanism, diagram and origins
How it works
A recurrent network is pretrained on text, adapted to the target domain and used for classification. Gradual unfreezing and discriminative learning rates limit forgetting during adaptation.
Limits and trade-offs
The focus is supervised classification, not yet an instruction-following assistant.
- Original demonstration
- Howard and Ruder: ULMFiT, based on AWD-LSTM.
Papers and sources
Direct link ↗ELMo: a word’s meaning depends on context
Represent a word using the sentence around it.
The problem
A fixed representation of a word cannot distinguish its different uses.
The idea
Two networks read the text in opposite directions. Their results are combined into a context-sensitive representation of each word, which another model can use.
An example to picture the idea
“Bank” receives different representations in “river bank” and “bank account”.
What this adds
Text-processing tools gain information about each word’s surrounding context.
The limit to remember
Tasks still need their own models and reading remains partly sequential.
Go deeper: mechanism, diagram and origins
How it works
Two recurrent language models read text in opposite directions. A combination of their layers provides representations incorporated into downstream task models.
Limits and trade-offs
Tasks retain their own architectures, and recurrence limits parallelism.
- Original demonstration
- Peters et al.: ELMo, a bidirectional LSTM language model.
Papers and sources
Direct link ↗Relative positions: encoding distances
Indicate distance between text pieces rather than only their absolute positions.
The problem
The same relationship can occur at the beginning or end of a sentence.
The idea
Attention comparisons incorporate learned values representing relative distances. A nearby relationship can therefore be reused elsewhere in the text.
An example to picture the idea
“Very” immediately before an adjective keeps that proximity even when an introduction is added to the sentence.
What this adds
The calculation gets direct information about distance between elements.
The limit to remember
The studied version groups large distances together and does not guarantee handling arbitrarily long text.
Go deeper: mechanism, diagram and origins
How it works
Learned representations of relative distances enter attention calculations. Two words separated by the same distance can share relational information wherever they occur.
Limits and trade-offs
Distances are clipped in the evaluated formulation, so extrapolation is not guaranteed.
- Original demonstration
- Shaw, Uszkoreit and Vaswani: a translation Transformer.
Papers and sources
Direct link ↗Adafactor: reducing optimizer memory
Reduce memory used to prepare learning updates.
The problem
Keeping a history for every learned value adds substantial memory during training.
The idea
For a table of learned values, the method retains statistics for each row and column. Combining them replaces some per-value statistics.
An example to picture the idea
Instead of a separate history for every cell in a table, summaries of its rows and columns are kept.
What this adds
Preparing updates requires less memory.
The limit to remember
These summaries are approximations and the update rules still need appropriate settings.
Go deeper: mechanism, diagram and origins
How it works
Instead of storing a squared-gradient average for each matrix element, it retains row and column statistics. Their factorization gives a more memory-efficient approximation.
Limits and trade-offs
The approximation and update rules require suitable settings.
- Original demonstration
- Shazeer and Stern: translation Transformer evaluations, before T5.
Papers and sources
Direct link ↗GPT: transferable generative pretraining
Learn to continue text, then adapt the same network to different tasks.
The problem
Training each text-processing tool separately makes limited use of large collections of unlabeled text.
The idea
The model predicts the next text piece from previous pieces. It then adjusts its learned values on task-specific examples while keeping almost the same structure.
An example to picture the idea
After general training, question-and-answer examples can be used to adapt the network.
What this adds
General learning becomes a reusable starting point for several tasks.
The limit to remember
The method still relies mainly on separate adaptation for each task.
Go deeper: mechanism, diagram and origins
How it works
The decoder first learns from unlabeled text with a causal mask. Supervised fine-tuning then adapts its weights to each task while retaining almost the same architecture.
Limits and trade-offs
Results still rely mainly on separate fine-tuning for each task.
- Original demonstration
- OpenAI: GPT, Improving Language Understanding by Generative Pre-Training.
Papers and sources
Direct link ↗BERT: learning to reconstruct masked text
Learn to recover hidden pieces using text on both sides.
The problem
Some tasks require considering what comes both before and after a word.
The idea
During training, the network reconstructs masked pieces in sentences. It is then adapted to tasks such as classification or finding an answer inside a passage.
An example to picture the idea
In “the cat drinks [masked]”, the surrounding sentence helps predict the missing piece.
What this adds
The resulting representations use context before and after each position.
The limit to remember
This calculation is not directly designed to write an answer one piece at a time.
Go deeper: mechanism, diagram and origins
Partially masked textContext from both sidesRecover hidden tokens
How it works
A Transformer encoder reconstructs hidden tokens using left and right context. The original model also learns a sentence-relationship task before downstream fine-tuning.
Limits and trade-offs
This mechanism is not directly designed for generating responses token by token.
- Original demonstration
- Devlin et al., Google: BERT; neither the invention of GELU nor LayerNorm.
Papers and sources
Direct link ↗Blockwise parallel decoding
Propose several output pieces, then check which ones can be kept.
The problem
Producing only one text piece per pass through a large network takes time.
The idea
Predictors propose a possible continuation block. A checking model keeps the beginning of the block that meets its acceptance rule, and generation resumes there.
An example to picture the idea
Fictional example: propose three pieces and keep only the first two after checking.
What this adds
Several pieces may be accepted in one verification pass.
The limit to remember
The benefit disappears when proposals rarely agree with the checking model.
Go deeper: mechanism, diagram and origins
How it works
Predictors propose an output block, then a scoring model validates the longest acceptable prefix. This precedes speculative decoding without being identical to later probabilistic algorithms.
Limits and trade-offs
Gains depend on agreement between proposals and validation.
- Original demonstration
- Stern, Shazeer and Uszkoreit: translation and image super-resolution.
Papers and sources
Direct link ↗GPipe: distributing layers in a pipeline
Spread network stages across chips and move small groups of examples through them.
The problem
A complete network can be too large for a single computing chip.
The idea
Successive network stages are placed on different chips. A group of examples is split into smaller groups that move through these stages before their updates are combined.
An example to picture the idea
Fictional example: one chip handles a new group while the next finishes the preceding group.
What this adds
Several chips can help train the same large network.
The limit to remember
Transfers and periods when some chips wait reduce efficiency.
Go deeper: mechanism, diagram and origins
How it works
Layers are distributed across devices. A batch is divided into microbatches moving through the pipeline, and their gradients are accumulated before updating parameters.
Limits and trade-offs
Idle periods and transfers limit pipeline efficiency.
- Original demonstration
- Huang et al.: GPipe, demonstrations in vision and multilingual translation.
Papers and sources
Direct link ↗2019
Transformer-XL: reusing the preceding segment
Reuse previous results to connect successive text segments.
The problem
Splitting text into independent segments breaks connections across their boundaries.
The idea
The network retains representations from the preceding segment and consults them while processing the next. Distance information avoids confusing positions when these results are reused.
An example to picture the idea
A sentence at the start of a segment can consult information from the end of the preceding segment.
What this adds
Processing can use context beyond the current segment.
The limit to remember
This working state is limited and is not persistent personal memory.
Go deeper: mechanism, diagram and origins
How it works
Hidden states from one segment are reused for the next, with gradients stopped at the boundary. Relative positions avoid confusing positions when those states are reused.
Limits and trade-offs
This segment memory is not the invention of the generation KV cache.
- Original demonstration
- Dai et al.: Transformer-XL.
Papers and sources
Direct link ↗Adapters: adapting a model with few weights
Add small task-specific modules without changing the whole network.
The problem
Keeping a complete copy of a large model for every task consumes substantial storage.
The idea
Small networks are inserted into the stages of an existing model. Adaptation mainly updates these additions while the main model’s learned values remain fixed.
An example to picture the idea
The same main model can be paired with one module for classification and another for a different task.
What this adds
Each task requires far fewer new values to learn and store.
The limit to remember
The modules add calculations and may limit performance depending on the task.
Go deeper: mechanism, diagram and origins
Frozen base weightsSmall learned adaptationAdapted output
How it works
Bottleneck networks are inserted into a pretrained model while its main weights remain frozen. Each task mainly stores its extra modules rather than a full model copy.
Limits and trade-offs
Modules add forward-pass operations and may limit performance.
- Original demonstration
- Houlsby et al.: adapters evaluated on BERT.
Papers and sources
Direct link ↗GPT-2: performing tasks from text
Use text continuation to attempt several tasks without always adapting a separate model.
The problem
Adapting and storing a different model for every task takes additional work.
The idea
The model learns to continue web text. Putting a question or the start of a summary in its context can then steer the kind of continuation it produces.
An example to picture the idea
An input ending with “Question: … Answer:” can steer the continuation toward an answer.
What this adds
The format of the supplied text becomes a way to request different behaviors.
The limit to remember
Predicting a plausible continuation guarantees neither truth nor instruction following.
Go deeper: mechanism, diagram and origins
How it works
A causal decoder predicts continuations over a large web corpus. Context formatting presents a task, such as a question followed by the beginning of an answer.
Limits and trade-offs
Results vary; text prediction guarantees neither truthfulness nor instruction following.
- Original demonstration
- OpenAI: GPT-2, Language Models are Unsupervised Multitask Learners.
Papers and sources
Direct link ↗Product Key Memory: large addressable memory
Efficiently select learned values from a large internal store.
The problem
A large internal store becomes expensive if every entry must be examined fully.
The idea
Search keys are built by combining two smaller sets. This organization helps find the best matches, whose values are then used in the network’s calculation.
An example to picture the idea
A text representation selects a few relevant entries instead of combining all available entries.
What this adds
The network can use a large internal store with a cheaper search.
The limit to remember
These are internal learned values, not an external document database to consult.
Go deeper: mechanism, diagram and origins
How it works
Keys are factorized into two sets to search efficiently over their Cartesian product. The best matches select learned values used in place of some dense transformations.
Limits and trade-offs
These are internal parameters, not an external document database or a conventional MoE.
- Original demonstration
- Lample et al.: experimental language models with PKM layers.
Papers and sources
Direct link ↗Megatron-LM: splitting matrices across GPUs
Share large calculations within one stage across several chips.
The problem
Some operations in a large network are too big for a single chip.
The idea
Large multiplications are divided among computing chips. The chips then exchange their results to reconstruct the output needed by the rest of the network.
An example to picture the idea
Fictional example: two chips each calculate part of the same large table of results.
What this adds
A calculation stage can exceed the capacity of an individual chip.
The limit to remember
Exchanges must be fast, or communication consumes much of the benefit.
Go deeper: mechanism, diagram and origins
How it works
Large multiplications in attention and feed-forward layers are split across devices. Collective communication recombines their results, complementing data parallelism.
Limits and trade-offs
GPUs must communicate quickly; slow interconnects reduce benefits.
- Original demonstration
- Shoeybi et al., NVIDIA: GPT- and BERT-style models with billions of parameters.
Papers and sources
Direct link ↗ZeRO: sharding states instead of duplicating them
Avoid keeping all the same training data on every chip.
The problem
Copying model values and all update information onto every chip wastes memory.
The idea
The method divides update histories among chips. Depending on the version, it also divides updates and learned values, exchanging the required elements when needed.
An example to picture the idea
Each chip keeps part of the update histories instead of a complete copy.
What this adds
Local memory can accommodate a share of a much larger model.
The limit to remember
Data saved from local storage must be exchanged, adding communication.
Go deeper: mechanism, diagram and origins
How it works
Optimizer states, then gradients and parameters depending on the stage, are partitioned across devices. Required data is communicated when it is needed.
Limits and trade-offs
Lower local memory usage requires managing additional communication.
- Original demonstration
- Rajbhandari et al., Microsoft: ZeRO, subsequently integrated into DeepSpeed.
Papers and sources
Direct link ↗RMSNorm: normalizing without centering
Adjust the scale of internal values with a simpler operation.
The problem
Rescaling internal values adds calculations at every network stage.
The idea
The method measures the average magnitude of values and divides them by that scale. Unlike fuller normalization, it does not first subtract their mean.
An example to picture the idea
A group of very large values can be brought to a more regular scale without being centered.
What this adds
The network retains rescaling with fewer operations.
The limit to remember
Getting comparable results depends on the network and how it is trained.
Go deeper: mechanism, diagram and origins
How it works
Each vector is divided by its root mean square and modulated by a learned scale. The method preserves scale invariance with fewer normalization operations.
Limits and trade-offs
Equivalent quality depends on the model and its training.
- Original demonstration
- Zhang and Sennrich: experiments across architectures, before LLaMA.
Papers and sources
Direct link ↗T5: every task becomes text-to-text
Give different tasks one shared form: text in, text out.
The problem
Different formats and structures for every task complicate comparison and reuse.
The idea
The network first learns to reconstruct passages removed from text. It is then adapted to tasks expressed as text, using one part to read and another to produce output.
An example to picture the idea
“Translate: bonjour” and “Summarize: …” both use textual input and textual output.
What this adds
One framework can serve translation, summarization and other tasks.
The limit to remember
A shared format removes neither training cost nor the need for data.
Go deeper: mechanism, diagram and origins
How it works
An encoder-decoder is pretrained to reconstruct masked spans, then adapted to text-formulated tasks. Relative-position biases group distances into buckets.
Limits and trade-offs
A unified format removes neither pretraining cost nor data requirements.
- Original demonstration
- Raffel et al., Google: T5 and the C4 corpus.
Papers and sources
Direct link ↗MQA: sharing keys and values
Let attention’s reading paths share the information kept from earlier text.
The problem
While writing, retaining separate information for every reading path uses substantial memory.
The idea
Paths keep different queries but share keys and values: data used to locate and combine earlier text. Their cache is temporary calculation memory.
An example to picture the idea
Several paths consult the same retained data instead of each keeping a separate set.
What this adds
Temporary memory and the amount of data reread can be reduced.
The limit to remember
Sharing can reduce output quality; speed gains depend on the hardware.
Follow the ideas
Earlier ideas
MQA retains Transformer attention heads but makes them share a single set of keys and values.
Paper: Fast Transformer Decoding: One Write-Head is All You Need ↗
Later developments
- Improved byGQA: sharing keys and values by group
GQA generalizes MQA: several groups share keys and values, instead of one shared set for all heads.
Paper: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints ↗
Go deeper: mechanism, diagram and origins
Queries QKeys K + values VCache to read
How it works
Heads keep different queries but share one set of keys and values. The cache retains previous tokens’ representations so they need not be recomputed at every step.
Limits and trade-offs
Sharing can reduce quality; actual gains depend on hardware and batch size.
- Original demonstration
- Shazeer: Transformer variants tested on translation and language modeling.
Papers and sources
Direct link ↗2020
Reformer: finding similar tokens
Limit comparisons and reconstruct some internal results to handle longer text.
The problem
Comparing every piece of long text and retaining all results uses substantial memory.
The idea
The method groups similar representations to limit comparisons. It also uses reversible stages that reconstruct some previous results instead of storing them.
An example to picture the idea
Pieces placed in the same group are compared without systematically examining every other possible pair.
What this adds
These choices reduce some memory and calculation costs for long sequences.
The limit to remember
Comparisons are approximate and practical benefits depend on the implementation.
Go deeper: mechanism, diagram and origins
How it works
Hashing groups similar vectors to restrict attention comparisons. Reversible layers reconstruct some activations instead of retaining all of them during training.
Limits and trade-offs
Comparisons are approximate, and practical performance depends on kernels.
- Original demonstration
- Kitaev, Kaiser and Levskaya: Reformer.
Papers and sources
Direct link ↗Scaling laws: linking compute, data and size
Measure how results change with model size, data and computation.
The problem
A limited budget forces a choice between a larger model and more training.
The idea
Researchers train models at different sizes and fit curves to their prediction errors. These curves help estimate budget tradeoffs.
An example to picture the idea
Before a large training run, smaller experiments help compare different allocations of computation.
What this adds
These measurements provide a method for planning training expenditure.
The limit to remember
The conclusions depend on the experiments and do not guarantee every capability of a future model.
Go deeper: mechanism, diagram and origins
How it works
Kaplan et al. fit power laws relating loss, parameters, data and compute. These curves suggest how to allocate a budget; they do not describe a new architecture.
Limits and trade-offs
Recommendations depend on the experiments and are later revised, notably by Chinchilla.
- Original demonstration
- OpenAI: experimental series of autoregressive language models.
Papers and sources
Direct link ↗Pre-LN: putting normalization before computation
Rescale values before the network’s major transformations.
The problem
The placement of rescaling can make learning corrections harder to pass through the network.
The idea
Normalization, which adjusts the scale of values, is placed before attention and internal transformations. A direct route also lets information pass through each block.
An example to picture the idea
Instead of normalizing after a transformation, its input is normalized before it is calculated.
What this adds
This arrangement can make the start of training easier.
The limit to remember
This milestone analyzes a placement already used earlier, notably in GPT-2.
Go deeper: mechanism, diagram and origins
How it works
Normalization precedes attention or the feed-forward network inside the residual branch. The paper analyzes gradients and explains why this arrangement facilitates optimization at initialization.
Limits and trade-offs
This marks a 2020 analysis, not the invention: GPT-2 already used this placement.
- Original demonstration
- Xiong et al.: analysis and Pre-LN/Post-LN Transformer experiments.
Papers and sources
Direct link ↗SwiGLU and GEGLU: gated transformations
Learn to modulate one internal transformation with another.
The problem
A single transformation offers less flexibility in selecting which information to pass on.
The idea
The network calculates two versions of its input. One is smoothly transformed, then multiplied value by value with the other before the result is combined.
An example to picture the idea
One branch can reduce selected values in the other without treating every value alike.
What this adds
This mechanism adds learned selection to the network’s internal transformations.
The limit to remember
Comparisons must account for the width of the stages and the computation budget.
Go deeper: mechanism, diagram and origins
Two input projectionsSiLU on one branch × the otherOutput projection
How it works
Two projections are multiplied elementwise, one passing through an activation such as Swish or GELU. An output projection combines the result; widths are adjusted to compare similar budgets.
Limits and trade-offs
Gains depend on budget and training; comparisons must account for different widths.
- Original demonstration
- Shazeer: variants tested in the T5 framework, several years before LLaMA.
Papers and sources
Direct link ↗Longformer: local and global attention
Read mostly nearby text and reserve a few connection points for the whole document.
The problem
Comparing each text piece with every other piece becomes too expensive in a long document.
The idea
Each piece consults a nearby window. Selected positions can also communicate with the whole document, connecting different regions.
An example to picture the idea
A position associated with a question can consult the whole document while others mainly read their neighborhood.
What this adds
The number of comparisons grows more slowly as text gets longer.
The limit to remember
Not every pair communicates directly and global positions must be selected.
Go deeper: mechanism, diagram and origins
How it works
Each token attends to a nearby window; selected tokens also receive global attention. With fixed window size and global-token count, the number of connections grows linearly.
Limits and trade-offs
Not every pair communicates directly, and global tokens must be chosen.
- Original demonstration
- Beltagy, Peters and Cohan: Longformer; LED appears in a later revision.
Papers and sources
Direct link ↗Synthesizer: testing attention without QK products
Combine text pieces without always calculating pairwise matches.
The problem
Classic attention calculates mixing scores by comparing pairs of representations.
The idea
These variants produce information-mixing coefficients differently: with a network or even random values. They test which comparisons are actually necessary.
An example to picture the idea
One variant directly learns mixing coefficients rather than comparing a query with every key.
What this adds
These experiments explore alternative ways for information to flow through a network.
The limit to remember
Removing those comparisons does not automatically make every calculation cheap on long text.
Go deeper: mechanism, diagram and origins
How it works
Some variants directly generate a mixing matrix from representations or use random weights. They avoid query-key products while still combining token values.
Limits and trade-offs
Avoiding QK does not automatically make every variant linear in length.
- Original demonstration
- Tay et al.: experimental Synthesizer models.
Papers and sources
Direct link ↗RAG: retrieving documents before generation
Retrieve documents, then supply them to the model to help prepare its answer.
The problem
A model cannot reliably recover every fact from its learned values alone.
The idea
A search system selects relevant passages from a collection. The generator receives these texts as additional information when producing its answer.
An example to picture the idea
For a question about a policy, the system can retrieve part of the policy and add it to the context.
What this adds
The answer can draw on documents that can be inspected and updated.
The limit to remember
Consulting text does not by itself update the model or guarantee a correct answer.
Go deeper: mechanism, diagram and origins
Question / contextRetrieve passagesCondition generation
How it works
A dense retriever selects passages, then a generator conditions its output on those texts. The paper trains a combination of a retriever and a BART-style model.
Limits and trade-offs
Incorrect retrieval can contaminate answers; finding text does not guarantee correct use.
- Original demonstration
- Lewis et al.: RAG-Sequence and RAG-Token, with a Wikipedia index.
Papers and sources
Direct link ↗GPT-3: learning a task in context
Put examples in the request to steer answers without retraining the model.
The problem
Updating model values for every new task requires data and computation.
The idea
A large model trained to continue text receives an instruction and possibly a few examples. It continues them without changing its learned values during this use.
An example to picture the idea
“chat → cat; chien → dog; maison →” demonstrates a task through examples.
What this adds
The request text is enough to attempt many tasks.
The limit to remember
Results remain sensitive to wording and examples guarantee neither accuracy nor reliable reasoning.
Go deeper: mechanism, diagram and origins
How it works
A causal decoder is pretrained at scale. During evaluation, zero, one or several examples appear in context without changing weights: this is in-context learning.
Limits and trade-offs
Abilities remain prompt-sensitive and guarantee neither reliable reasoning nor truthfulness.
- Original demonstration
- Brown et al., OpenAI: GPT-3, up to 175 billion parameters.
Papers and sources
Direct link ↗Linear attention: summarizing instead of comparing everything
Accumulate a calculable summary of earlier text instead of comparing every pair of pieces.
The problem
Attention that compares every pair becomes expensive on long sequences.
The idea
Representations are transformed and calculations reorganized to accumulate statistics of earlier elements. The current piece consults this compact state rather than a full comparison table.
An example to picture the idea
With each new element, the system updates its state and uses it to calculate the output.
What this adds
Calculation can grow more slowly with length and operate step by step.
The limit to remember
The mechanism changes attention and compresses details of the past into a limited state.
Go deeper: mechanism, diagram and origins
How it works
Queries and keys pass through a feature map. Associativity allows key-value statistics to accumulate in a state that can be queried, enabling recurrent computation.
Limits and trade-offs
The formulation changes attention and compresses history into a fixed-size state.
- Original demonstration
- Katharopoulos et al.: Linear Transformer; one milestone among contemporary approaches.
Papers and sources
Direct link ↗GShard: distributing a giant MoE
Distribute a very large network with subnetworks selected according to the input.
The problem
Greatly increasing network capacity creates storage and computation-distribution problems.
The idea
Some stages are replaced by experts: subnetworks, two of which are selected for each text piece. The system arranges their distribution and exchanges across many chips.
An example to picture the idea
A piece is sent to the selected subnetworks even when they are located on different chips.
What this adds
Large translation models become trainable across a collection of chips.
The limit to remember
All experts must be stored and their exchanges remain costly.
Go deeper: mechanism, diagram and origins
One tokenRouter → selected expertsCombine their outputs
How it works
Some translation Transformer layers are replaced by experts with top-2 routing. The system distributes tensors and organizes communication across many accelerators.
Limits and trade-offs
Total capacity still requires storage and communication even when few experts are active.
- Original demonstration
- Lepikhin et al., Google: multilingual MoE Transformer up to 600 billion parameters.
Papers and sources
Direct link ↗BigBird: local, global and random connections
Connect long-text regions through local, global and random links.
The problem
A local reading window alone may poorly connect distant passages.
The idea
The network combines comparisons with neighbors, a few globally accessible positions and randomly selected links. These paths let information circulate without connecting every pair.
An example to picture the idea
A passage exchanges information with its neighbors and a few distant passages rather than the whole document.
What this adds
Fewer comparisons are needed while paths between distant regions remain available.
The limit to remember
Those paths do not guarantee the same results as full attention on every task.
Go deeper: mechanism, diagram and origins
How it works
BigBird combines local neighborhoods, global tokens and random links. These connections reduce score counts while retaining information paths between distant sections.
Limits and trade-offs
Theoretical guarantees do not imply identical quality to dense attention on every task.
- Original demonstration
- Zaheer et al., Google: BigBird.
Papers and sources
Direct link ↗RLHF: generating summaries preferred by humans
Train a summary generator using evaluators’ preferences.
The problem
Predicting the next text piece is not enough to produce the summary readers prefer.
The idea
People compare summaries. A model learns to assign scores matching their choices, then the generator is trained to improve those scores while remaining close to a reference.
An example to picture the idea
Evaluators may prefer a summary that preserves the main point over one that omits it.
What this adds
Human preferences directly guide part of summary training.
The limit to remember
Evaluator biases and flaws in scoring can be reproduced or exploited.
Go deeper: mechanism, diagram and origins
How it works
Evaluators compare summaries and a reward model learns those comparisons. The generator is then optimized using PPO with a constraint keeping it close to a reference model.
Limits and trade-offs
Evaluator bias and reward exploitation remain possible.
- Original demonstration
- Stiennon et al., OpenAI: GPT-based summarization models, before InstructGPT.
Papers and sources
Direct link ↗Performer: approximating softmax with random features
Approximate attention comparisons without building their complete table.
The problem
The table linking every text piece to every other piece becomes very large.
The idea
Random transformations produce representations that approximate the usual calculation. Operations are rearranged to use these representations without retaining all comparisons.
An example to picture the idea
Instead of filling every cell of a relationship table, approximate numerical summaries are combined.
What this adds
Attention cost can grow more slowly with text length.
The limit to remember
The number of values used in the approximation sets a tradeoff between accuracy and cost.
Go deeper: mechanism, diagram and origins
How it works
FAVOR+ uses positive random features to approximate the attention kernel. Computation is reordered to avoid the full interaction matrix.
Limits and trade-offs
Accuracy and cost depend on feature count; gains are not universal.
- Original demonstration
- Choromanski et al.: Performer and experiments across several modalities.
Papers and sources
Direct link ↗QK-Norm: controlling attention score scale
Adjust representation scale before calculating matches.
The problem
Overly large values can make attention scores excessively concentrated.
The idea
The representations used to make queries and identify consulted elements are brought to a common numerical length. A learned scale then controls the strength of their comparison.
An example to picture the idea
A representation no longer gets a high score simply because its values are larger overall.
What this adds
Matching becomes less dependent on the raw magnitude of representations.
The limit to remember
Recent models may use different variants; their methods are not all identical to this one.
Go deeper: mechanism, diagram and origins
How it works
The paper L2-normalizes Q and K and applies a learned scale to their dot product. Later models use other variants of query and key normalization.
Limits and trade-offs
Variants differ: this exact formulation should not automatically be attributed to every recent model.
- Original demonstration
- Henry, Dachapally, Pawar and Chen: low-resource translation Transformers.
Papers and sources
Direct link ↗ViT: reading an image as a sequence
Process an image as a sequence of small pieces.
The problem
Using a Transformer on images requires converting pixels into suitable inputs.
The idea
The image is split into small squares. Each square becomes a numerical representation with position information, then attention combines these representations into an overall result.
An example to picture the idea
To classify an image, the network combines information from different regions instead of reading text.
What this adds
An attention-based architecture can be used to recognize images.
The limit to remember
Results depend on training and increasing the number of pieces raises comparison costs.
Go deeper: mechanism, diagram and origins
How it works
An image is divided into patches, projected into vectors and given position information. A Transformer encoder mixes these representations to learn a global representation suitable for classification.
Limits and trade-offs
Effectiveness depends on pretraining; more patches increase attention cost.
- Original demonstration
- Dosovitskiy et al., Google: Vision Transformer, ViT.
Papers and sources
Direct link ↗2021
Prefix tuning: learning a continuous prefix
Learn small numerical additions to steer a model that remains fixed.
The problem
Adapting every value of a large model for each task costs storage and memory.
The idea
Extra representations are learned that several network stages can consult like an initial context. Only these additions change during adaptation.
An example to picture the idea
One set of additions can be learned for summarization and used with the unchanged main model.
What this adds
A task can be adapted by learning far fewer new values.
The limit to remember
These additions take context space and their effectiveness depends on the task.
Go deeper: mechanism, diagram and origins
How it works
Trainable vectors act as virtual tokens that layers can attend to. Adaptation learns this prefix rather than updating every network weight.
Limits and trade-offs
Prefixes consume context and may be less effective for some tasks.
- Original demonstration
- Li and Liang: generation experiments on GPT-2 and BART.
Papers and sources
Direct link ↗Switch Transformer: one expert per token
Use only one selected subnetwork for each text piece.
The problem
Networks with multiple experts can become difficult to distribute and stabilize.
The idea
A selector sends each text piece to just one expert, meaning a subnetwork. Rules try to distribute work and avoid overloading the same experts.
An example to picture the idea
Fictional example: each piece goes to one of four experts without running the other three for that piece.
What this adds
Selection becomes simpler while allowing many learned values overall.
The limit to remember
An expert can receive too much work and exchanges between chips remain expensive.
Follow the ideas
Earlier ideas
Switch retains routing to sparse experts and simplifies it: a single expert is chosen for each token.
Paper: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity ↗
Go deeper: mechanism, diagram and origins
One tokenRouter → selected expertsCombine their outputs
How it works
A router sends each token to a single layer expert. Capacity rules, a balancing loss and numerical choices aim to stabilize workload distribution.
Limits and trade-offs
Expert capacity can overflow, and communication remains a major cost.
- Original demonstration
- Fedus, Zoph and Shazeer: Switch Transformer, with T5-derived experiments.
Papers and sources
Direct link ↗DALL·E: generating an image from text
Produce an image by continuing a sequence that begins with its description.
The problem
A text model needs a suitable representation to produce images.
The idea
A system converts images into compact codes. The Transformer learns to predict these codes after the description codes, then a decoder turns them into pixels.
An example to picture the idea
A description such as “a pear-shaped armchair” begins the sequence of image codes.
What this adds
Image generation can be guided by a text description.
The limit to remember
The compressed codes can lose detail and this first version does not use diffusion.
Go deeper: mechanism, diagram and origins
How it works
A discrete autoencoder turns images into tokens. A Transformer then predicts a sequence combining text followed by image tokens; the autoencoder’s decoder reconstructs pixels.
Limits and trade-offs
Compression can lose detail; this first version is not a diffusion model.
- Original demonstration
- Ramesh et al., OpenAI: DALL·E; the date indicates paper submission.
Papers and sources
Direct link ↗CLIP: aligning image and text representations
Learn to match an image with its corresponding description.
The problem
Recognizing new categories often requires labeled examples for each one.
The idea
One network turns images into numbers and another does the same for text. Training brings matching pair representations closer and moves other pairs apart.
An example to picture the idea
A photo can be compared with “a dog” and “a bicycle” to choose the closer description.
What this adds
Written descriptions can serve as categories without new task-specific training.
The limit to remember
Matching images and text is not enough to build an assistant that can converse about an image.
Go deeper: mechanism, diagram and origins
How it works
An image encoder and a text encoder learn a shared space. A contrastive loss pulls matching pairs together and pushes others apart; text descriptions become zero-shot recognition classes.
Limits and trade-offs
Recognition or representation matching alone does not produce multimodal conversation.
- Original demonstration
- Radford et al., OpenAI: CLIP; the date indicates paper submission.
Papers and sources
Direct link ↗Prompt tuning: learning only input tokens
Learn a small numerical request prefix instead of changing the whole model.
The problem
Keeping a complete model version for every task is expensive.
The idea
Learned numerical representations are added to the input, without necessarily corresponding to readable words. Only these additions change during adaptation; the main model’s values remain fixed.
An example to picture the idea
A numerical prefix can be learned to classify reviews and reused before each review.
What this adds
Several tasks can share the same large model with small separate additions.
The limit to remember
Results depend strongly on model size and the task.
Go deeper: mechanism, diagram and origins
How it works
Trainable vectors are added to input embeddings and only those vectors are optimized. Unlike prefix tuning, the method does not directly add a separate prefix at every layer.
Limits and trade-offs
Observed effectiveness depends strongly on model size and task.
- Original demonstration
- Lester, Al-Rfou and Constant: a study on the T5 family.
Papers and sources
Direct link ↗RoPE: rotating queries and keys
Use numerical rotations to represent distances in text.
The problem
Comparisons between text pieces must account for their order and distance.
The idea
The numbers used to seek and identify information are rotated in pairs according to position. Their comparison then depends on the gap between those positions.
An example to picture the idea
Two neighboring words have a rotation difference linked to their proximity even when they occur elsewhere in a sentence.
What this adds
Relative distance between pieces directly affects attention scores.
The limit to remember
This mechanism does not guarantee good performance on text far longer than training examples.
Follow the ideas
Later developments
- Built on byPosition interpolation: stretching RoPE
Interpolation reduces position indices before applying RoPE, fitting a longer text into the learned range.
Paper: Extending Context Window of Large Language Models via Positional Interpolation ↗
Go deeper: mechanism, diagram and origins
RoPE: position becomes a rotation
On small screens, scroll the diagram horizontally.
How it works
Pairs of Q and K components are rotated by position-dependent angles. Their dot product then depends on angle differences and therefore on token distance.
Limits and trade-offs
Positions far beyond training lengths often require specific adaptation.
- Original demonstration
- Su et al.: RoFormer; arXiv v1 on April 20, not April 9, 2021.
Papers and sources
- RoFormer: Enhanced Transformer with Rotary Position Embedding ↗
- LLaMA: Open and Efficient Foundation Language Models ↗
LoRA: adapting with low-rank matrices
Adapt a model by learning a small structured correction.
The problem
Changing every value in a large model requires substantial memory during adaptation.
The idea
Existing learned values remain fixed. Two small numerical tables are learned and multiplied to represent a correction to selected large tables in the model.
An example to picture the idea
To adapt a model to a writing style, these small tables are learned instead of an independent change to every value.
What this adds
Far fewer values need to be trained and stored for each adaptation.
The limit to remember
Table size and targeted stages limit the changes that can be represented.
Follow the ideas
Later developments
- Combined inQLoRA: adapting a quantized model
QLoRA trains LoRA’s small matrices through a frozen base model quantized to 4 bits.
Paper: QLoRA: Efficient Finetuning of Quantized LLMs ↗
Go deeper: mechanism, diagram and origins
LoRA: learn a small update
On small screens, scroll the diagram horizontally.
How it works
Pretrained weights remain frozen. Their update is represented by the product of two small matrices learned during adaptation; the product can later be merged into the original weights.
Limits and trade-offs
Rank and target-layer choices limit what the adaptation can represent.
- Original demonstration
- Hu et al.: demonstrations on RoBERTa, DeBERTa, GPT-2 and GPT-3.
Papers and sources
Direct link ↗ALiBi: directly penalizing distances
Add a comparison penalty that increases with distance.
The problem
Attention calculations need to distinguish nearby elements from distant ones.
The idea
Each reading path subtracts a penalty proportional to distance between pieces. Paths use different slopes without adding a position representation to every piece.
An example to picture the idea
With equal content relevance, a nearby word receives a higher score than a more distant word.
What this adds
The same position rule can be applied to lengths absent from training.
The limit to remember
Applying the rule to long text does not guarantee finding information at every distance.
Go deeper: mechanism, diagram and origins
Token positionModify Q/K or the scoreDistance relationships
How it works
Each head subtracts a distance-proportional penalty from attention scores using its own slope. This favors nearby tokens and applies to lengths unseen during training.
Limits and trade-offs
Measured extrapolation does not guarantee reliable retrieval at every distance.
- Original demonstration
- Press, Smith and Lewis: experimental autoregressive models, before BLOOM.
Papers and sources
Direct link ↗FLAN: learning to follow instructions
Train the model on many tasks presented as instructions.
The problem
A model that can continue text does not necessarily follow a new request.
The idea
An existing model is adapted using examples pairing an instruction with an expected output. Some task families are held aside to test following other instructions.
An example to picture the idea
The model sees requests such as “translate this sentence” and “classify this review”, with their expected answers.
What this adds
This training can improve results on tasks absent from adaptation.
The limit to remember
Transfer depends on the quality and diversity of the supplied tasks.
Go deeper: mechanism, diagram and origins
How it works
A pretrained model is fine-tuned on examples whose inputs contain instructions. Evaluation uses held-out task families to test transfer beyond training tasks.
Limits and trade-offs
Task quality and diversity determine transfer; this is not RLHF.
- Original demonstration
- Wei et al., Google: FLAN, derived from a 137-billion-parameter language model.
Papers and sources
Direct link ↗NormFormer: balancing gradients with extra normalization
Add scale adjustments to better balance learning across stages.
The problem
Learning corrections can have very different sizes at different network stages.
The idea
The method adds rescaling after attention and within internal transformations. It also learns a scale for each reading path to reduce some imbalances.
An example to picture the idea
A stage receiving much larger corrections than another may benefit from these scale adjustments.
What this adds
Corrections can become better balanced and training more effective in the studied experiments.
The limit to remember
These operations add cost and their usefulness depends on the configuration.
Go deeper: mechanism, diagram and origins
How it works
The block adds normalization after attention, per-head scaling and normalization inside the feed-forward network. These changes reduce some gradient-scale mismatches across layers.
Limits and trade-offs
Additional operations have a cost and do not imply universal adoption.
- Original demonstration
- Shleifer, Weston and Ott: NormFormer, experimental language models.
Papers and sources
Direct link ↗S4: modeling sequences with a structured state
Process a long sequence by updating a compact numerical state.
The problem
Keeping and separately comparing every past element becomes expensive.
The idea
Each new input changes a limited-size state according to a structured mathematical system. This structure enables parallel calculation during training and step-by-step updates during use.
An example to picture the idea
When a new element arrives, the system updates its state without individually rereading all previous elements.
What this adds
This approach offers another way to process long sequences.
The limit to remember
The state compresses the past and does not provide direct access to every old detail.
Follow the ideas
Later developments
- Improved byMamba: selective memory
Mamba extends state-space models by making what is retained or forgotten depend on the input.
Paper: Mamba: Linear-Time Sequence Modeling with Selective State Spaces ↗
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
A dynamical system updates a compact state from inputs. Its structure enables efficient convolution-kernel computation during training and recurrent processing during generation.
Limits and trade-offs
The state compresses history and differs from direct token-to-token retrieval.
- Original demonstration
- Gu, Goel and Ré: S4; 2111.00396, not 2208.04933, which is S5.
Papers and sources
Direct link ↗RETRO: integrating retrieval into the architecture
Build consultation of similar passages into the model’s calculations.
The problem
Storing all useful information only in learned values requires very large models.
The idea
The system retrieves passages neighboring the current segment from a large collection. Dedicated network stages read these passages and incorporate them into prediction of the next text.
An example to picture the idea
A segment discussing a subject can be accompanied by similar passages retrieved from the collection.
What this adds
The experiments show that a searchable collection can support a model with fewer learned values.
The limit to remember
The system depends on retrieval quality and the availability of its collection.
Go deeper: mechanism, diagram and origins
Question / contextRetrieve passagesCondition generation
How it works
The model retrieves neighboring text chunks with a frozen retriever. An encoder and cross-attention layers integrate retrieved passages during token prediction.
Limits and trade-offs
The system depends on the quality and availability of its document index.
- Original demonstration
- Borgeaud et al., DeepMind: Retrieval-Enhanced Transformer, RETRO.
Papers and sources
Direct link ↗2022
Chain-of-Thought: intermediate steps
Breaking an answer into steps can help solve some problems.
The problem
A direct answer sometimes skips a useful calculation.
The idea
Showing examples with written intermediate steps encourages the model to write similar steps before producing its final answer.
An example to picture the idea
Fictional example: calculate a subtotal before applying a discount.
What this adds
Some problems become easier to solve.
The limit to remember
A plausible explanation can be wrong or fail to reflect the internal computation.
Follow the ideas
Later developments
Self-consistency generates several Chain-of-Thought reasoning paths, then selects the most frequent answer.
Paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models ↗
Go deeper: mechanism, diagram and origins
Problem + demonstrationsGenerate intermediate stepsProduce an answer
How it works
The model imitates demonstrations that show intermediate steps before the answer. These additional tokens split calculations or deductions into simpler operations.
Limits and trade-offs
Plausible reasoning guarantees neither correctness nor a faithful explanation.
- Original demonstration
- GPT-3 and LaMDA in early experiments; PaLM in later versions.
Papers and sources
Direct link ↗Chinchilla: balancing parameters and data
A smaller model can learn better when given more text.
The problem
Making a model bigger can exhaust the budget before it has learned enough.
The idea
Balance the number of learned values against the amount of training text.
An example to picture the idea
Fictional example: train a smaller model longer instead of enlarging its counterpart.
What this adds
The same budget can yield better results.
The limit to remember
The best balance changes when the cost of answering is included.
Go deeper: mechanism, diagram and origins
How it works
Experiments vary model size, token count and compute. Fitted laws recommend scaling parameters and data together to use the training budget efficiently.
Limits and trade-offs
The optimum changes when inference cost or different data distributions are included.
- Original demonstration
- Chinchilla 70B, compared notably with Gopher 280B.
Papers and sources
Direct link ↗InstructGPT: following human intent
Human examples and preferences make answers more useful.
The problem
Completing text is not enough to follow a request.
The idea
People demonstrate useful answers and rank alternatives to the same request. A learned scoring system then guides further training.
An example to picture the idea
Fictional example: prefer a clear answer over an irrelevant continuation.
What this adds
The model follows some instructions better.
The limit to remember
An answer that people prefer can still contain factual errors.
Follow the ideas
Earlier ideas
InstructGPT applies reward learning from human preferences to tune a language model.
Paper: Training language models to follow instructions with human feedback ↗
Other approaches
- Alternative toDPO: learning preferences directly
DPO learns directly from preferred and rejected responses, without training the separate reward model used in the InstructGPT recipe.
Paper: Direct Preference Optimization: Your Language Model is Secretly a Reward Model ↗
Go deeper: mechanism, diagram and origins
How it works
After supervised demonstrations, humans rank responses. A reward model learns these preferences and guides reinforcement learning with PPO.
Limits and trade-offs
Quality depends on collected preferences; factual errors remain.
- Original demonstration
- InstructGPT, adapted from GPT-3; this paper does not introduce ChatGPT.
Papers and sources
Direct link ↗µP / µTransfer: tune small, train large
Experiments on a small model help configure its larger counterpart.
The problem
Testing every setting directly on a large model is expensive.
The idea
A rule adjusts the scales of learned values and their updates as size increases.
An example to picture the idea
Fictional example: choose the learning speed using a reduced version.
What this adds
Some settings transfer with fewer expensive experiments.
The limit to remember
Transfer requires compatible versions and does not work for every architectural change.
Go deeper: mechanism, diagram and origins
How it works
µP adjusts weight and update scales to network width. Hyperparameters are tuned on a smaller model and then transferred.
Limits and trade-offs
Transfer requires consistent parameterization; arbitrary architecture changes remain difficult.
- Original demonstration
- Demonstrations on Transformers, BERT and GPT-style models.
Papers and sources
Direct link ↗Self-consistency: voting across reasoning paths
Comparing several solutions sometimes helps select an answer.
The problem
A single attempt can follow an unsuccessful path.
The idea
The model generates several different sequences of steps for the same question, then the most frequent final answer is selected.
An example to picture the idea
Fictional example: five attempts return the same result three times.
What this adds
Voting can eliminate some isolated errors.
The limit to remember
The attempts consume computation and may all repeat the same mistake.
Follow the ideas
Earlier ideas
Self-consistency generates several Chain-of-Thought reasoning paths, then selects the most frequent answer.
Paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models ↗
Go deeper: mechanism, diagram and origins
How it works
Sample several reasoning chains, then select the most frequent final answer. Different paths can converge on the same solution.
Limits and trade-offs
Cost grows with attempts, and correlated mistakes can win the vote.
- Original demonstration
- Experiments on GPT-3, LaMDA and PaLM families in the paper.
Papers and sources
Direct link ↗STaR: learning from successful reasoning
Successful solutions become new training examples.
The problem
Human-written demonstrations are limited.
The idea
The model generates worked solutions to questions with known answers. Those producing a correct answer are retained as examples for further training.
An example to picture the idea
Fictional example: retain a solution that reaches the expected total.
What this adds
More examples become available for learning.
The limit to remember
A correct final answer does not guarantee that every intermediate step is sound.
Go deeper: mechanism, diagram and origins
How it works
The model generates rationales; those reaching the correct answer become fine-tuning data. For failures, access to the answer helps generate another rationale.
Limits and trade-offs
A correct answer can hide faulty reasoning; filtering is crucial.
- Original demonstration
- STaR, demonstrated notably with GPT-J.
Papers and sources
Direct link ↗Flamingo: connecting vision and language
Visual components enable answers about images mixed with text.
The problem
A text-only model does not directly receive information from an image.
The idea
One component condenses the image, then reading stages connect this information to the text.
An example to picture the idea
Fictional example: show a photograph and ask for its dominant colour.
What this adds
The model can use examples containing both images and sentences.
The limit to remember
Compression can lose details, and answers remain fallible.
Go deeper: mechanism, diagram and origins
How it works
A Perceiver Resampler compresses visual representations. Cross-attention layers connect them to a language model while large pretrained components remain frozen.
Limits and trade-offs
Visual compression loses detail and does not eliminate hallucinations.
- Original demonstration
- Flamingo, DeepMind; not the origin of multimodality as a whole.
Papers and sources
Direct link ↗PaLM: training across TPU pods
Large-scale distributed training explores the abilities of bigger models.
The problem
A very large model exceeds the capacity of a single machine.
The idea
A system coordinates many chips to train the different parts of the model together.
An example to picture the idea
Fictional example: distribute a long computation across several groups of machines.
What this adds
The experiment measures the effects of increasing scale.
The limit to remember
This model combines earlier techniques rather than inventing all of them.
Go deeper: mechanism, diagram and origins
How it works
Pathways coordinates PaLM training across TPU groups. The model combines earlier architecture choices and tests how scale improves few-shot learning and prompted reasoning.
Limits and trade-offs
This is a large-scale synthesis, not the invention of its component techniques.
- Original demonstration
- PaLM 540B, Google; paper in April 2022.
Papers and sources
Direct link ↗FlashAttention: moving less data
Computing attention in small blocks saves working memory.
The problem
Comparing every text piece creates a large intermediate table.
The idea
Computation proceeds in blocks in the chip's fast memory, without storing the complete table.
An example to picture the idea
Fictional example: process several groups of words in succession.
What this adds
The same mathematical attention needs fewer memory transfers.
The limit to remember
The number of comparisons still grows with the square of the text length.
Follow the ideas
Later developments
- Improved byFlashAttention-2: better GPU utilization
The second version retains tiled computation and distributes work more effectively across GPU units.
Paper: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning ↗
Go deeper: mechanism, diagram and origins
FlashAttention: change the memory traffic
On small screens, scroll the diagram horizontally.
How it works
Computation is tiled into blocks held in fast GPU memory. An online softmax avoids materializing the entire attention score matrix.
Limits and trade-offs
Operation count remains quadratic; gains depend on hardware and dimensions.
- Original demonstration
- Kernel by Dao et al., evaluated on text and vision Transformers.
Papers and sources
Direct link ↗IA³: tuning a few multipliers
Small additional adjustments can sometimes specialise a model.
The problem
Updating all learned values requires considerable memory.
The idea
The main learned values remain fixed. Small learned lists of numbers strengthen or reduce selected internal information during processing.
An example to picture the idea
Fictional example: adapt a model to classify messages.
What this adds
Each specialisation needs few additional values.
The limit to remember
This adaptation offers less flexibility than full training and requires suitable examples.
Go deeper: mechanism, diagram and origins
Frozen base weightsSmall learned adaptationAdapted output
How it works
Learned vectors rescale keys, values and internal activations while main weights remain frozen. Each task therefore needs only small additional parameter sets.
Limits and trade-offs
This adaptation is less expressive than full fine-tuning and needs task data.
- Original demonstration
- IA³, demonstrated with T0/T5 in the T-Few framework.
Papers and sources
Direct link ↗OPT: documenting large-model training
Publishing models and a development log makes large models easier to study.
The problem
A result alone does not reveal the difficulties encountered while building it.
The idea
The release provides several model sizes, code and observations about training.
An example to picture the idea
Fictional example: compare two sizes using a documented recipe.
What this adds
Researchers can study more concrete design choices.
The limit to remember
Access to the files does not mean that every use is permitted.
Go deeper: mechanism, diagram and origins
How it works
The family uses autoregressive decoder Transformers at several scales. The publication exposes infrastructure and stability challenges beyond describing the network alone.
Limits and trade-offs
The milestone concerns access and documentation; permitted use depends on published terms.
- Original demonstration
- OPT 125M–175B, Meta; not a new attention mechanism.
Papers and sources
Direct link ↗UL2: mixing learning objectives
Varying text reconstruction exercises teaches several ways to handle requests.
The problem
A single training exercise prepares poorly for some tasks.
The idea
The model learns to recover hidden passages of different lengths, with markers identifying the exercise.
An example to picture the idea
Fictional example: complete a word, then reconstruct a paragraph.
What this adds
One recipe covers several kinds of text transformation.
The limit to remember
Benefits vary with the tasks and the way the model is served.
Go deeper: mechanism, diagram and origins
How it works
Mixture-of-Denoisers combines different masking rates and span lengths. Mode markers indicate the expected reconstruction task to the encoder-decoder model.
Limits and trade-offs
Benefits depend on tasks; encoder-decoder serving has specific constraints.
- Original demonstration
- UL2 20B, Google.
Papers and sources
Direct link ↗Fill-in-the-Middle: completing between two spans
The model fills a gap using the text that follows it too.
The problem
Ordinary continuation only sees what comes before.
The idea
Training presents the beginning and ending before asking for the missing passage.
An example to picture the idea
Fictional example: complete a function between its declaration and return statement.
What this adds
Insertions can fit both sides of the text.
The limit to remember
The model must have learned this particular format to use it correctly.
Go deeper: mechanism, diagram and origins
How it works
Texts are split into prefix, middle and suffix, then reordered so the model predicts the middle after seeing the other two. Training remains autoregressive.
Limits and trade-offs
Format mixtures and markers must be learned; not every model supports them.
- Original demonstration
- Experimental OpenAI models in the paper, notably for code.
Papers and sources
Direct link ↗Orca: continuous batching
A server replaces completed requests without waiting for the others.
The problem
Waiting for the longest text to finish leaves capacity unused.
The idea
At each generation step, the server inserts new requests into the freed slots.
An example to picture the idea
Fictional example: start one question while another answer continues.
What this adds
More requests can progress on the same hardware.
The limit to remember
The server must also limit waiting times and handle different text lengths.
Go deeper: mechanism, diagram and origins
How it works
The scheduler decides at each iteration which sequences advance one token. Completed requests leave the batch and new requests enter immediately.
Limits and trade-offs
Scheduling must also preserve latency and handle variable sequence lengths.
- Original demonstration
- Orca system, published at OSDI 2022; not a new model.
Papers and sources
Direct link ↗S5: simplifying state-space models
A structured internal memory efficiently processes long sequences.
The problem
Comparing every position with all the others becomes expensive.
The idea
A state summarises the past. A mathematical structure allows several updates to be calculated in parallel.
An example to picture the idea
Fictional example: process a long series of measurements.
What this adds
Processing can make efficient use of parallel computation.
The limit to remember
Efficiency on sequences does not guarantee the abilities of a general-purpose assistant.
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
S5 replaces multiple single-input/output systems with a multivariate system initialized from S4. An associative scan parallelizes the recurrence.
Limits and trade-offs
This is a sequence-modeling milestone; efficiency does not guarantee LLM capabilities.
- Original demonstration
- S5, Smith et al.; 2208.04933 is S5, not S4.
Papers and sources
Direct link ↗FP8: eight-bit floating-point computation
Storing numbers with lower precision reduces some computation costs.
The problem
Detailed numerical representations take considerable memory.
The idea
Some numbers use eight bits, with suitable scaling factors to limit rounding errors and represent a useful range of values.
An example to picture the idea
Fictional example: represent a value using a coarser rounding step.
What this adds
Compatible chips compute and transfer these numbers more efficiently.
The limit to remember
Not every operation tolerates this reduced precision without difficulty.
Go deeper: mechanism, diagram and origins
How it works
Two eight-bit floating-point formats trade precision against numerical range. Scaling factors help represent activations, weights or gradients without excessive overflow.
Limits and trade-offs
Suitable kernels and hardware are required; some operations retain higher precision.
- Original demonstration
- Joint NVIDIA, Intel and Arm proposal; not a single model.
Papers and sources
Direct link ↗Flan: generalizing through instructions
Varied instructions prepare a model for new requests.
The problem
Learning only to continue text is not enough to follow instructions.
The idea
The model trains on many different tasks described in ordinary language, instead of practising just one kind of request.
An example to picture the idea
Fictional example: alternate translation, summarisation and sentence classification.
What this adds
It can better follow some instructions never encountered in exactly that form.
The limit to remember
Diverse examples do not guarantee success on every new task.
Go deeper: mechanism, diagram and origins
How it works
A model is fine-tuned on many instructions, with or without reasoning demonstrations. The paper jointly studies task count, model size and supervision formats.
Limits and trade-offs
Instruction diversity and quality matter; the model does not automatically learn every new task.
- Original demonstration
- Flan-T5 and Flan-PaLM; extends earlier FLAN work.
Papers and sources
Direct link ↗GPTQ: post-training weight quantization
Carefully rounding learned values reduces the model's memory requirements.
The problem
A trained model can be too large for the available hardware.
The idea
Its learned values are rounded progressively, compensating for some rounding errors using a small collection of representative calibration examples.
An example to picture the idea
Fictional example: retain fewer numerical levels for the values.
What this adds
The model can fit in less memory.
The limit to remember
Quality can decrease, and speed improvements depend on the hardware.
Go deeper: mechanism, diagram and origins
Weights + calibration dataChoose a quantizationStore fewer bits
How it works
A small calibration corpus estimates rounding error effects. Weights are quantized incrementally while errors are compensated in the remaining weights of each layer.
Limits and trade-offs
Accuracy can decrease; reduced memory does not imply speedup on every GPU.
- Original demonstration
- GPTQ, Frantar et al., evaluated notably on OPT and BLOOM.
Papers and sources
Direct link ↗ReAct: reason, act, observe
Alternating text and tool calls allows external information to be used.
The problem
The model cannot infer everything from the text already available.
The idea
It produces an action, receives a tool's result, then continues its answer.
An example to picture the idea
Fictional example: consult a source before summarising its contents.
What this adds
The continuation can account for a recent observation.
The limit to remember
Choosing the wrong tool or misreading its result remains possible and takes time.
Go deeper: mechanism, diagram and origins
How it works
The prompt alternates textual reasoning, a tool action and an observation of its result. This loop revises plans using external information.
Limits and trade-offs
Tool and planning errors remain; calls add latency.
- Original demonstration
- Demonstrations by Yao et al., notably with PaLM-540B.
Papers and sources
Direct link ↗BLOOM: a collaborative multilingual model
A collective project documents training a model in many languages.
The problem
Resources and documentation often concentrate on a few languages.
The idea
A collaboration assembles multilingual text and publishes its model alongside its training recipe.
An example to picture the idea
Fictional example: compare answers in two languages from the collection.
What this adds
More languages and training choices become available for study.
The limit to remember
Including a language does not guarantee quality equal to the others.
Go deeper: mechanism, diagram and origins
How it works
The decoder Transformer learns text continuation across many natural and programming languages. It adopts ALiBi positional biases in a distributed training recipe.
Limits and trade-offs
Multilingual coverage does not imply equal quality across languages or instruction following.
- Original demonstration
- BLOOM 176B, BigScience; the paper does not credit it with inventing ALiBi.
Papers and sources
Direct link ↗SmoothQuant: taming activation outliers
Redistributing large values makes computation with rounded numbers easier.
The problem
A few extreme values make rounding internal calculations difficult.
The idea
A transformation transfers some of this range into the model's fixed values before rounding.
An example to picture the idea
Fictional example: rebalance two factors without changing their original product.
What this adds
Some computations become suitable for more compact integers.
The limit to remember
Calibration must suit the intended uses, and some loss of precision remains possible.
Go deeper: mechanism, diagram and origins
How it works
An equivalent transformation rescales channels, moving part of activation outliers into the weights. This makes W8A8 quantization more manageable.
Limits and trade-offs
Calibration depends on data and model-specific accuracy tradeoffs.
- Original demonstration
- SmoothQuant, demonstrated on OPT, BLOOM, GLM and other LLMs.
Papers and sources
Direct link ↗Speculative decoding: draft, then verify
A small model proposes text that a large model checks together.
The problem
Generating each text piece separately multiplies expensive processing passes.
The idea
A fast model proposes a continuation. The main model checks it using a corrected acceptance rule.
An example to picture the idea
Fictional example: propose several words and keep the accepted beginning.
What this adds
The exact method preserves the main model's probabilities.
The limit to remember
The benefit disappears when proposals are too costly or too often rejected.
Go deeper: mechanism, diagram and origins
Draft multiple tokensVerify with the target modelAccept a prefix
How it works
A fast model drafts a continuation; the target model checks it in parallel. An acceptance and correction rule preserves the target distribution in the exact method.
Limits and trade-offs
Speedup depends on acceptance rate and drafting cost.
- Original demonstration
- Leviathan et al.: T5-XXL; contemporary work by Chen et al.
Papers and sources
Direct link ↗Constitutional AI: learning from AI feedback
Written principles guide answer revision and evaluation.
The problem
Having a person evaluate every example requires substantial work.
The idea
A model critiques and revises proposed answers against written principles, then generates comparisons between answers to guide further training.
An example to picture the idea
Fictional example: request removal of an unnecessarily insulting claim.
What this adds
Some evaluation examples can be generated automatically.
The limit to remember
The chosen principles and automatic evaluators can introduce their own errors.
Go deeper: mechanism, diagram and origins
How it works
A model critiques and revises responses using explicit principles. AI-generated preferences then train a reward model and a reinforcement-learning stage.
Limits and trade-offs
Principles and judges introduce their own biases; control remains imperfect.
- Original demonstration
- Experimental Anthropic assistants in the paper.
Papers and sources
Direct link ↗H3: structured memory for language
A summarised memory is enriched to retrieve certain associations.
The problem
A summary of the past can lose useful links between items.
The idea
Computations combine a stored state with current content to retain and retrieve associations.
An example to picture the idea
Fictional example: connect a name with information provided earlier.
What this adds
The method improves some retrieval abilities of recurrent models.
The limit to remember
The memory remains compressed and can lose important information.
Go deeper: mechanism, diagram and origins
How it works
H3 combines state-space layers with multiplicative interactions to store and retrieve associations. The paper also explores hybrids with a few attention layers.
Limits and trade-offs
Recurrent compression can lose information; replacing all attention remains a tradeoff.
- Original demonstration
- Hungry Hungry Hippos, Fu et al..
Papers and sources
Direct link ↗Self-Instruct: generating instruction data
A model creates new instructions from a few examples.
The problem
Writing a large collection of tasks by hand takes time.
The idea
Human examples start the generation of new instructions and answers. A filtering stage removes cases judged invalid or too repetitive.
An example to picture the idea
Fictional example: create a new summarisation request.
What this adds
More examples become available for training.
The limit to remember
The generator's mistakes can also end up in the training data.
Go deeper: mechanism, diagram and origins
How it works
A model proposes instructions, inputs and outputs from examples. Filtering removes invalid or overly similar cases before supervised fine-tuning.
Limits and trade-offs
Generator errors and biases propagate into the corpus; diversity needs monitoring.
- Original demonstration
- Self-Instruct, Wang et al., using GPT-3.
Papers and sources
Direct link ↗Whisper: learning speech at scale
Varied audio training enables transcription in several languages.
The problem
Real recordings differ greatly from clean laboratory audio.
The idea
The model learns to turn sound representations into written text using a large collection of recordings paired with transcriptions.
An example to picture the idea
Fictional example: transcribe a short sentence spoken with an accent.
What this adds
Transcription can better withstand some audio variations.
The limit to remember
On noise or silence, the model can invent text.
Go deeper: mechanism, diagram and origins
How it works
A Transformer encodes audio spectrograms and decodes text and task markers. Training mixes transcription, language identification and translation into English.
Limits and trade-offs
The model can hallucinate text over noise; it is not a complete conversational assistant.
- Original demonstration
- Whisper, OpenAI; December paper follows the model release.
Papers and sources
Direct link ↗2023
Hyena: long convolutions and gates
Long filters offer another way to mix information across a sequence.
The problem
Full attention compares all positions and becomes expensive.
The idea
Filters combine distant positions, while content controls some of the information flow.
An example to picture the idea
Fictional example: process a long passage without a table of every comparison.
What this adds
The cost can grow more slowly than full attention.
The limit to remember
This efficiency does not guarantee equally precise retrieval of every detail.
Go deeper: mechanism, diagram and origins
How it works
Hyena composes implicitly parameterized long convolutions with input-dependent gates. Convolutions can be computed efficiently without a full attention matrix.
Limits and trade-offs
Good asymptotic cost does not ensure the same precise retrieval as explicit attention.
- Original demonstration
- Hyena models by Poli et al.
Papers and sources
Direct link ↗Lion: a lighter optimizer
A learning rule retains less tracking information.
The problem
Training also stores numbers used to guide each update.
The idea
Lion mainly uses the direction of a combination of recent updates and retains reduced tracking information.
An example to picture the idea
Fictional example: keep a direction without storing every indicator used by Adam.
What this adds
Tracking the learning process takes less memory.
The limit to remember
The update speed needs tuning, and benefits vary between experiments.
Go deeper: mechanism, diagram and origins
How it works
Lion uses the sign of a combination of gradient and moving average to choose update direction. It keeps only one momentum state per parameter.
Limits and trade-offs
Learning rate needs adjustment; gains vary with task and batch size.
- Original demonstration
- Discovered through program search, evaluated on vision and language.
Papers and sources
Direct link ↗LLaMA: an efficient, reusable recipe
Relatively compact models popularise an efficient recipe for research.
The problem
Studying a very large model requires resources that are difficult to obtain.
The idea
LLaMA combines several existing techniques with extended training on public text.
An example to picture the idea
Fictional example: test an adaptation on a smaller version.
What this adds
More teams can experiment with these models.
The limit to remember
The model popularises a combination of components without having invented each one.
Go deeper: mechanism, diagram and origins
How it works
LLaMA combines earlier techniques: RMSNorm pre-normalization, SwiGLU activation and RoPE positions. Extended training on public data also prioritizes inference efficiency.
Limits and trade-offs
The model popularizes a combination; it does not separately invent these components.
- Original demonstration
- LLaMA 7B–65B, Meta.
Papers and sources
Direct link ↗Toolformer: learning when to call a tool
The model learns where a tool call can help continue text.
The problem
A few manual demonstrations do not cover every useful call.
The idea
The model proposes calls. Those improving prediction of the following text are retained for training.
An example to picture the idea
Fictional example: insert a calculator call before a result.
What this adds
Some examples of tool use are generated automatically.
The limit to remember
Improving a continuation does not guarantee good organisation of a long task.
Go deeper: mechanism, diagram and origins
How it works
The model inserts candidate calls into text. Calls whose results improve next-token prediction are retained, then used for training.
Limits and trade-offs
A call useful for prediction does not ensure good long-task planning.
- Original demonstration
- Toolformer, built from GPT-J 6B.
Papers and sources
Direct link ↗LLaVA: following instructions about an image
A connection between vision and text enables questions about an image.
The problem
Image information is not directly readable by a text model.
The idea
A component transforms visual information. Example conversations teach the model how to use it.
An example to picture the idea
Fictional example: ask which objects appear in a photograph.
What this adds
The model can follow some visual instructions.
The limit to remember
Some details remain inaccessible, and absent objects may be mentioned.
Go deeper: mechanism, diagram and origins
How it works
A projector connects a CLIP image encoder to a language model. Partly synthetic visual instruction data teaches the model to answer image-related requests.
Limits and trade-offs
Encoder and resolution limit visible detail; answers may hallucinate.
- Original demonstration
- LLaVA, Liu et al., using Vicuna and CLIP.
Papers and sources
Direct link ↗DoReMi: choosing the data mixture
Choosing the proportions of text sources can improve learning.
The problem
A large collection can devote too much space to less useful sources.
The idea
A small model tests how to weight text categories. The resulting mixture is then used for the larger model.
An example to picture the idea
Fictional example: revise the proportions of books and web pages.
What this adds
The same volume of data can become more useful.
The limit to remember
The mixture need not transfer to every size and task.
Go deeper: mechanism, diagram and origins
How it works
A small proxy model adjusts domain weights through robust optimization. The large model is then trained on the resulting mixture.
Limits and trade-offs
Domains and proxy model affect results; transfer is not universal.
- Original demonstration
- DoReMi, Xie et al., tested on corpus mixtures.
Papers and sources
Direct link ↗DPO: learning preferences directly
Directly comparing two answers simplifies learning from preferences.
The problem
Some methods require a separate evaluator and an expensive training loop.
The idea
Pairs identify which answer people prefer for the same request. Training adjusts their probabilities relative to a fixed reference version.
An example to picture the idea
Fictional example: prefer a clear summary to a confusing one.
What this adds
The preference-training procedure has fewer components.
The limit to remember
Learning preferences guarantees neither truthfulness nor freedom from bias.
Follow the ideas
Other approaches
- Alternative toInstructGPT: following human intent
DPO learns directly from preferred and rejected responses, without training the separate reward model used in the InstructGPT recipe.
Paper: Direct Preference Optimization: Your Language Model is Secretly a Reward Model ↗
Go deeper: mechanism, diagram and origins
Preferred / rejected answerPreference lossUpdate the model
How it works
The loss compares both responses' probabilities relative to a reference model. It reformulates regularized preference optimization without a separate reward model or PPO.
Limits and trade-offs
Out-of-distribution preferences and pair-dataset biases can limit results.
- Original demonstration
- DPO, Rafailov et al., experiments on sentiment, summarization and dialogue.
Papers and sources
Direct link ↗GQA: sharing keys and values by group
Groups of reading paths share part of their memory.
The problem
Each independent reading path normally stores its own information about past text.
The idea
Each path formulates its own query but shares keys and values: the information used to locate and read the past.
An example to picture the idea
Fictional example: two groups of two paths each share the same stored data.
What this adds
The cache, temporary storage of computations about past text, becomes smaller.
The limit to remember
Sharing can reduce quality depending on the model and group count.
Follow the ideas
Earlier ideas
- ImprovesMQA: sharing keys and values
GQA generalizes MQA: several groups share keys and values, instead of one shared set for all heads.
Paper: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints ↗
Other approaches
- Alternative toMLA: compressing keys and values into a latent
MLA also reduces cache memory, but compresses keys and values into a small vector instead of sharing them in groups.
Paper: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model ↗
Go deeper: mechanism, diagram and origins
Queries QKeys K + values VCache to read
How it works
Several query heads share a key/value head pair. Group count directly controls cache memory; the paper also studies converting existing models.
Limits and trade-offs
Fewer groups can reduce quality; the best choice depends on the model.
- Original demonstration
- Ainslie et al., T5 conversion experiments.
Papers and sources
Direct link ↗MPT: assembling components for an open model
A public model family shows how to combine several efficient techniques.
The problem
Reproducing a recipe requires more than a list of good ideas.
The idea
MPT combines existing choices about position representation, attention computation and learning, with additional versions adapted to particular uses.
An example to picture the idea
Fictional example: continue training on long stories.
What this adds
Teams gain a concrete combination to study.
The limit to remember
Accepting long inputs does not guarantee good use of every distant detail.
Go deeper: mechanism, diagram and origins
How it works
MPT-7B adopts ALiBi, FlashAttention and the Lion optimizer. StoryWriter continues training on long passages to use a larger context window.
Limits and trade-offs
This is technique adoption; positional extrapolation does not ensure all long-range understanding.
- Original demonstration
- MPT-7B, MosaicML, official announcement May 5, 2023.
Papers and sources
Direct link ↗NoPE: questioning explicit positions
Order can sometimes be learned without adding explicit position markers.
The problem
Encoding each text piece's position imposes representation choices.
The idea
The model uses only the restriction against reading future positions to learn some ordering relationships.
An example to picture the idea
Fictional example: compare sequences longer than those used in training.
What this adds
Some controlled tasks show interesting generalisation.
The limit to remember
These tests do not demonstrate general superiority for large language models.
Go deeper: mechanism, diagram and origins
How it works
The model adds no explicit positional encoding. The authors compare length generalization against RoPE, ALiBi and other schemes on controlled tasks.
Limits and trade-offs
Controlled-task results do not prove general superiority for LLMs.
- Original demonstration
- Study by Kazemnejad et al., experimental Transformers.
Papers and sources
Direct link ↗Process supervision: scoring each step
Evaluating steps helps identify where a solution becomes wrong.
The problem
A final score does not reveal which step caused the error.
The idea
People annotate steps. An evaluator trained on those annotations then ranks several solutions.
An example to picture the idea
Fictional example: flag the first incorrect calculation in a demonstration.
What this adds
Selection can favour solutions with stronger intermediate steps.
The limit to remember
Annotations require work, and the evaluator can also make mistakes.
Go deeper: mechanism, diagram and origins
How it works
Annotators evaluate steps rather than only final answers. A process reward model then ranks candidate solutions to help select stronger ones.
Limits and trade-offs
Annotations are costly and a verifier remains fallible beyond its training data.
- Original demonstration
- Let's Verify Step by Step, Lightman et al.; PRM800K dataset.
Papers and sources
Direct link ↗QLoRA: adapting a quantized model
A compressed model can be specialised using small trainable additions.
The problem
Adapting a large model normally requires considerable memory.
The idea
The base values remain fixed and compressed. Only small additions, called adapters, are trained.
An example to picture the idea
Fictional example: adapt answer style without retraining the base.
What this adds
Specialisation can fit in less graphics memory.
The limit to remember
Not all training takes place in four bits, and results depend on the settings.
Follow the ideas
Earlier ideas
- Combines withLoRA: adapting with low-rank matrices
QLoRA trains LoRA’s small matrices through a frozen base model quantized to 4 bits.
Paper: QLoRA: Efficient Finetuning of Quantized LLMs ↗
Go deeper: mechanism, diagram and origins
QLoRA: adapt a quantized base
On small screens, scroll the diagram horizontally.
How it works
Base weights stay frozen in four-bit NF4 while small LoRA adapters are trained. Double quantization and paged optimizers further reduce memory needs.
Limits and trade-offs
This is not full four-bit training; quality and speed depend on configuration.
- Original demonstration
- QLoRA, Dettmers et al.; Guanaco models derived from LLaMA.
Papers and sources
Direct link ↗RWKV: recurrence and parallel training
A fixed-size state replaces detailed storage of the entire past.
The problem
Memory used by conventional attention grows with the text.
The idea
The model updates a summary state for each text piece, while allowing parallel training.
An example to picture the idea
Fictional example: continue an answer without adding a memory record for every word.
What this adds
Storage for the past can remain constant during generation.
The limit to remember
Older details are compressed and do not all remain directly accessible.
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
Time mixing and channel mixing combine gates with weighted aggregation. The formulation supports recurrent evaluation and sequence-parallel training.
Limits and trade-offs
Finite state compresses history rather than keeping individually addressable token caches.
- Original demonstration
- RWKV, Peng et al.; paper notably describing RWKV-4.
Papers and sources
Direct link ↗AWQ: protecting important weights
Protecting influential information helps compress an already trained model.
The problem
Rounding every value the same way can seriously degrade answers.
The idea
Examples identify influential channels, then a change of scale reduces their rounding error.
An example to picture the idea
Fictional example: better preserve a sensitive group of values.
What this adds
The model can use less memory without complete retraining.
The limit to remember
Calibration must remain representative, and acceleration depends on available implementations.
Go deeper: mechanism, diagram and origins
Weights + calibration dataChoose a quantizationStore fewer bits
How it works
Calibration activations identify important channels. Rescaling weights before quantization reduces their error without retraining the whole network.
Limits and trade-offs
Calibration must remain representative and acceleration requires suitable kernels.
- Original demonstration
- AWQ, Lin et al., evaluated on multiple LLM families.
Papers and sources
Direct link ↗Position interpolation: stretching RoPE
Bringing positions closer adapts a model to longer texts.
The problem
Positions far beyond those seen during training can disrupt the model.
The idea
New positions are mapped into the familiar range, followed by a short period of further training.
An example to picture the idea
Fictional example: fit a longer ruler onto the same scale.
What this adds
The usable text window can be extended.
The limit to remember
The ruler describes scaling only: not every detail becomes equally accessible.
Follow the ideas
Earlier ideas
- Builds onRoPE: rotating queries and keys
Interpolation reduces position indices before applying RoPE, fitting a longer text into the learned range.
Paper: Extending Context Window of Large Language Models via Positional Interpolation ↗
Later developments
- Improved byYaRN: selectively extending RoPE
YaRN refines interpolation by treating RoPE rotation frequencies differently and adjusting attention scaling.
Paper: YaRN: Efficient Context Window Extension of Large Language Models ↗
Go deeper: mechanism, diagram and origins
Token positionModify Q/K or the scoreDistance relationships
How it works
Long-sequence positions are mapped into RoPE's familiar interval. The model is then fine-tuned to use the new scale without abrupt extrapolation.
Limits and trade-offs
An advertised window does not guarantee uniform retrieval of all information.
- Original demonstration
- Chen et al., extending LLaMA to 32K.
Papers and sources
Direct link ↗FlashAttention-2: better GPU utilization
Distributing work better speeds up exact attention computation.
The problem
A chip can wait while some parts of the computation remain busy.
The idea
The implementation distributes blocks better and reduces communication and additional operations.
An example to picture the idea
Fictional example: share computation for one reading path across several units.
What this adds
The chip can spend more time on useful calculation.
The limit to remember
Benefits depend on hardware, and comparisons grow with the square of the length.
Follow the ideas
Earlier ideas
- ImprovesFlashAttention: moving less data
The second version retains tiled computation and distributes work more effectively across GPU units.
Paper: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning ↗
Later developments
- Improved byFlashAttention-3: exploiting Hopper
The third version adapts this exact attention to Hopper GPUs by overlapping transfers and computation.
Paper: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision ↗
Go deeper: mechanism, diagram and origins
FlashAttention: change the memory traffic
On small screens, scroll the diagram horizontally.
How it works
The kernel reduces non-matrix operations and better distributes blocks from one head across GPU units. It also reduces communication between thread groups.
Limits and trade-offs
Gains depend on tensor shapes and GPU; complexity remains quadratic.
- Original demonstration
- FlashAttention-2, Tri Dao, demonstrations on GPT-style models.
Papers and sources
Direct link ↗Llama 2: documenting a dialogue model
A documented family combines base models with conversational versions.
The problem
A text-continuation model still needs adaptation to act as an assistant.
The idea
After learning from text, demonstrations and human preferences guide the conversational versions.
An example to picture the idea
Fictional example: prefer answering the question over merely continuing its wording.
What this adds
The publication describes more of the adaptation recipe.
The limit to remember
Its components predate the model, and results depend on the chosen examples.
Go deeper: mechanism, diagram and origins
How it works
Base models receive further pretraining, then Chat variants undergo supervised and human-preference tuning. The 70B variant adopts GQA to reduce attention cache.
Limits and trade-offs
Components predate the release; behavior also depends on instruction and preference datasets.
- Original demonstration
- Llama 2 and Llama 2-Chat, Meta; a synthesis and adoption milestone.
Papers and sources
Direct link ↗RetNet: memory in three forms
One mechanism supports parallel learning and generation with summarised memory.
The problem
A fast way to learn is not always practical for answering.
The idea
A retention rule combines information and gradually reduces the influence of some older positions.
An example to picture the idea
Fictional example: update a state after every new word.
What this adds
Several computation forms implement the same mechanism.
The limit to remember
The state compresses the past, and performance must be compared at equivalent quality.
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
Retention combines query-key interactions with temporal decay. Equivalent formulations support parallel, recurrent or chunkwise recurrent computation.
Limits and trade-offs
Recurrent state summarizes history; gains need comparison at matched quality.
- Original demonstration
- RetNet, Sun et al., Microsoft.
Papers and sources
Direct link ↗SARATHI: chunking the prefill
Splitting long inputs reduces disruption to answers already being generated.
The problem
Reading a long request in one go temporarily monopolises the chip.
The idea
The server processes the input in chunks, mixed with steps for answers already underway.
An example to picture the idea
Fictional example: read part of a report between two generation steps.
What this adds
The hardware can distribute its work better.
The limit to remember
Chunks that are too small or too large worsen some waiting times.
Go deeper: mechanism, diagram and origins
How it works
Prompt prefill is split into chunks. The scheduler mixes chunks with decoding steps to improve GPU use and reduce imbalance.
Limits and trade-offs
Chunk size trades throughput against latency and scheduling overhead.
- Original demonstration
- SARATHI system, Agrawal et al.
Papers and sources
Direct link ↗YaRN: selectively extending RoPE
Adapting position scales differently helps extend accepted text length.
The problem
Extending every position scale in the same way can degrade a model.
The idea
Some scales are stretched, others preserved, followed by further training to adapt the model.
An example to picture the idea
Fictional example: retain nearby landmarks while expanding long distances.
What this adds
The usable window can grow with less adaptation effort.
The limit to remember
Long-distance quality still needs measurement for each model.
Follow the ideas
Earlier ideas
YaRN refines interpolation by treating RoPE rotation frequencies differently and adjusting attention scaling.
Paper: YaRN: Efficient Context Window Extension of Large Language Models ↗
Go deeper: mechanism, diagram and origins
Token positionModify Q/K or the scoreDistance relationships
How it works
YaRN interpolates some RoPE frequencies while preserving others, together with attention scaling adjustments. Additional training adapts the model to the new window.
Limits and trade-offs
Tuning and actual long-context quality depend on the model.
- Original demonstration
- Peng et al., LLaMA extension experiments; v1 on August 31.
Papers and sources
Direct link ↗PagedAttention: paging the KV cache
Reusable blocks organise conversation memory more efficiently.
The problem
Reserving one large continuous space per request wastes memory.
The idea
The information stored about past text is divided into small blocks linked by a table. Identical beginnings can share these blocks.
An example to picture the idea
Fictional example: expand an answer's storage block by block.
What this adds
The server can accommodate more simultaneous requests.
The limit to remember
Blocks need management, and storage for past text is still required.
Go deeper: mechanism, diagram and origins
Logical token sequenceBlock tableNon-contiguous KV blocks
How it works
The cache is split into physical blocks linked through a logical table. Sequences no longer require large contiguous allocations; prefixes can share blocks.
Limits and trade-offs
Paging does not eliminate the cache and adds block management.
- Original demonstration
- vLLM / PagedAttention, Kwon et al.; paper date, not first code.
Papers and sources
Direct link ↗StreamingLLM: retaining attention sinks
Keeping the beginning and recent past stabilises some prolonged generations.
The problem
Simply removing old pieces from the cache can disrupt attention.
The idea
The model retains a few initial pieces and a recent window while removing the others.
An example to picture the idea
Fictional example: keep a stream's opening and its recent sentences.
What this adds
Generation can continue with bounded memory.
The limit to remember
Removed content is no longer accessible: this is not unlimited memory.
Go deeper: mechanism, diagram and origins
How it works
Alongside a recent window, retain initial tokens that attract attention. These sinks stabilize computation as older tokens are removed from cache.
Limits and trade-offs
Evicted information is no longer accessible: unlimited streaming does not mean unlimited memory.
- Original demonstration
- StreamingLLM, Xiao et al., evaluated notably on LLaMA.
Papers and sources
Direct link ↗Mistral 7B: sliding windows and GQA
Local reading and shared memory make a compact model more economical.
The problem
Reading and storing the entire text at every stage is expensive.
The idea
Each stage reads a nearby window. Several reading paths also share some stored information.
An example to picture the idea
Fictional example: directly consult neighbouring sentences during generation.
What this adds
The cache, temporary storage for the past, can become smaller.
The limit to remember
Distant text is not directly accessible at every stage.
Go deeper: mechanism, diagram and origins
Queries QKeys K + values VCache to read
How it works
Each layer attends to a local window; stacking propagates information beyond it. GQA shares keys/values and a rolling buffer bounds window storage.
Limits and trade-offs
Direct access to distant tokens is limited; this is not free global attention.
- Original demonstration
- Mistral 7B; October paper follows September release.
Papers and sources
Direct link ↗Falcon: web data and shared attention
A recipe documents the value of text cleaning and economical computation choices.
The problem
Adding raw text does not guarantee efficient learning.
The idea
The family combines cleaned web text with several choices that reduce computation and memory costs.
An example to picture the idea
Fictional example: remove repetitive content before training.
What this adds
The report makes these decisions available for joint study.
The limit to remember
Variants differ, and the report follows the models' first releases.
Go deeper: mechanism, diagram and origins
How it works
The recipe emphasizes corpus cleaning and efficient computation choices, including attention key/value sharing depending on the variant. The report compares architecture and data decisions.
Limits and trade-offs
The November report follows model releases; MQA and GQA are not interchangeable.
- Original demonstration
- Falcon family, TII; paper November 28, 2023.
Papers and sources
Direct link ↗Mamba: selective memory
Content determines how a fixed-size memory is updated.
The problem
A rigid summary of the past can retain the wrong information.
The idea
Each text piece controls updates to an internal state, computed efficiently during training.
An example to picture the idea
Fictional example: update the state differently depending on the word received.
What this adds
Generation retains a state whose size does not grow with the text.
The limit to remember
This compression can lose older details that need exact retrieval.
Follow the ideas
Earlier ideas
Mamba extends state-space models by making what is retained or forgotten depend on the input.
Paper: Mamba: Linear-Time Sequence Modeling with Selective State Spaces ↗
Later developments
Mamba-2 reformulates Mamba’s selective mechanism through the duality between state spaces and structured attention.
Paper: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality ↗- Combined inJamba: mixing Mamba and attention
Jamba combines Mamba layers with Transformer attention layers and expert layers.
Paper: Jamba: A Hybrid Transformer-Mamba Language Model ↗
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
State-space parameters depend on the input. An efficient GPU scan computes the recurrence during training, while decoding keeps only fixed-size state.
Limits and trade-offs
This compression can weaken exact retrieval of distant context elements.
- Original demonstration
- Mamba, Albert Gu and Tri Dao.
Papers and sources
Direct link ↗RadixAttention: reusing prefixes across calls
Reusing an identical beginning avoids repeating the same computation.
The problem
Several requests can start with the same long text.
The idea
A tree organises memory blocks for shared beginnings and allows their reuse.
An example to picture the idea
Fictional example: ask several questions after the same initial instruction.
What this adds
Computation already performed on that beginning can be saved.
The limit to remember
The beginning must match exactly, and blocks must remain in a compatible cache.
Go deeper: mechanism, diagram and origins
Prefix shared by requestsFind computed KV blocksCompute only the continuation
How it works
A radix tree organizes KV-cache blocks by shared prefixes. SGLang retains and reuses those blocks across calls, beyond one generation's internal cache; eviction frees memory.
Limits and trade-offs
Identical prefixes and compatible caches are required; evicted blocks must be recomputed.
- Original demonstration
- SGLang / RadixAttention, Zheng et al.; v1 December 12, 2023, not invention of caching in general.
Papers and sources
Direct link ↗2024
DeepSeekMoE: fine-grained and shared experts
Small specialised subnetworks divide the computation.
The problem
Activating an entire large network for every text piece is expensive.
The idea
A selection component chooses several small networks, called experts. Others remain active for shared information.
An example to picture the idea
Fictional example: activate a small selection of experts for a word.
What this adds
Total capacity can increase without computing through every expert.
The limit to remember
Their workload must be balanced, and data may need moving between machines.
Go deeper: mechanism, diagram and origins
MoE: choose experts for each token
On small screens, scroll the diagram horizontally.
How it works
Expert networks are split more finely, allowing the router to combine more small experts. Always-active experts capture common knowledge to reduce redundancy among routed experts.
Limits and trade-offs
Load balance and communication remain major constraints.
- Original demonstration
- DeepSeekMoE, demonstrated notably at 16B total parameters.
Papers and sources
Direct link ↗DistServe: separating prefill and decoding
Separating input reading from answer generation allows different resource allocations.
The problem
These phases do not use machines in the same way.
The idea
Separate chip groups process the input and generate the answer, transferring the computed memory between them.
An example to picture the idea
Fictional example: assign long documents to a dedicated group.
What this adds
The server can better meet some response-time targets.
The limit to remember
Transferring memory consumes network capacity and can cancel the benefit.
Go deeper: mechanism, diagram and origins
How it works
Prompt processing and token generation run on separate GPU groups. The system sizes these groups and transfers KV cache between phases.
Limits and trade-offs
Cache transfer consumes network bandwidth; gains depend on infrastructure and traffic.
- Original demonstration
- DistServe, system by Zhong et al.; not a model architecture.
Papers and sources
Direct link ↗EAGLE: speculating on representations
A small predictor proposes continuations using internal information.
The problem
An ordinary draft does not always exploit the main model's computation.
The idea
A small component predicts internal numerical representations rather than only text, and builds proposals that the main model then checks.
An example to picture the idea
Fictional example: check several candidate word sequences together.
What this adds
More frequently accepted proposals can accelerate generation.
The limit to remember
The extra component needs training, and its proposals can be rejected.
Go deeper: mechanism, diagram and origins
Draft multiple tokensVerify with the target modelAccept a prefix
How it works
The drafter predicts target-model internal features, also conditioned on tokens. It builds candidate continuations verified through speculative sampling.
Limits and trade-offs
The predictor must be trained and served; gains depend on acceptance rate.
- Original demonstration
- EAGLE, Li et al., tested notably on Llama 2 and Vicuna.
Papers and sources
Direct link ↗Medusa: multiple drafting heads
Additional outputs propose several future positions at once.
The problem
Producing one text piece per pass limits speed.
The idea
Small components added to the model propose several continuations, arranged in a tree for joint checking.
An example to picture the idea
Fictional example: propose several next words and their continuations.
What this adds
Some computation passes can validate more text.
The limit to remember
The additions require training, and some acceptance rules change the original probabilities.
Go deeper: mechanism, diagram and origins
Draft multiple tokensVerify with the target modelAccept a prefix
How it works
Heads trained on LLM representations predict several future positions. Tree attention then verifies combinations of proposed tokens together.
Limits and trade-offs
Heads require training; not every acceptance rule preserves the distribution.
- Original demonstration
- Medusa, Cai et al., demonstrated on open models including Vicuna.
Papers and sources
Direct link ↗Mixtral: opening sparse experts
Activating only selected experts reduces computation in a high-capacity model.
The problem
Using every learned value for each word would be expensive.
The idea
A selection component chooses two expert networks out of eight for each text piece.
An example to picture the idea
Fictional example: only the two selected paths perform their specialised computation.
What this adds
Active computation remains below that of all experts combined.
The limit to remember
Every expert still needs storage, and distributing them complicates execution.
Follow the ideas
Earlier ideas
Mixtral uses the sparse-expert principle: each token is sent to two of the eight experts in each MoE layer.
Paper: Mixtral of Experts ↗
Go deeper: mechanism, diagram and origins
MoE: choose experts for each token
On small screens, scroll the diagram horizontally.
How it works
At each layer, a router selects two of eight expert networks for each token. Attention remains shared; only some expert weights participate in computation.
Limits and trade-offs
All experts must be stored; routing complicates distributed execution.
- Original demonstration
- Mixtral 8x7B, released December 2023; paper in January 2024.
Papers and sources
Direct link ↗BitNet b1.58: ternary weights
Learning with three main values reduces a model's potential cost.
The problem
Detailed numerical values take considerable space.
The idea
Training constrains some values to be negative, zero or positive, with a suitable shared scale.
An example to picture the idea
Fictional example: represent a coefficient as minus one, zero or one.
What this adds
Suitable implementations can exploit this compact representation.
The limit to remember
Total memory includes more than these values, and acceleration requires appropriate hardware support.
Go deeper: mechanism, diagram and origins
How it works
Linear-layer weights take −1, 0 or +1 with scaling factors; activations are also quantized. Training incorporates the constraint from the start.
Limits and trade-offs
1.58 bits describes ternary weights, not total memory; gains need suitable kernels.
- Original demonstration
- BitNet b1.58, extending earlier BitNet.
Papers and sources
Direct link ↗GRPO: comparing a group of responses
Comparing answers within a group simplifies some reward-based training.
The problem
A large auxiliary evaluator adds cost.
The idea
Several answers to the same question receive scores. Their relative scores guide the model's updates.
An example to picture the idea
Fictional example: favour solutions that pass a test within the group.
What this adds
A large auxiliary network estimating expected success can be avoided.
The limit to remember
Generation remains expensive, and a bad reward rule teaches undesirable behaviour.
Follow the ideas
Earlier ideas
GRPO adapts PPO by comparing rewards for several answers to the same question, without a separate value critic.
Paper: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models ↗
Go deeper: mechanism, diagram and origins
Multiple answers to one problemGroup-relative rewardsPolicy update
How it works
For one question, generate several answers and compare rewards within the group. This relative baseline estimates the advantage used to update the model with regularization.
Limits and trade-offs
Generation cost remains and reward quality is critical.
- Original demonstration
- DeepSeekMath 7B; GRPO predates DeepSeek-R1.
Papers and sources
Direct link ↗KIVI: quantizing the KV cache
Rounding stored information differently reduces storage for the past.
The problem
Temporary text memory becomes large for long inputs.
The idea
Keys, used to locate information, and values, used to read it, are compressed differently. A recent portion retains higher precision.
An example to picture the idea
Fictional example: compress the older blocks of a conversation.
What this adds
Storage for the past decreases without retraining the model.
The limit to remember
Quality and speed depend on settings and the computation implementation.
Go deeper: mechanism, diagram and origins
How it works
Keys are quantized per channel and values per token because their outliers are distributed differently. A small recent portion remains at higher precision.
Limits and trade-offs
Accuracy and throughput depend on groups, context and kernels.
- Original demonstration
- KIVI, Liu et al., evaluated on LLMs including Llama and Falcon.
Papers and sources
Direct link ↗LongRoPE: stretching context nonuniformly
Searching position scales extends some text windows.
The problem
One uniform position setting does not suit every distance.
The idea
A search chooses different scaling factors for position components, then progressively adapts the model while also checking its behaviour on shorter inputs.
An example to picture the idea
Fictional example: test several settings on short and long passages.
What this adds
Some models can accept much longer inputs.
The limit to remember
The advertised length does not guarantee reliable understanding of every passage.
Go deeper: mechanism, diagram and origins
Token positionModify Q/K or the scoreDistance relationships
How it works
LongRoPE adjusts RoPE dimensions and positions with nonuniform factors. Progressive extension and readjustment also aim to preserve short-sequence performance.
Limits and trade-offs
Millions of advertised tokens do not guarantee reliable understanding throughout the window.
- Original demonstration
- LongRoPE, Ding et al., experimental LLaMA and Mistral extensions.
Papers and sources
Direct link ↗Jamba: mixing Mamba and attention
Combining summarised memory and attention retains two ways of using the past.
The problem
Compact memory can lose details that attention retrieves directly.
The idea
The model alternates summary-state stages with attention stages, adding expert networks in some places.
An example to picture the idea
Fictional example: retain a few direct accesses to text among compact stages.
What this adds
Storage for the past can decrease.
The limit to remember
Multiple mechanisms complicate computation implementations and memory allocation.
Follow the ideas
Earlier ideas
- Combines withMamba: selective memory
Jamba combines Mamba layers with Transformer attention layers and expert layers.
Paper: Jamba: A Hybrid Transformer-Mamba Language Model ↗ - Combines withTransformer: attention becomes the architecture
Jamba retains attention layers within a model that also uses Mamba’s recurrent memory.
Paper: Jamba: A Hybrid Transformer-Mamba Language Model ↗
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
Jamba interleaves Mamba and Transformer layers and adds MoE experts to some layers. The presented configuration uses many more Mamba than attention layers.
Limits and trade-offs
Multiple layer types complicate kernels and resource allocation.
- Original demonstration
- Jamba, AI21 Labs.
Papers and sources
Direct link ↗Infini-attention: compressive memory
A condensed memory connects earlier passages to local reading.
The problem
A short window no longer provides access to earlier passages.
The idea
Each block combines reading nearby text with an updated summary of older segments.
An example to picture the idea
Fictional example: retain condensed associations after a paragraph has passed.
What this adds
A stream can exceed the local window with bounded memory.
The limit to remember
Older passages are compressed with losses and do not remain fully accessible.
Go deeper: mechanism, diagram and origins
How it works
Each block combines local attention with linear memory of past segments. A recurrent update compresses previous associations and a gate mixes the two outputs.
Limits and trade-offs
The past is compressed lossily; long streams do not become fully addressable.
- Original demonstration
- Infini-attention, Munkhdalai et al., experimental 1B and 8B models.
Papers and sources
Direct link ↗MiniCPM: transferable tuning and WSD schedule
A stable learning phase makes continued training easier.
The problem
Reducing learning speed too early makes continuation less convenient.
The idea
The speed rises, stays stable, then falls at the end. Small experiments also help select settings.
An example to picture the idea
Fictional example: extend the plateau before the final decrease.
What this adds
The recipe facilitates some experiments with smaller models.
The limit to remember
Transferring settings depends on the recipe and is not universal.
Go deeper: mechanism, diagram and origins
How it works
Small-scale experiments tune hyperparameters with Tensor Programs. WSD separates warmup, a stable plateau and final learning-rate decay, easing continued training.
Limits and trade-offs
Transferring settings depends on the training recipe and model architecture.
- Original demonstration
- MiniCPM, OpenBMB; first submission April 9, 2024.
Papers and sources
Direct link ↗Mixture-of-Depths: selecting which tokens compute
Some text pieces pass through fewer computation stages.
The problem
Sending every position through every stage spends a uniform budget.
The idea
A selection component chooses which text positions are processed by some stages. The other positions continue through the model without that particular computation.
An example to picture the idea
Fictional example: reserve an expensive stage for selected positions.
What this adds
Active computation can be reduced.
The limit to remember
Selection must respect generation order and remain efficient on the hardware.
Go deeper: mechanism, diagram and origins
How it works
A router scores tokens and selects a fixed-size subset for some layers. Others take the residual path, saving that computation.
Limits and trade-offs
Routing must remain compatible with causality and hardware constraints.
- Original demonstration
- Mixture-of-Depths, Raposo et al., experimental models.
Papers and sources
Direct link ↗Multi-Token Prediction: learning several futures
Learning several future positions enriches the training signal.
The problem
Predicting only the next piece provides a very local objective.
The idea
Several outputs share the main computation and learn to predict different upcoming positions.
An example to picture the idea
Fictional example: learn the next word and the one after it.
What this adds
These outputs can also supply proposals to accelerate generation.
The limit to remember
Future positions remain uncertain, and their proposals need checking.
Go deeper: mechanism, diagram and origins
Draft multiple tokensVerify with the target modelAccept a prefix
How it works
Several heads share a common trunk and learn different future positions. They provide a richer training signal and can support speculative decoding.
Limits and trade-offs
Future predictions remain uncertain; acceleration needs suitable verification.
- Original demonstration
- Gloeckle et al., Meta, experimental code and language models.
Papers and sources
Direct link ↗Mamba-2 / SSD: connecting attention and recurrence
A mathematical structure makes recurrent-memory computation more efficient.
The problem
Some successive updates make poor use of chips' parallel capabilities.
The idea
A new formulation groups updates into blocks suited to matrix multiplication.
An example to picture the idea
Fictional example: process a segment together rather than every position separately.
What this adds
Chips can make better use of their computation units.
The limit to remember
The mathematical equivalence applies to certain structures, not every form of attention.
Follow the ideas
Earlier ideas
- ImprovesMamba: selective memory
Mamba-2 reformulates Mamba’s selective mechanism through the duality between state spaces and structured attention.
Paper: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality ↗
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
State-space duality describes some attention and recurrent mechanisms using related structured matrices. Mamba-2 uses this structure to compute blocks efficiently through matrix multiplication.
Limits and trade-offs
The equivalence concerns specific structures; arbitrary softmax attention is not an equivalent recurrence.
- Original demonstration
- Mamba-2, Tri Dao and Albert Gu.
Papers and sources
Direct link ↗MLA: compressing keys and values into a latent
A condensed representation reduces the memory used to reread the past.
The problem
Storing every attention key and value takes considerable space.
The idea
The model stores a shorter list of numbers and a separate position key. Computation uses this condensed version.
An example to picture the idea
Fictional example: retain a shorter numerical description per position.
What this adds
Memory can decrease substantially compared with conventional attention.
The limit to remember
Depending on the chosen dimensions, this cache can exceed one shared set for all reading paths.
Follow the ideas
Other approaches
- Alternative toGQA: sharing keys and values by group
MLA also reduces cache memory, but compresses keys and values into a small vector instead of sharing them in groups.
Paper: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model ↗
Go deeper: mechanism, diagram and origins
Queries QKeys K + values VCache to read
How it works
A low-rank projection compresses keys and values into a cached latent vector. Rearranged transformations and separate positional handling compute attention from this compression.
Limits and trade-offs
The reported 93.3% reduction compares DeepSeek-V2 with DeepSeek 67B; it is not universal.
- Original demonstration
- DeepSeek-V2.
Papers and sources
Direct link ↗DeltaNet: correcting associative memory
A targeted correction updates associations in summarised memory.
The problem
Adding new information can leave an older incorrect association in place.
The idea
The rule compares the value associated with an item against its newly expected value, then writes their difference.
An example to picture the idea
Fictional example: correct an association instead of simply adding it again.
What this adds
Memory can update some information more effectively.
The limit to remember
It remains finite in size and can lose details from the past.
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
Instead of merely adding a key-value association, the delta rule subtracts the current prediction for that key and writes the correction. The paper parallelizes these updates across sequences.
Limits and trade-offs
Memory remains finite; the paper improves an earlier delta rule.
- Original demonstration
- Yang et al., DeltaNet with a parallel algorithm; not invention of the delta rule.
Papers and sources
Direct link ↗EAGLE-2: adapting the draft tree
The draft explores continuations it estimates as promising more extensively.
The problem
A fixed proposal tree spends computation on less useful branches.
The idea
The small predictor's scores shape the tree of continuations the main model will check.
An example to picture the idea
Fictional example: extend a likely branch instead of an unlikely one.
What this adds
The proposal budget can be used more effectively.
The limit to remember
Scores remain imperfect, and a large tree adds work of its own.
Go deeper: mechanism, diagram and origins
Draft multiple tokensVerify with the target modelAccept a prefix
How it works
EAGLE-2 builds a candidate tree using drafter confidence scores. Its structure changes with position and probabilities rather than remaining fixed.
Limits and trade-offs
Larger trees add computation; tree shape must fit the hardware.
- Original demonstration
- EAGLE-2, Li et al., extending EAGLE.
Papers and sources
Direct link ↗Nemotron-4: generating and filtering synthetic data
A generator and an evaluator organise the production of training data.
The problem
Manually producing and checking many examples is expensive.
The idea
One model generates example answers and another gives them quality scores, so that suitable examples can be selected for further training.
An example to picture the idea
Fictional example: retain an answer judged clearer for future adaptation.
What this adds
The pipeline can provide many synthetic examples.
The limit to remember
Errors and biases from both generator and evaluator can accumulate.
Go deeper: mechanism, diagram and origins
How it works
Instruct models produce responses and a Reward model evaluates quality. The report documents post-training based overwhelmingly on synthetic examples and releases the associated pipeline.
Limits and trade-offs
This does not invent synthetic data; generator and judge errors can accumulate.
- Original demonstration
- Nemotron-4 340B, NVIDIA; an example of scaling an earlier approach.
Papers and sources
Direct link ↗FlashAttention-3: exploiting Hopper
Overlapping transfers and computation better uses some recent chips.
The problem
Computation units can wait for data to arrive.
The idea
The implementation advances several attention operations simultaneously and also offers a reduced-precision variant.
An example to picture the idea
Fictional example: load one block while computing the previous one.
What this adds
Attention can run faster on the targeted chips.
The limit to remember
Benefits depend strongly on hardware, and the low-precision variant adds numerical approximation.
Follow the ideas
Earlier ideas
The third version adapts this exact attention to Hopper GPUs by overlapping transfers and computation.
Paper: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision ↗
Go deeper: mechanism, diagram and origins
FlashAttention: change the memory traffic
On small screens, scroll the diagram horizontally.
How it works
The kernel overlaps memory transfers, matrix multiplication and softmax. An FP8 variant adds transformations to limit quantization error on Hopper GPUs.
Limits and trade-offs
Benefits strongly depend on GPU architecture; FP8 introduces numerical approximation.
- Original demonstration
- FlashAttention-3, Shah et al., Hopper optimizations.
Papers and sources
Direct link ↗Gemma 2: distilling and bounding scores
A teacher model provides richer learning targets for smaller models.
The problem
Small models have less capacity to learn independently.
The idea
They learn a teacher model's probabilities and alternate local and global reading of text.
An example to picture the idea
Fictional example: learn several continuations considered possible by the teacher.
What this adds
Compact models can achieve better results.
The limit to remember
Producing targets costs computation, and their weaknesses can come from the teacher.
Go deeper: mechanism, diagram and origins
How it works
Smaller models learn teacher distributions; the architecture alternates local windows and global attention. Soft-capping smoothly bounds attention scores and output logits.
Limits and trade-offs
Distillation depends on the teacher and adds target-generation costs.
- Original demonstration
- Gemma 2, Google; the report does not describe QK-Norm as its technique.
Papers and sources
Direct link ↗Llama 3: a documented training recipe
A report brings together improvements behind a new model generation.
The problem
Changing one component does not explain every difference in results.
The idea
The recipe combines text selection, large-scale computation, longer inputs and preference adaptation.
An example to picture the idea
Fictional example: improve both training data and conversation examples.
What this adds
The report makes the overall development process available for study.
The limit to remember
It documents a combination without dating the invention of every technique.
Go deeper: mechanism, diagram and origins
How it works
The report documents data filtering and mixing, large-scale training, context extension and repeated supervised and preference tuning. It also presents multimodal extensions.
Limits and trade-offs
This is a combination of techniques; the report does not date their individual invention.
- Original demonstration
- The Llama 3 Herd of Models report, Meta, including Llama 3.1.
Papers and sources
Direct link ↗Test-time compute: adapting effort to the problem
Spending more computation on some questions can improve answers.
The problem
The same effort does not suit both easy and difficult questions.
The idea
Revisions and solution searches are compared, using a verifier and a budget adapted to estimated difficulty.
An example to picture the idea
Fictional example: test more possibilities for a difficult problem.
What this adds
Answers can improve without enlarging the model.
The limit to remember
Benefits depend on the verifier, and extra attempts take time.
Go deeper: mechanism, diagram and origins
How it works
The paper compares successive revisions and verifier-guided search. It allocates computation according to estimated question difficulty instead of applying one strategy to every problem.
Limits and trade-offs
Gains depend on model, difficulty and verifier; this is not always better than training larger.
- Original demonstration
- Snell et al., experimental study of models and solving strategies; v1 August 6, 2024.
Papers and sources
Direct link ↗Hyper-Connections: enriching residual paths
Several internal paths enrich exchanges between model stages.
The problem
A single path combines results in a relatively constrained way.
The idea
Several streams replace that path, and learned coefficients control their mixtures before and after each stage.
An example to picture the idea
Fictional example: combine two streams with different weights.
What this adds
The model gains more ways to pass information onwards.
The limit to remember
These mixtures can increase memory, computation and training difficulties.
Go deeper: mechanism, diagram and origins
How it works
Multiple representation streams replace a single residual path. Learned coefficients control how streams feed a layer and how its output updates them.
Limits and trade-offs
More streams and mixing can increase memory, cost and stability challenges.
- Original demonstration
- Hyper-Connections, Zhu et al., ByteDance.
Papers and sources
Direct link ↗OLMoE: making MoE models inspectable
An open recipe makes expert routing easier to study.
The problem
Without data and analysis, routing choices are difficult to understand.
The idea
The model selects eight expert networks out of sixty-four and releases a recipe with comparative experiments.
An example to picture the idea
Fictional example: compare two rules for selecting experts.
What this adds
Researchers can examine several design decisions.
The limit to remember
Openness removes neither the storage of all experts nor their costly communication.
Go deeper: mechanism, diagram and origins
One tokenRouter → selected expertsCombine their outputs
How it works
The model activates eight of sixty-four experts per token. The paper releases a recipe and ablation analyses, including routing, stability and normalization choices.
Limits and trade-offs
Openness supports study but eliminates neither total memory nor expert communication costs.
- Original demonstration
- OLMoE, Allen Institute for AI.
Papers and sources
Direct link ↗Sigmoid attention: independent weights
An alternative rule turns reading scores into contributions.
The problem
The usual rule couples the weights of all positions being read.
The idea
A function transforms each score separately, with settings to stabilise learning.
An example to picture the idea
Fictional example: adjust the contributions of two past positions independently.
What this adds
This alternative can achieve comparable results in some experiments.
The limit to remember
Comparing every position remains expensive, and these results do not establish general superiority.
Go deeper: mechanism, diagram and origins
How it works
A sigmoid transforms each query-key score independently instead of normalizing a full row with softmax. The paper studies initial norms and proposes the FlashSigmoid kernel.
Limits and trade-offs
Dense computation remains quadratic; this work revisits an earlier idea rather than inventing it.
- Original demonstration
- Ramapuram et al., 2024.
Papers and sources
Direct link ↗nGPT: learning on a hypersphere
Keeping vectors at a fixed length changes how learning works.
The problem
Very different numerical magnitudes can complicate updates.
The idea
Representations and some learned values are normalised. Learning mainly changes their directions, with suitable scaling factors.
An example to picture the idea
Fictional example: change an arrow's direction rather than its length.
What this adds
Some experiments show more efficient learning.
The limit to remember
The arrow represents a numerical vector, and the recipe requires specific settings.
Go deeper: mechanism, diagram and origins
How it works
nGPT normalizes relevant vectors and frames updates as motion on a hypersphere. Learned factors control layer contribution magnitudes.
Limits and trade-offs
Reported gains depend on the experiments; this geometry needs a dedicated recipe.
- Original demonstration
- nGPT, Loshchilov et al., NVIDIA; arXiv 2410.01131.
Papers and sources
Direct link ↗BLT: bytes grouped into patches
Variable byte groups replace a fixed text segmentation.
The problem
A vocabulary of predefined pieces imposes a segmentation that can be unsuitable.
The idea
A component groups bytes, the units encoding text. Main computation processes these groups, then another component reconstructs bytes.
An example to picture the idea
Fictional example: create small groups in an unpredictable passage.
What this adds
Segmentation can adapt to content.
The limit to remember
Grouping and local components add work and complexity.
Go deeper: mechanism, diagram and origins
How it works
A local encoder groups bytes into variable-size patches according to predictability. A global Transformer operates on patches and a local decoder predicts bytes.
Limits and trade-offs
Local components and segmentation add complexity; gains depend on content.
- Original demonstration
- Byte Latent Transformer, Pagnoni et al., Meta.
Papers and sources
Direct link ↗DeepSeek-V3: integrating efficiency and scale
A recipe combines experts, condensed memory and more compact numerical computation.
The problem
A large model combines training, storage and communication costs.
The idea
The recipe activates selected experts, condenses the past and uses lower-precision numbers with suitable precautions.
An example to picture the idea
Fictional example: better distribute work between lightly and heavily used experts.
What this adds
These choices jointly target greater efficiency.
The limit to remember
The report retains a per-sequence balancing term despite simplifying other constraints.
Go deeper: mechanism, diagram and origins
How it works
The model balances routed experts using load-adjusted biases, uses a latent cache and learns future tokens. FP8 training includes precision and communication safeguards.
Limits and trade-offs
The model retains a small sequence-wise balancing loss; auxiliary-loss-free does not mean no such loss anywhere.
- Original demonstration
- DeepSeek-V3; recipe integrating techniques sometimes published earlier.
Papers and sources
Direct link ↗Gated DeltaNet: forgetting and correcting
A condensed memory can forget globally and correct particular associations.
The problem
Keeping old information indefinitely can interfere with updates.
The idea
A gate controls forgetting, while a targeted correction changes what the memory predicts incorrectly.
An example to picture the idea
Fictional example: reduce older traces while correcting an association.
What this adds
Memory combines two complementary ways to update itself.
The limit to remember
It remains compressed, and benefits depend on the tasks and budgets compared.
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
A gate controls state forgetting while the delta rule corrects particular associations. A chunkwise algorithm makes training parallel and GPU-friendly.
Limits and trade-offs
Memory remains compressed; comparisons depend on selected tasks and budgets.
- Original demonstration
- Gated DeltaNet, Yang, Kautz and Hatamizadeh.
Papers and sources
Direct link ↗Memory Layers: large sparse memory
Large learned tables add capacity without consulting everything.
The problem
Using every value in a large network is expensive.
The idea
A lookup selects a few entries from a large table of associations learned during training.
An example to picture the idea
Fictional example: use only the entries matching an internal query.
What this adds
Stored capacity can grow without as much active computation.
The limit to remember
Storage and access remain costly: this is not retrieval from external documents.
Go deeper: mechanism, diagram and origins
How it works
A token-derived query retrieves entries from a large key-value memory. Only selected values contribute to the layer, using efficient lookup mechanisms.
Limits and trade-offs
Storage and memory access remain costly; this parametric memory is not a RAG corpus.
- Original demonstration
- Memory Layers at Scale, Berges et al., Meta; extends product-key memories.
Papers and sources
Direct link ↗Titans: memory that learns while reading
Add compact memory that changes as the model reads.
The problem
Revisiting an entire long text requires substantial computation and memory.
The idea
Titans keeps a compact representation of the text and updates its values while reading. Another mechanism directly consults nearby passages.
An example to picture the idea
Fictional example: a name introduced early may influence an answer much later.
What this adds
Some earlier information remains usable without revisiting everything at each step.
The limit to remember
This representation can forget and guarantees no memory between conversations.
Go deeper: mechanism, diagram and origins
How it works
A module learns key-value associations as context arrives; gradients, momentum, and forgetting govern updates during inference.
Limits and trade-offs
This memory is compressed and may forget; it guarantees neither exact recall nor persistence between conversations.
- Original demonstration
- Experimental Titans family; arXiv v1 dated December 31, 2024 despite the 2501 identifier.
Papers and sources
Direct link ↗2025
MiniMax-01: large-scale linear attention
Combine different ways of consulting text to handle long documents.
The problem
Consulting every earlier passage becomes expensive as text grows longer.
The idea
Some calculation stages use a compact representation of earlier text; others consult passages directly. Only selected specialized parts of the network process each text piece.
An example to picture the idea
Fictional example: processing a dossier before answering a question about its contents.
What this adds
This combination aims to make long texts less expensive to process.
The limit to remember
Accepting a long input does not guarantee retrieving every detail.
Go deeper: mechanism, diagram and origins
How it works
Lightning Attention computes linear attention, combined with softmax layers and a MoE; the report describes training on one million tokens.
Limits and trade-offs
Advertised context length does not guarantee perfect recall; speed depends on kernels.
- Original demonstration
- MiniMax-Text-01 and MiniMax-VL-01; this report did not originate linear attention.
Papers and sources
Direct link ↗DeepSeek-R1: learning reasoning from rewards
Train a model to produce solutions using checkable outcomes.
The problem
Copying written answers does not always suffice to solve new problems.
The idea
The model generates attempts; checking assigns rewards that guide learning. R1 also combines prepared examples with several training stages.
An example to picture the idea
Fictional example: automatically checking whether a program passes the requested exercises.
What this adds
This recipe can improve problem solving where outcomes can be checked.
The limit to remember
Written steps guarantee neither a correct answer nor a faithful account of internal computation.
Go deeper: mechanism, diagram and origins
How it works
R1-Zero applies RL without preliminary SFT; R1 adds cold-start data, several RL stages, and distillation into smaller models.
Limits and trade-offs
Verifiable rewards suit objectively assessable tasks; longer answers consume compute.
- Original demonstration
- DeepSeek-R1-Zero then R1; GRPO originates in DeepSeekMath, in 2024.
Papers and sources
Direct link ↗s1: controlling thinking budgets
Control how much text the model generates before its final answer.
The problem
Some questions require more solution steps than others.
The idea
After training on examples, generation is stopped or extended before the final answer. Adding “Wait” encourages further written steps.
An example to picture the idea
Fictional example: extending a solution to attempt an additional check.
What this adds
The system can adjust the computation time spent on a question.
The limit to remember
Longer text costs more and guarantees neither a better answer nor a faithful account of reasoning.
Go deeper: mechanism, diagram and origins
How it works
After SFT on one thousand questions, budget forcing stops reasoning or extends it with “Wait” to encourage another verification step.
Limits and trade-offs
More tokens increase latency and do not guarantee a better answer to every question.
- Original demonstration
- s1-32B, adapted from Qwen2.5-32B-Instruct; not the invention of test-time compute.
Papers and sources
Direct link ↗SSMax: adapting attention concentration to length
Adjust how attention is distributed as the text grows longer.
The problem
Among many passages, useful information may receive too little attention.
The idea
The model calculates scores for passages it might consult. The method adjusts their scale according to text length, separately for each reading pathway.
An example to picture the idea
Fictional example: a relevant name needs to stand out among many sentences.
What this adds
This adjustment can prevent overly diffuse attention in the models studied.
The limit to remember
Experiments on small models do not establish a universal benefit.
Go deeper: mechanism, diagram and origins
How it works
Scalable-Softmax multiplies scores by a factor proportional to the logarithm of position count, with a learned scale per head.
Limits and trade-offs
A small-model study; results do not establish universal improvements at large scale.
- Original demonstration
- Experimental 162M-parameter Transformers, Ken M. Nakanishi.
Papers and sources
Direct link ↗LLaDA: generating text by unmasking
Build an answer by filling in masked text pieces.
The problem
Producing text strictly from left to right limits opportunities for parallel computation.
The idea
The model learns to recover hidden pieces of text. To generate an answer, it progressively fills masked positions over several passes.
An example to picture the idea
Fictional example: completing “The [mask] sleeps” and then refining the choices.
What this adds
Multiple positions can be processed together instead of each waiting its turn.
The limit to remember
Repeated passes cost computation: parallel work does not automatically mean a faster answer.
Go deeper: mechanism, diagram and origins
Noisy or masked sequenceParallel predictionsRefine over multiple steps
How it works
A Transformer learns to reconstruct masked tokens; successive unmasking steps turn corrupted text into a response, following pretraining and SFT.
Limits and trade-offs
Multiple passes are needed; potential parallelism and actual acceleration must be distinguished.
- Original demonstration
- LLaDA 8B; a demonstration at this scale, following earlier text diffusion work.
Papers and sources
Direct link ↗Muon: orthogonalizing updates
Improve how learned values change during training.
The problem
Expensive training can waste computation when its updates are poorly adjusted.
The idea
Muon transforms the tables of corrections applied to learned values. The Moonlight work also adjusts their scale and limits growth of those values to train large models.
An example to picture the idea
Fictional example: comparing two training recipes with the same computation budget.
What this adds
The method aims to obtain better learning within a given budget.
The limit to remember
Its benefit depends on the model and the training conditions.
Go deeper: mechanism, diagram and origins
Loss gradientTransform the updateAdjust parameters
How it works
Update orthogonalization is combined with weight decay and update-scale calibration; the authors train Moonlight.
Limits and trade-offs
The gain over AdamW is experimental and depends on the training regime.
- Original demonstration
- Moonlight 16B/3B active; this paper scales Muon rather than inventing it.
Papers and sources
Direct link ↗EAGLE-3: drafting from multiple layers
Let a small model propose a continuation, then check it.
The problem
Having the large model calculate every new text piece is slow.
The idea
A small model uses intermediate results from several stages of the large model to propose a continuation. The large model then checks these proposals.
An example to picture the idea
Fictional example: proposing “the door is open” and accepting only the validated beginning.
What this adds
Accepting several proposals together can reduce generation time.
The limit to remember
Proposals that are frequently rejected or expensive to produce can erase the advantage.
Go deeper: mechanism, diagram and origins
Draft multiple tokensVerify with the target modelAccept a prefix
How it works
The drafter fuses representations from multiple layers and directly predicts tokens; training better simulates its generation conditions.
Limits and trade-offs
Acceleration depends on acceptance rate, batch size, and drafter cost.
- Original demonstration
- EAGLE-3 is a decoding method evaluated on several chat and reasoning models.
Papers and sources
Direct link ↗Gemma 3: alternating local and global attention
Mostly consult nearby text, with some stages looking further back.
The problem
Consulting all text at every calculation stage requires substantial memory.
The idea
Most stages consult only a window of nearby text. Other stages consult the full context; some variants also process images.
An example to picture the idea
Fictional example: using neighboring sentences while connecting a name to the beginning of a document.
What this adds
Most stages retain less information for generation.
The limit to remember
Global stages remain expensive, and distant information may still be used poorly.
Go deeper: mechanism, diagram and origins
How it works
Five local-window layers alternate with one global layer; multimodal models add a vision encoder and extended context.
Limits and trade-offs
Global layers remain expensive; a large context does not mean perfect access to every detail.
- Original demonstration
- Gemma 3; arXiv report date, distinct from launch.
Papers and sources
Direct link ↗Nemotron-H: reserving few layers for global attention
Combine a compact representation of the past with direct text consultation.
The problem
Retaining all intermediate information about text can take considerable memory.
The idea
Most stages update a compact representation of preceding text. A few stages retain direct attention over the text.
An example to picture the idea
Fictional example: tracking the overall subject while consulting earlier passages at selected stages.
What this adds
This combination aims to reduce generation cost while retaining some direct consultation.
The limit to remember
The compact representation loses details; memory used by attention stages still grows.
Go deeper: mechanism, diagram and origins
How it works
Most attention layers are replaced by Mamba layers; pruning and distillation also compress selected models.
Limits and trade-offs
Mamba layers retain fixed state, but remaining attention layers keep their growing cache.
- Original demonstration
- NVIDIA Nemotron-H family; arXiv v1 on April 4, 2025.
Papers and sources
Direct link ↗Llama 4: iRoPE, MoE, and multimodal fusion
Combine text, images, and computation distributed across selected subnetworks.
The problem
Processing long content containing text and images can become very expensive.
The idea
Text pieces and image representations pass through a shared network. The model varies how positions are represented and selects specialized network parts for computation.
An example to picture the idea
Fictional example: answering a question about a photo accompanied by a long commentary.
What this adds
This combination aims to handle varied content with selectively activated computation.
The limit to remember
All network parts require storage; accepting extensive text does not guarantee recalling it.
Go deeper: mechanism, diagram and origins
One tokenRouter → selected expertsCombine their outputs
How it works
iRoPE interleaves RoPE layers with layers without positional encoding, with temperature adjustment; visual and text tokens share the backbone.
Limits and trade-offs
An advertised multimillion-token context does not guarantee recall; all experts still need storage.
- Original demonstration
- Llama 4 Scout and Maverick; official announcement, not an arXiv article.
Papers and sources
Direct link ↗Gated attention: controlling each head’s output
Control how much information each reading pathway passes onward.
The problem
Information that was consulted is not necessarily useful for the next calculation stage.
The idea
After consulting the text, each reading pathway receives a setting calculated from the input. This setting attenuates or passes through its results.
An example to picture the idea
Fictional example: reducing the contribution of a pathway that is less useful for the current sentence.
What this adds
The model can filter attention results more finely.
The limit to remember
Observed benefits depend on the tested models and are not guaranteed everywhere.
Go deeper: mechanism, diagram and origins
How it works
A sigmoid function modulates head outputs after attention; the study observes greater sparsity and less artificial concentration on initial tokens.
Limits and trade-offs
Results concern the tested architectures and do not prove universal improvement.
- Original demonstration
- Experimental Qwen-team models; not the origin of all gating mechanisms.
Papers and sources
Direct link ↗Qwen3: direct responses or thinking in one model
Choose between a direct answer and generation with more written steps.
The problem
A long written solution may be unnecessary for a simple question.
The idea
Training prepares the same model either to answer directly or to produce steps before answering. The selected mode changes how generation proceeds.
An example to picture the idea
Fictional example: responding directly to a greeting but writing out a mathematical solution.
What this adds
One model offers different balances between response time and problem-solving work.
The limit to remember
Extra steps cost computation without guaranteeing their faithfulness or the final answer’s correctness.
Go deeper: mechanism, diagram and origins
How it works
Post-training unifies immediate responses and reasoning traces; GQA, RoPE, SwiGLU, and QK-Norm form an already well-tested architecture.
Limits and trade-offs
A larger thinking budget increases cost; the report does not describe MTP for initial Qwen3.
- Original demonstration
- Qwen3 family, dense and MoE; report date.
Papers and sources
Direct link ↗Gemma 3n: per-layer embeddings and nested models
Adapt a model to the memory constraints of small devices.
The problem
A phone has less memory and computation available than a server.
The idea
Some representations can be prepared and stored outside main memory. A nested network also allows a smaller version of the model to run.
An example to picture the idea
Fictional example: selecting a smaller version for a phone application.
What this adds
The method can reduce working memory and the amount of computation required.
The limit to remember
Stored values, values used for computation, and actual memory consumption remain different quantities.
Go deeper: mechanism, diagram and origins
How it works
Per-layer embeddings can be prepared and stored outside main memory; MatFormer allows nested submodels to run with fewer parameters.
Limits and trade-offs
Total parameters, active parameters, and effective memory differ; savings depend on implementation.
- Original demonstration
- Gemma 3n; preview announcement date, with no identified arXiv report.
Papers and sources
Direct link ↗MoNE: replacing redundant experts
Replace some specialized model parts with smaller approximations.
The problem
Storing every part of a large model may exceed available memory.
The idea
The method measures the use and outputs of specialized network parts, called experts. It replaces selected parts with small modules that approximate their contribution.
An example to picture the idea
Fictional example: replacing a rarely used part after testing on a collection of texts.
What this adds
The transformed model can require less storage.
The limit to remember
Approximation can degrade its answers and depends on the examples used to select replacements.
Go deeper: mechanism, diagram and origins
How it works
MoNE measures access frequency and output variance to replace selected experts with lightweight modules approximating their contribution.
Limits and trade-offs
Compression changes the model and may reduce quality; calibration data matters.
- Original demonstration
- MoNE, evaluated notably on Qwen2-57B-A14B; a 2025 paper, not a 2026 invention.
Papers and sources
Direct link ↗Comparing hybrids: language quality and recall diverge
Compare combinations of mechanisms before choosing an architecture.
The problem
A mechanism that performs well alone may be less effective when combined with another.
The idea
The study compares different proportions of direct consultation and compact-memory stages. It trains multiple models to measure tradeoffs instead of assuming a winning combination.
An example to picture the idea
Fictional example: comparing more compact stages against more direct consultations.
What this adds
These measurements help select a combination for a desired cost and outcome.
The limit to remember
They do not establish an ideal proportion that applies to every model.
Go deeper: mechanism, diagram and origins
How it works
The authors train 72 models with different mechanisms and ratios; the best standalone component does not necessarily win when hybridized.
Limits and trade-offs
Tradeoffs observed at the studied sizes do not establish a universal optimal ratio.
- Original demonstration
- Wang and collaborators; experimental 340M and 1.3B families, not the invention of hybrids.
Papers and sources
Direct link ↗LFM2: choosing architecture through hardware measurements
Choose an architecture by measuring it on the intended devices.
The problem
A model that runs quickly on a server may run slowly on a phone.
The idea
The search measures delay and memory on devices. It mainly combines computations over short text neighborhoods with a few broader consultation stages.
An example to picture the idea
Fictional example: measuring several variants on the phone that will run the application.
What this adds
The architecture can better match actual hardware constraints.
The limit to remember
A result measured on one device does not automatically transfer to other hardware.
Go deeper: mechanism, diagram and origins
How it works
Architecture search measures on-device memory and latency; it combines mostly short double-gated convolutions with a few GQA layers.
Limits and trade-offs
An acceleration measured on mobile CPUs does not automatically transfer to every GPU or workload.
- Original demonstration
- Liquid AI LFM2; July 10 announcement, arXiv report on November 28, 2025.
Papers and sources
Direct link ↗Kimi K2: MuonClip and agentic training
Stabilize a very large model and train it to use tools.
The problem
A very large network is difficult to train and does not automatically chain tool actions.
The idea
A correction limits excessive attention values during learning. Trials in simulated environments then train the model to produce sequences of tool actions.
An example to picture the idea
Fictional example: finding a file, running a calculation, then using its result.
What this adds
The recipe combines more stable training with learning successive actions.
The limit to remember
All network parts remain large to store even when only selected parts compute.
Go deeper: mechanism, diagram and origins
Loss gradientTransform the updateAdjust parameters
How it works
MuonClip extends Muon with attention-logit control; synthetic environments and RL provide tool-use trajectories.
Limits and trade-offs
A trillion total parameters remain expensive to store even when only a small fraction is active.
- Original demonstration
- Kimi K2; MuonClip is presented in this report.
Papers and sources
Direct link ↗GLM-4.5: deep MoE, GQA, and multiple-token prediction
Combine shared memory, selected computation, and text proposals for checking.
The problem
A large model needs to limit the cost of memory and generation.
The idea
Reading pathways share some stored information. Selected network parts perform computation; an additional stage also proposes text pieces that the model can check together.
An example to picture the idea
Fictional example: proposing a short continuation rather than checking each piece separately.
What this adds
This combination aims to reduce the cost of serving the model.
The limit to remember
Results also depend on training and data, rather than on these mechanisms alone.
Go deeper: mechanism, diagram and origins
Queries QKeys K + values VCache to read
How it works
The report describes GQA with partial RoPE, a balanced-routing MoE, and an MTP layer proposing speculative tokens.
Limits and trade-offs
Quality and speed also depend on data and post-training, not architecture alone.
- Original demonstration
- GLM-4.5 and GLM-4.5-Air; they do not use MLA in this report.
Papers and sources
Direct link ↗gpt-oss: MoE quantized into MXFP4
Store some model values with reduced numerical precision.
The problem
The learned values of a large model can take too much memory.
The idea
Specialized network parts use a compact number format that is accounted for during adaptation. Only selected parts participate in computation for each text piece.
An example to picture the idea
Fictional example: loading a model on a machine with limited memory.
What this adds
Reducing precision can decrease the space occupied by stored values.
The limit to remember
Memory for the text being processed and suitable hardware are still required.
Go deeper: mechanism, diagram and origins
One tokenRouter → selected expertsCombine their outputs
How it works
Expert weights are quantized into MXFP4 during post-training; each token uses four experts out of 128 or 32 depending on the model.
Limits and trade-offs
Weight memory excludes the KV cache; performance depends on compatible hardware.
- Original demonstration
- gpt-oss-120b and 20b; arXiv date, after the August 5 launch.
Papers and sources
Direct link ↗Qwen3-Next: three recurrent layers per global layer
Alternate compact memory with direct consultation to reduce long-text costs.
The problem
Keeping all required information at every reading stage becomes expensive.
The idea
Several stages update a compact representation of preceding text, then one stage consults the text directly. The network also selects which parts compute and proposes continuations for checking.
An example to picture the idea
Fictional example: answering about a long document without retaining everything at every stage.
What this adds
This combination aims for less expensive generation over long contexts.
The limit to remember
Compact representations lose detail; few values used for computation does not mean little storage.
Go deeper: mechanism, diagram and origins
How it works
The model alternates three Gated DeltaNet layers with one Gated Attention layer, using a highly sparse MoE and MTP for speculative decoding.
Limits and trade-offs
The recurrent state compresses history; throughput gains depend strongly on implementation.
- Original demonstration
- Qwen3-Next-80B-A3B; announcement, not arXiv date; 80B total and 3B active.
Papers and sources
Direct link ↗Kimi Linear: recurrent memory with fine-grained gates
Control more finely what a compact memory retains from text.
The problem
Compressing a long history can erase information that is still useful.
The idea
Most stages finely control adding and erasing information in a compact representation. Other stages consult text directly using compressed memory.
An example to picture the idea
Fictional example: retaining still-useful information while weakening an association that has become less relevant.
What this adds
The mechanism aims for a better balance between memory cost and using context.
The limit to remember
Compression can lose details, and global stages retain memory that grows with the text.
Go deeper: mechanism, diagram and origins
Token + previous stateUpdate the stateOutput + new state
How it works
KDA extends Gated DeltaNet with finer gates; Kimi Linear alternates three KDA layers and one global MLA layer.
Limits and trade-offs
Global layers retain a growing cache; detailed retrieval remains difficult with purely recurrent memory.
- Original demonstration
- Kimi Linear 48B/3B active; first-report date.
Papers and sources
Direct link ↗DeepSeek Sparse Attention: selecting useful memories
Select passages to consult before performing the main calculation.
The problem
Comparing each new text piece with all earlier pieces becomes expensive.
The idea
A first lightweight calculation selects positions that may be useful. The main attention calculation then consults this selection instead of treating every passage the same way.
An example to picture the idea
Fictional example: retaining passages about a date before answering a question about it.
What this adds
A shorter selection can reduce computation for long texts.
The limit to remember
Selection can omit useful information and has its own computational cost.
Go deeper: mechanism, diagram and origins
How it works
A lightweight indexer selects relevant tokens before main attention; DSA complements DeepSeek’s latent mechanism to reduce long-context cost.
Limits and trade-offs
Selection may discard information; the indexer itself still incurs cost.
- Original demonstration
- DeepSeek-V3.2-Exp precedes the complete V3.2 report; displayed date: arXiv.
Papers and sources
Direct link ↗mHC: constraining connections between layers
Add pathways between calculation stages while keeping transmission stable.
The problem
Multiplying connections can disrupt information transmitted through a deep network.
The idea
Multiple pathways carry results between stages. The method constrains how these pathways are mixed to limit amplification and preserve more stable transmission.
An example to picture the idea
Fictional example: comparing a multiple-pathway network with and without the constraint during training.
What this adds
The network can explore more pathways while limiting some instabilities.
The limit to remember
These connections add data transfers and require implementation optimizations.
Go deeper: mechanism, diagram and origins
How it works
Multiple Hyper-Connections are projected onto a constrained space to restore identity-like transmission properties and limit instability.
Limits and trade-offs
Additional connections increase memory traffic; efficiency requires dedicated optimization.
- Original demonstration
- Experimental DeepSeek models; this paper extends the 2024 Hyper-Connections work.
Papers and sources
Direct link ↗2026
LongCat-Flash-Lite: expanding n-gram memory
Add large learned tables representing groups of text pieces.
The problem
Enlarging every network calculation substantially increases the cost of using a model.
The idea
Groups of text pieces retrieve representations from large learned tables. The model consults selected entries instead of computing through an entire additional network.
An example to picture the idea
Fictional example: consulting a representation associated with a frequent group of words.
What this adds
The model can add learned values without using every value at every step.
The limit to remember
The tables require storage and must remain fast to access.
Go deeper: mechanism, diagram and origins
How it works
Large embedding tables represent groups of tokens; sparse lookups add parameters without activating an entire additional network.
Limits and trade-offs
Tables need efficient storage and lookup; gains depend on budget and hardware.
- Original demonstration
- LongCat-Flash-Lite, 68.5B parameters and roughly 3B active; arXiv date.
Papers and sources
Direct link ↗DFlash: drafting a block in parallel
Produce several proposals in parallel before having them checked.
The problem
Even a small drafting model can be slow if it writes every piece sequentially.
The idea
The small model uses results from the large model to propose a block of text in parallel. The large model then checks the proposed pieces before accepting them.
An example to picture the idea
Fictional example: proposing a complete short continuation rather than its pieces one at a time.
What this adds
Less sequential drafting can reduce generation delay.
The limit to remember
The benefit depends on accepted proposals, hardware, and workload.
Go deeper: mechanism, diagram and origins
How it works
A diffusion drafter conditioned on target-model representations proposes a block in one pass; the target model then verifies the tokens.
Limits and trade-offs
Gains depend on acceptance, batch size, and engine; verification remains essential.
- Original demonstration
- DFlash, notably demonstrated with Qwen3; first-paper date.
Papers and sources
Direct link ↗Qwen3.5: hybrid architecture becomes multimodal
Extend a computation-efficient architecture to images and text.
The problem
Combining long texts with images can require substantial resources.
The idea
The model learns from visual and textual content. It alternates compact representations with direct context consultation, selects certain network parts, and can propose multiple pieces for checking.
An example to picture the idea
Fictional example: answering about a screenshot and its long description.
What this adds
The combination aims to process different kinds of content with controlled computation.
The limit to remember
All learned values still require storage even when few participate in each calculation.
Go deeper: mechanism, diagram and origins
How it works
Gated DeltaNet and Gated Attention combine with a sparse MoE, multiple-token prediction, and training incorporating visual modalities.
Limits and trade-offs
A low active-parameter count does not mean low total storage; advertised throughput depends on comparators.
- Original demonstration
- Qwen3.5-397B-A17B; date displayed by the official announcement, not an arXiv submission.
Papers and sources
Direct link ↗DeepSeek-V4: compressed sparse attention
Compress and select past text to make very long contexts less expensive.
The problem
Consulting a very long text can require too much memory and computation.
The idea
Several mechanisms compress retained information and select what to consult. The model also combines stabilized connections between stages with a different update recipe during training.
An example to picture the idea
Fictional example: answering about a large dossier using reduced representations.
What this adds
This combination aims to reduce the cost of long contexts.
The limit to remember
Compression can lose details; reported savings depend on the measured comparison.
Go deeper: mechanism, diagram and origins
How it works
The report describes CSA and HCA for context compression and selection, mHC for residual connections, and Muon for training.
Limits and trade-offs
Advertised savings compare against V3.2 in a specific setup; compression may lose detail.
- Original demonstration
- DeepSeek-V4-Pro and Flash; official announcement date.
Papers and sources
Direct link ↗R-SWA: a local window with visual reference
Keep the document available while limiting memory for already transcribed text.
The problem
Transcribing a long document can accumulate too much information about text already produced.
The idea
The model keeps access to representations of the document image and a window of recent text. It does not retain the entire transcription in its generation memory.
An example to picture the idea
Fictional example: continuing a transcription while still consulting the original image.
What this adds
Transcription-related memory stops growing with the full transcription length.
The limit to remember
The visual reference and recent-text window still occupy memory.
Go deeper: mechanism, diagram and origins
How it works
Unlimited OCR retains access to visual reference tokens and a recent-text window; this attention replaces DeepSeek OCR decoder layers.
Limits and trade-offs
Memory stays constant with transcription length but still depends on reference size and window.
- Original demonstration
- Baidu Unlimited OCR; R means Reference, not Recurrent.
Papers and sources
Direct link ↗Gemma 4: one multimodal family, multiple architectures
Offer different architectures for different content and devices.
The problem
One model does not suit every device or every type of input.
The idea
The family provides different sizes and different ways to distribute computation. Depending on the variant, it can receive images or sound and produce steps before answering.
An example to picture the idea
Fictional example: choosing a variant suitable for audio inputs on a particular device.
What this adds
Several tradeoffs become available within one family.
The limit to remember
Each variant has its own capabilities; none automatically combines all the family’s features.
Go deeper: mechanism, diagram and origins
How it works
The report presents dense and MoE models, image and audio inputs, a thinking mode, and a 12B variant without separate encoders.
Limits and trade-offs
Features differ by size; PLE, MoE, and every modality cannot be attributed to every variant.
- Original demonstration
- Gemma 4; arXiv date, following the initial April 2 launch.
Papers and sources
Direct link ↗DSpark: choosing how much draft to verify
Adapt draft checking to the proposals and the server workload.
The problem
Checking a long, unpromising draft can waste computation.
The idea
The system estimates which proposals are likely to be accepted. It adjusts the work sent to the large model while also accounting for server load.
An example to picture the idea
Fictional example: shortening the block to check when proposals are uncertain.
What this adds
The system seeks a better balance between individual waiting time and the number of answers served.
The limit to remember
Useful settings depend on workload, and proposals still require checking.
Go deeper: mechanism, diagram and origins
How it works
The framework combines semi-autoregressive generation with adaptive verification: proposal confidence and server load guide work submitted to the target model.
Limits and trade-offs
The latency-throughput tradeoff depends on actual load; a long draft is not always worthwhile.
- Original demonstration
- DeepSeek DSpark; a decoding framework, not a user-facing model architecture.
Papers and sources
Direct link ↗DiffusionGemma: converting a model to block generation
Experimentally adapt a model to refine blocks of text in parallel.
The problem
Strictly sequential generation limits opportunities to produce several pieces together.
The idea
Adaptation teaches the model to complete and correct text blocks. Multiple passes refine proposals; additional training stages seek a balance between speed and results.
An example to picture the idea
Fictional example: filling and then correcting positions in a short answer.
What this adds
The experimental model explores more parallel generation.
The limit to remember
Refinement costs computation, and measured speedups are not guaranteed on every device.
Go deeper: mechanism, diagram and origins
Noisy or masked sequenceParallel predictionsRefine over multiple steps
How it works
The model refines 256-token blocks in parallel; denoising SFT followed by RL and sampler distillation balances quality and speed.
Limits and trade-offs
Multiple refinements are needed; H100 measurements do not guarantee identical gains on every device.
- Original demonstration
- DiffusionGemma, derived from Gemma 4 MoE; arXiv date.
Papers and sources
Direct link ↗Ideas and their connections
Follow an idea into work that builds on, improves or combines it. Choose a family, then a connection to read its explanation and source. Alternatives are shown separately from lines of development.
Reducing attention cache memory
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1Transformer → MQAImprovement
MQA retains Transformer attention heads but makes them share a single set of keys and values.
Read the paper ↗2MQA → GQAImprovement
GQA generalizes MQA: several groups share keys and values, instead of one shared set for all heads.
Read the paper ↗3GQA ↔ MLAAlternative
MLA also reduces cache memory, but compresses keys and values into a small vector instead of sharing them in groups.
Read the paper ↗Accelerating attention computation
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1FlashAttention → FlashAttention-2Improvement
The second version retains tiled computation and distributes work more effectively across GPU units.
Read the paper ↗2FlashAttention-2 → FlashAttention-3Improvement
The third version adapts this exact attention to Hopper GPUs by overlapping transfers and computation.
Read the paper ↗Adapting with less memory
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1LoRA → QLoRACombination
QLoRA trains LoRA’s small matrices through a frozen base model quantized to 4 bits.
Read the paper ↗Learning from preferences and rewards
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
Learning rewards from human preferencesUse human comparisons to define what training should encourage.PPO: limiting reinforcement-learning updatesAvoid changing behavior too much within one series of updates.
InstructGPT: following human intentHuman examples and preferences make answers more useful.DPO: learning preferences directlyDirectly comparing two answers simplifies learning from preferences.GRPO: comparing a group of responsesComparing answers within a group simplifies some reward-based training.
1Learning rewards from human preferences → InstructGPTBuilds on
InstructGPT applies reward learning from human preferences to tune a language model.
Read the paper ↗2InstructGPT ↔ DPOAlternative
DPO learns directly from preferred and rejected responses, without training the separate reward model used in the InstructGPT recipe.
Read the paper ↗3PPO → GRPOImprovement
GRPO adapts PPO by comparing rewards for several answers to the same question, without a separate value critic.
Read the paper ↗Reading longer texts with RoPE
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1RoPE → Position interpolationBuilds on
Interpolation reduces position indices before applying RoPE, fitting a longer text into the learned range.
Read the paper ↗2Position interpolation → YaRNImprovement
YaRN refines interpolation by treating RoPE rotation frequencies differently and adjusting attention scaling.
Read the paper ↗Activating selected experts
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1MoE → Switch TransformerImprovement
Switch retains routing to sparse experts and simplifies it: a single expert is chosen for each token.
Read the paper ↗2MoE → MixtralBuilds on
Mixtral uses the sparse-expert principle: each token is sent to two of the eight experts in each MoE layer.
Read the paper ↗Recurrent memory and hybrid models
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1S4 → MambaImprovement
Mamba extends state-space models by making what is retained or forgotten depend on the input.
Read the paper ↗2Mamba → Mamba-2 / SSDImprovement
Mamba-2 reformulates Mamba’s selective mechanism through the duality between state spaces and structured attention.
Read the paper ↗3Mamba → JambaCombination
Jamba combines Mamba layers with Transformer attention layers and expert layers.
Read the paper ↗4Transformer → JambaCombination
Jamba retains attention layers within a model that also uses Mamba’s recurrent memory.
Read the paper ↗Comparing several reasoning paths
View entries in the timeline ↓→ Builds on, improves or combines↔ Alternative
1Chain-of-Thought → Self-consistencyImprovement
Self-consistency generates several Chain-of-Thought reasoning paths, then selects the most frequent answer.
Read the paper ↗Sources and dates
Dates refer to the first arXiv submission or the indicated official publication. A model may be available before its paper is published.
The original model is the one described in the publication. A method may be demonstrated on an experimental model and later popularized by other models.
Reported results and numerical gains depend on the model, hardware and measurement protocol.
Download the bilingual catalogue (JSON)