llm.curionode.com
◐ 中文
Part One · The Model
Vol. I What a Language Model Is What does it mean to predict the next token, and why is that enough to learn so much? Predicting the Next Token Models From Counting The Probability of a Text Perplexity and Bits Generating Text Why Counting Fails A Neural Language Model What Prediction Teaches The Life of a Model Drills: Language Models Vol. II Tokens What does a model actually read? Why Not Characters or Words Bytes and Unicode Byte-Pair Encoding Encoding With BPE The Unigram Tokeniser Choosing a Vocabulary Size The Cost of a Language When Tokens Get in the Way Special Tokens and Templates Drills: Tokens Vol. III Learning Machinery How does a network learn from a loss, at the scale a language model needs? Embeddings Softmax and Cross-Entropy Backpropagation From SGD to AdamW Initialisation Normalisation Residual Connections The MLP Memorising and Generalising Drills: Learning Machinery Vol. IV Attention How can each token use the tokens before it? The Bottleneck Queries, Keys and Values Scaled Dot-Product Attention The Causal Mask Many Heads Reading Attention Patterns What Attention Costs What Attention Weights Do Not Say Drills: Attention Vol. V The Transformer What is in a block, and why is it built that way? The Transformer Block The Residual Stream Pre-Norm and Post-Norm Decoder-Only, Encoder-Decoder Counting Parameters FLOPs per Token The Output Head A GPT in the Page Training It Drills: The Transformer Vol. VI Positions and Long Context How does a model know where a token is, and how far can it see? Attention Is Order-Blind Absolute Positions Rotary Position Embedding Linear Biases Beyond the Training Length Stretching the Context Windows and Sinks Using a Long Context What a Long Context Costs Drills: Positions Vol. VII Modern Architectures What has changed in the Transformer since 2017, and why? Gated MLPs Fewer Keys and Values Latent Attention Mixture of Experts Keeping Experts Busy Total and Active Parameters Small Changes That Stabilise Beyond Quadratic Attention Wide or Deep Drills: Architectures Part Two · Pretraining
Vol. VIII The Pretraining Run What happens when a model reads a trillion tokens? The Objective Packing Sequences Reading a Loss Curve What a Loss Value Means Abilities Over Training Emergence, Measured Memorisation Where Knowledge Lives What a Base Model Is Drills: Pretraining Vol. IX Data What goes into a model, and who decides? The Web as Corpus From HTML to Text Which Language Deduplication Quality Filters Safety and Privacy Filters Beyond the Web Data Mixtures Synthetic Data Contamination Data, Consent and Law Drills: Data Vol. X Scaling Laws How much better does a model get with more parameters, data and compute? Power Laws Model Size and Data Compute-Optimal Training Predicting From Small Runs Training Past Optimal When Data Runs Out From Loss to Skills Where Scaling Laws Break Drills: Scaling Vol. XI Optimisation at Scale How do you train a model with hundreds of billions of parameters without it diverging? Learning-Rate Schedules Batch Size AdamW at Scale Gradient Clipping Loss Spikes Mixed Precision Eight-Bit Training Transferring Hyperparameters New Optimisers Drills: Optimisation Vol. XII Mid-Training What happens between pretraining on the web and fine-tuning for chat? What Mid-Training Is Annealing on Good Data Upsampling Domains The Long-Context Phase Continued Pretraining Forgetting Learning From a Larger Model Averaging and Merging The Knowledge Cutoff Drills: Mid-Training Part Three · Infrastructure
Vol. XIII Hardware What does a training chip actually do, and what limits it? Accelerators The Memory Hierarchy The Roofline Arithmetic Intensity in a Transformer Interconnect Model FLOPs Utilisation Number Formats in Silicon Reading a Spec Sheet Drills: Hardware Vol. XIV Parallelism How is one model spread over thousands of chips? Where the Memory Goes Recomputing Activations Collective Communication Data Parallelism Sharding the States Splitting Every Layer Pipelines and Bubbles Splitting the Sequence Spreading the Experts Putting It Together Drills: Parallelism Vol. XV Running a Training Cluster What keeps a run of ten thousand GPUs going for months? Failures at Scale Checkpointing Stragglers Silent Data Corruption Feeding the Machine Watching a Run Cost and Energy Reading a Training Report Drills: Clusters Vol. XVI Kernels and Efficiency Where does the time inside a step go, and how is it won back? Profiling a Step Kernel Fusion The Online Softmax FlashAttention Compilers and Kernel Languages Low-Precision Matrix Multiply Predicting a Kernel’s Time Drills: Kernels Part Four · Post-Training
Vol. XVII Supervised Fine-Tuning How does a text predictor become an assistant? From Base to Chat Chat Templates Learning Only the Answers Instruction Data Quality Over Quantity LoRA Fine-Tuning a Quantised Model What Fine-Tuning Costs Distilling a Teacher Drills: Fine-Tuning Vol. XVIII Learning From Preferences How is a model taught what people prefer when no answer is simply correct? Comparisons, Not Labels The Bradley–Terry Model Reward Models RLHF With PPO Staying Close Over-Optimising a Reward Direct Preference Optimisation The DPO Family Feedback From Models Pleasing Instead of Helping Drills: Preferences Vol. XIX Reinforcement Learning for Reasoning How does a model learn to reason from rewards it can check? Policies, Rewards and Returns The Policy Gradient Rewards You Can Check PPO in Detail Group-Relative Advantages Keeping the Good Samples Thinking Out Loud Rewarding the Steps When Exploration Dies Spending Compute at Test Time Distilling Reasoning Drills: RL for Reasoning Vol. XX Safety Training How is a model trained to be helpful, honest and harmless, and how would we know? What Alignment Means Writing the Rules Down Refusal and Over-Refusal Training for Robustness Honesty and Calibration Hallucination Specification Gaming Oversight That Scales Evaluating Safety Drills: Safety Training Part Five · Evaluation and Inference
Vol. XXI Evaluation How do we know what a model can do? Benchmarks Error Bars One Prompt, Many Scores Pass@k Models as Judges Head-to-Head Saturation and Leakage Measuring Real Tasks Designing an Evaluation Drills: Evaluation Vol. XXII Decoding How is a probability distribution turned into text? Greedy and Beam Search Temperature Truncation Repetition and Degeneration Structured Output Calibration Stopping Drills: Decoding Vol. XXIII Serving How does one model answer millions of requests? Prefill and Decode The KV Cache Running Out of Cache Continuous Batching Latency or Throughput Reusing Prefixes Splitting Prefill From Decode The Cost of a Token Drills: Serving Vol. XXIV Making Inference Cheap How is the same answer produced with less memory, time and energy? Quantising Weights Smarter Quantisation Quantising the Cache Speculative Decoding Drafting Without a Draft Model Smaller Students Pruning and Sparsity Serving a Mixture of Experts Models on Devices Drills: Cheap Inference Part Six · Agents and Environments
Vol. XXV Tools and Agents What changes when a model acts instead of only answering? Calling Tools The Agent Loop Retrieval Managing the Context Errors Compound Using a Computer Many Agents Agents and Risk Drills: Agents Vol. XXVI Training Agents How is a model trained to complete long, multi-step tasks? Trajectories as Data Learning From Demonstrations Multi-Turn RL Which Step Mattered Training on Actions, Not Observations Learning From Its Own Attempts Long Horizons RL Infrastructure What Goes Wrong Drills: Training Agents Vol. XXVII RL Environments What makes a good place for a model to learn? What an Environment Is Graders and Verifiers Code Environments Generating Tasks Curricula Reward Hacking Hardening a Grader Breadth of Environments Training and Test Environments Simulated Users and Tools Drills: Environments Part Seven · Inside the Model and the Frontier
Vol. XXVIII Interpretability What is going on inside? The Logit Lens Linear Probes Activation Patching Induction Heads Superposition Sparse Autoencoders Steering Do Explanations Explain Drills: Interpretability Vol. XXIX Multimodal Models How do language models see, hear and speak? Images as Tokens Training to See Audio and Speech One Model for Everything Generating Images and Sound What Pictures Cost Drills: Multimodal Vol. XXX The Research Frontier What is the field working on now, and what is still unknown? Reasoning Models Scaling Reinforcement Learning Memory Beyond the Context After the Transformer Learning From Itself Running Out of Data Doing More With Less World Models and Embodiment Open Problems in Safety What We Do Not Know Drills: The Frontier Part Eight · History and Projects
Vol. XXXI History and Projects How did the field get here, and can the reader build each stage? From Shannon to n-grams Neural Language Models Attention and the Transformer The Age of Scale Assistants and Reasoning Project: A Tokeniser Project: Pretrain a Small Model Project: Plan a Training Run Project: Align a Small Model Project: RL With a Verifier Project: Build an Environment Project: Serve a Model The Misreadings Met Again The Journey Map
Settings: send automatic error reports when a plate breaks
A Curionode course. Nothing on this site is fetched from a third party; your progress stays in your browser. Every claim about a production model is dated and cited.