---
title: "Research Papers reading paths"
description: "Ordered research reading paths with source notes and citation-manager exports."
canonical: "https://papers.highsignal.app/paths"
last_updated: "2026-08-27"
---

# Research Papers reading paths

Ordered, copyright-safe research paths built from bibliographic metadata,
source links, and original notes. The catalog does not host PDFs or copy
abstracts.

## Sutskever/Carmack Core

The public mirror of the famous deep-learning list, ordered as preserved online.

- Audience: Readers who want the pre-LLM deep-learning spine behind modern systems.
- Difficulty: advanced
- Estimate: 27 readings, 4-6 focused weeks
- Outcome: You can trace a compact path from sequence models, convolution, attention, scaling, compression, and algorithmic information theory.
- Provenance: Public mirrors describe this as a list Ilya Sutskever gave John Carmack; the stable mirrors list 27 items and mark the attribution as not fully confirmed.
- Trust note: Treat this as an influential public mirror, not a verified official Ilya publication. The shorthand says about 30 papers, but the durable public list currently preserves 27 readings.

1. [The Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/): Executable transformer walkthrough
   - A code-first explanation of the transformer that makes the architecture concrete before heavier papers.
   - Read for: Use it to connect equations, modules, and implementation details.
2. [The First Law of Complexodynamics](https://scottaaronson.blog/?p=762): Complexity intuition
   - Builds intuition for why complexity can rise and fall rather than only accumulate.
   - Read for: Read as a conceptual lens for later compression and representation readings.
3. [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/): Sequence modeling intuition
   - Shows how simple character-level recurrent models learn surprising structure.
   - Read for: Pay attention to the qualitative behavior of learned sequence models.
4. [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/): Gated memory
   - Explains the gating mechanism that made recurrent memory easier to reason about.
   - Read for: Understand the cell state and gates before reading recurrent regularization work.
5. [Recurrent Neural Network Regularization](https://arxiv.org/abs/1409.2329): Training recurrent nets
   - Studies dropout placement for recurrent networks without destroying temporal memory.
   - Read for: Notice how a small regularization detail changes sequence-model training behavior.
6. [Keeping Neural Networks Simple by Minimizing the Description Length of the Weights](https://www.cs.toronto.edu/~hinton/absps/colt93.pdf): Simplicity prior
   - Connects neural-network regularization to a compression-style preference for simpler explanations.
   - Read for: Use it as the bridge from practical regularization to MDL-style thinking.
7. [Pointer Networks](https://arxiv.org/abs/1506.03134): Attention as addressing
   - Uses attention to choose positions in the input sequence as outputs.
   - Read for: Read for attention as selection rather than only summarization.
8. [ImageNet Classification with Deep Convolutional Neural Networks](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks): Deep vision breakthrough
   - Marks the practical arrival of large convolutional networks on ImageNet-scale vision.
   - Read for: Separate the architecture from the compute, data, and training choices that made it work.
9. [Order Matters: Sequence to Sequence for Sets](https://arxiv.org/abs/1511.06391): Permutation-sensitive sequence learning
   - Explores how sequence models behave when the target object is naturally unordered.
   - Read for: Read for the mismatch between model order and problem structure.
10. [GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism](https://arxiv.org/abs/1811.06965): Training scale
   - Shows a practical route for splitting large model training across accelerator partitions.
   - Read for: Read for the systems bottleneck: scale needs scheduling as much as architecture.
11. [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385): Residual depth
   - Makes very deep convolutional networks easier to optimize through residual connections.
   - Read for: Understand why identity-style paths help gradients and representation reuse.
12. [Multi-Scale Context Aggregation by Dilated Convolutions](https://arxiv.org/abs/1511.07122): Large context without pooling
   - Uses dilation to expand receptive fields while preserving resolution.
   - Read for: Read for the tradeoff between local detail and broad context in vision models.
13. [Neural Message Passing for Quantum Chemistry](https://arxiv.org/abs/1704.01212): Graph neural networks
   - Frames molecule modeling as message passing over graph structure.
   - Read for: Use it to understand graph computation as repeated local communication.
14. [Attention Is All You Need](https://arxiv.org/abs/1706.03762): Transformer architecture
   - Replaces recurrence with stacked attention and feed-forward blocks.
   - Read for: Spend time on multi-head attention, positional information, and parallelism.
15. [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473): Attention precursor
   - Introduces alignment-style attention for translation models.
   - Read for: Read before or after transformers to see what changed and what carried forward.
16. [Identity Mappings in Deep Residual Networks](https://arxiv.org/abs/1603.05027): Residual refinement
   - Clarifies the role of identity paths in residual network optimization.
   - Read for: Read after ResNet to understand why the skip path details matter.
17. [A Simple Neural Network Module for Relational Reasoning](https://arxiv.org/abs/1706.01427): Relational module
   - Introduces a simple module for pairwise relational reasoning over objects.
   - Read for: Track how object representations become inputs to explicit relation computation.
18. [Variational Lossy Autoencoder](https://arxiv.org/abs/1611.02731): Lossy representation
   - Separates useful latent structure from nuisance detail in a variational setting.
   - Read for: Read for what a representation should keep and discard.
19. [Relational Recurrent Neural Networks](https://arxiv.org/abs/1806.01822): Relational memory
   - Combines recurrent state with relational computation over memory slots.
   - Read for: Compare this with plain LSTMs and later attention-based memory.
20. [Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton](https://arxiv.org/abs/1405.6903): Complexity toy model
   - Turns the complexodynamics intuition into a small formal model.
   - Read for: Use it as a theory-side complement to representation-learning papers.
21. [Neural Turing Machines](https://arxiv.org/abs/1410.5401): External memory
   - Augments neural networks with differentiable memory access.
   - Read for: Read for the memory interface and addressing mechanism.
22. [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://arxiv.org/abs/1512.02595): End-to-end scale
   - Demonstrates end-to-end speech recognition at large data and compute scale.
   - Read for: Read for the scaling and systems recipe, not only the acoustic model.
23. [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361): Scaling economics
   - Quantifies how loss varies with model size, data, and compute.
   - Read for: Focus on the compute allocation mindset and its assumptions.
24. [A Tutorial Introduction to the Minimum Description Length Principle](https://arxiv.org/abs/math/0406077): MDL foundation
   - Introduces model selection as compression in a clear tutorial form.
   - Read for: Read before making broad claims about compression and intelligence.
25. [Machine Super Intelligence](https://www.vetta.org/documents/Machine_Super_Intelligence.pdf): Intelligence framing
   - A dissertation-length treatment of machine intelligence and capability framing.
   - Read for: Skim for definitions and assumptions rather than implementation technique.
26. [Kolmogorov Complexity and Algorithmic Randomness](https://www.lirmm.fr/~ashen/kolmbook-eng.pdf): Algorithmic information theory
   - Provides the mathematical background for complexity and randomness claims.
   - Read for: Use selectively; the list points readers to later sections, not necessarily cover-to-cover reading.
27. [CS231n: Convolutional Neural Networks for Visual Recognition](https://cs231n.github.io/): Vision course backbone
   - A practical course bridge for convolutional networks, optimization, and visual recognition.
   - Read for: Use it to ground the vision papers in implementation and training practice.

## Karpathy LLM Systems

A practical LLM stack ordered around transformer mechanics, scaling, instruction tuning, and deployment caution.

- Audience: Builders using Karpathy-style LLM education as the on-ramp to modern language models.
- Difficulty: intermediate
- Estimate: 7 papers, 2-3 focused weeks
- Outcome: You can explain the mainstream LLM recipe from architecture to scale, chat tuning, and frontier evaluation.
- Provenance: Derived from Karpathy's public LLM education track and public community reading-list mirrors around that talk; not presented as an official personal bibliography.
- Trust note: This is an ordered educational bucket. It uses original notes and links to public sources rather than copying talk transcripts or article summaries.

1. [Attention Is All You Need](https://arxiv.org/abs/1706.03762): Architecture
   - The transformer architecture that became the default substrate for LLMs.
   - Read for: Map the architecture to actual inference costs: attention, MLPs, context, and parallelism.
2. [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf): Generative pretraining
   - Shows broad task behavior emerging from next-token training at larger scale.
   - Read for: Read for the move from task-specific NLP systems to broad pretrained models.
3. [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165): In-context learning
   - Makes prompting and few-shot adaptation a central interface for large language models.
   - Read for: Read the evaluations and limitations, not only the scale headline.
4. [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556): Data/compute balance
   - Revises how to allocate compute between model size and training tokens.
   - Read for: Compare this against earlier scaling-law assumptions.
5. [Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155): Chat behavior
   - Turns a pretrained model into an instruction-following assistant through demos, reward modeling, and PPO.
   - Read for: Separate base-model capability from the preference-trained interface.
6. [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971): Efficient open model
   - Shows how careful data and training choices can make smaller foundation models competitive.
   - Read for: Read for the data and efficiency story that shaped the open-weight ecosystem.
7. [Sparks of Artificial General Intelligence: Early Experiments with GPT-4](https://arxiv.org/abs/2303.12712): Capability probing
   - Documents qualitative probes of GPT-4-era behavior across broad tasks.
   - Read for: Read critically: it is useful for capability intuition, but not a substitute for controlled evaluation.

## LeCun World Models

The self-supervised, JEPA, and world-model path around LeCun's autonomous intelligence agenda.

- Audience: Readers comparing the LLM-centric path with a predictive-world-model research agenda.
- Difficulty: advanced
- Estimate: 6 papers, 2-3 focused weeks
- Outcome: You can explain why LeCun emphasizes self-supervised representation learning, prediction, planning, and world models.
- Provenance: Anchored on LeCun's public position paper and related JEPA work; supporting papers are ordered to make that agenda legible.
- Trust note: This is not a ranked LeCun favorites list. It is a source-attributed agenda bucket built from researcher-authored or agenda-adjacent material.

1. [A Tutorial on Energy-Based Learning](https://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf): Energy view
   - Introduces the energy-based modeling frame that underlies parts of LeCun's later agenda.
   - Read for: Read for the scoring view of learning rather than a generative token model.
2. [Learning Deep Representations by Mutual Information Estimation and Maximization](https://arxiv.org/abs/1808.06670): Representation pressure
   - A useful contrastive/self-supervised representation-learning bridge before JEPA-style prediction.
   - Read for: Compare mutual-information pressure with later non-generative predictive objectives.
3. [World Models](https://arxiv.org/abs/1803.10122): Latent environment model
   - A compact world-model demonstration for agents that plan through learned latent dynamics.
   - Read for: Use it to separate representation, dynamics, and controller pieces.
4. [A Path Towards Autonomous Machine Intelligence](https://openreview.net/pdf?id=BZ5a1r-kVsf): Research agenda
   - Lays out an architecture-level argument for self-supervised world models and planning.
   - Read for: Read for the system decomposition and claims about what current LLMs miss.
5. [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://arxiv.org/abs/2301.08243): I-JEPA
   - Applies joint-embedding predictive learning to image representations.
   - Read for: Read for prediction in representation space instead of pixel reconstruction.
6. [Revisiting Feature Prediction for Learning Visual Representations from Video](https://arxiv.org/abs/2404.08471): V-JEPA
   - Extends predictive representation learning toward video-based world understanding.
   - Read for: Read for what changes when prediction moves from images to temporal visual input.

## Human-Compatible AI Safety

A CHAI/Russell-aligned safety bucket from capabilities background to robustness and preference inference.

- Audience: Readers who want a safety path grounded in classic ML, RL, robustness, and preference learning.
- Difficulty: intermediate
- Estimate: 8 papers, 3 focused weeks
- Outcome: You can connect capability milestones to the safety questions around robustness, uncertainty, and preference inference.
- Provenance: Derived from the Center for Human-Compatible AI recommended-materials bibliography, associated with Stuart Russell's research program.
- Trust note: This is a compact ordered subset of a much larger bibliography. Priority and grouping are product-curation choices, not a complete CHAI syllabus.

1. [Deep Learning](https://www.nature.com/articles/nature14539): Capability background
   - A broad review of the deep-learning shift that safety readers need as shared context.
   - Read for: Use it to align vocabulary before moving to control, robustness, and alignment papers.
2. [Human-Level Control through Deep Reinforcement Learning](https://www.nature.com/articles/nature14236): Deep RL milestone
   - Shows neural agents learning from pixels and rewards in Atari environments.
   - Read for: Read for the reward-driven control loop and where specification pressure enters.
3. [Mastering the Game of Go with Deep Neural Networks and Tree Search](https://www.nature.com/articles/nature16961): Search plus learning
   - Combines supervised learning, reinforcement learning, and tree search in a high-profile capability milestone.
   - Read for: Read for how learned priors and planning interact.
4. [Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565): Problem taxonomy
   - Frames practical accident risks such as reward hacking, safe exploration, and distribution shift.
   - Read for: Use it as the checklist for translating model capability into deployment risk.
5. [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137): Preference uncertainty
   - Models alignment as a cooperative game where human preferences are not directly known by the agent.
   - Read for: Read for uncertainty about objectives, not only inverse reinforcement learning mechanics.
6. [Algorithms for Inverse Reinforcement Learning](https://ai.stanford.edu/~ang/papers/icml00-irl.pdf): Preference inference roots
   - An early formal treatment of inferring reward functions from behavior.
   - Read for: Read before CIRL if you want the older objective-inference setup.
7. [Intriguing Properties of Neural Networks](https://arxiv.org/abs/1312.6199): Adversarial fragility
   - Shows that neural networks can be highly sensitive to small input perturbations.
   - Read for: Use this as the entry point for robustness concerns in learned systems.
8. [Explaining and Harnessing Adversarial Examples](https://arxiv.org/abs/1412.6572): Adversarial mechanism
   - Offers a simple explanation for adversarial examples and a practical training response.
   - Read for: Read after the Szegedy paper to understand one mechanism and one mitigation path.

## Agentic LLMs

From prompted reasoning to tools, search, memory, and embodied action.

- Audience: Builders who want the agent paper trail without drowning in benchmarks.
- Difficulty: intermediate
- Estimate: 6 papers, 2 focused evenings
- Outcome: You can explain the basic agent loop and choose when to add tools, memory, search, and skill reuse.
- Provenance: Mixed arXiv and web sources; chosen for mechanism coverage, not recency ranking.
- Trust note: Treat this as a learning sequence. It is not a claim that these are the only or latest agent papers.

1. [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629): Reason and act loop
   - Introduces an interleaved thought/action/observation pattern that made agent traces concrete and inspectable.
   - Read for: Read for the control loop. The core idea is less about one prompt format and more about making reasoning observable between tool calls.
2. [Toolformer: Language Models Can Teach Themselves to Use Tools](https://arxiv.org/abs/2302.04761): Tool-use training signal
   - Shows a self-supervised route for teaching a language model when and how to call external tools.
   - Read for: Focus on how candidate tool calls are generated, filtered, and folded back into training data.
3. [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/abs/2305.10601): Search over thoughts
   - Turns language-model reasoning into a search problem where intermediate thoughts can be sampled, evaluated, and expanded.
   - Read for: Compare this with ReAct: ReAct grounds actions in observations, while Tree of Thoughts spends compute on deliberation.
4. [Reflexion: Language Agents with Verbal Reinforcement Learning](https://arxiv.org/abs/2303.11366): Memory and self-critique
   - Adds verbal feedback and episodic memory so agents can improve behavior across attempts without weight updates.
   - Read for: Watch the boundary between useful reflection and benchmark-specific retry loops.
5. [Generative Agents: Interactive Simulacra of Human Behavior](https://arxiv.org/abs/2304.03442): Long-lived behavior
   - Builds memory, reflection, and planning into believable simulated agents with persistent state.
   - Read for: Read for the memory architecture and scheduling loop, not the demo novelty.
6. [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://arxiv.org/abs/2305.16291): Skill library
   - Combines exploration, code generation, curriculum, and a reusable skill library in an embodied environment.
   - Read for: Look at how skills are represented and reused. That is the portable idea for product agents.

## Transformer Foundations

The shortest route from attention to modern foundation-model intuition.

- Audience: Engineers who know ML basics and want the historical dependency chain.
- Difficulty: starter
- Estimate: 6 papers, 3 focused evenings
- Outcome: You can trace the dependency chain from alignment-style attention to pretraining and scaling.
- Provenance: arXiv and conference papers selected from widely cited transformer-era work.
- Trust note: Ordered for concept buildup, not by publication date alone.

1. [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473): Attention before transformers
   - Introduces alignment-style attention for sequence-to-sequence translation.
   - Read for: Read this first so transformer attention feels like a generalization rather than a sudden invention.
2. [Pointer Networks](https://arxiv.org/abs/1506.03134): Attention as selection
   - Uses attention to point at input positions, making variable-size structured outputs possible.
   - Read for: Notice how attention becomes an addressing mechanism, not just a soft summary.
3. [Attention Is All You Need](https://arxiv.org/abs/1706.03762): Transformer architecture
   - Removes recurrence and convolution in favor of stacked self-attention and feed-forward blocks.
   - Read for: Spend time on positional encoding, multi-head attention, and why parallelism changed scaling economics.
4. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805): Pretraining pattern
   - Establishes masked-language-model pretraining as a reusable foundation for language understanding tasks.
   - Read for: Separate the architecture from the training objective. BERT matters because the objective made transfer routine.
5. [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165): In-context learning
   - Shows that scale can unlock task adaptation from prompts without gradient updates.
   - Read for: Read the evaluation spread and failure modes, not just the headline few-shot result.
6. [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361): Scaling economics
   - Quantifies how loss changes with model size, data, and compute, shaping how labs planned frontier training.
   - Read for: Focus on the tradeoff curves and assumptions. Later scaling work revises details, but this sets the mindset.

## Alignment And RLHF

The path from preference learning to helpful, harmless, and instruction-following assistants.

- Audience: Product and ML engineers evaluating what alignment training actually changed.
- Difficulty: intermediate
- Estimate: 5 papers, 2 focused evenings
- Outcome: You can separate SFT, reward modeling, RLHF, constitutional feedback, and direct preference objectives.
- Provenance: arXiv papers selected for training-process coverage.
- Trust note: This is a mechanism path, not a policy or safety-completeness claim.

1. [Fine-Tuning Language Models from Human Preferences](https://arxiv.org/abs/1909.08593): Preference modeling
   - Early demonstration that human preference data can steer language model behavior through reward modeling.
   - Read for: Read for the data collection and reward-model loop before later assistant-scale versions.
2. [Learning to Summarize with Human Feedback](https://arxiv.org/abs/2009.01325): RLHF applied
   - Applies preference learning to summarization and makes the reward-model plus policy-optimization recipe concrete.
   - Read for: Track where human preference wins and where reward hacking or evaluation gaps can enter.
3. [Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155): Assistant RLHF
   - Shows the instruction-following recipe behind InstructGPT: supervised demos, reward model, and PPO.
   - Read for: Compare model size versus alignment data. Smaller aligned models can beat larger base models on preferred behavior.
4. [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073): AI feedback
   - Uses a written constitution and model-generated critiques to reduce reliance on direct human labels.
   - Read for: Focus on what the constitution controls and what still depends on human choices.
5. [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290): Preference objective
   - Replaces explicit reward modeling and RL with a direct objective over preference pairs.
   - Read for: Read after PPO-based RLHF so the simplification and tradeoffs are obvious.

## Retrieval And RAG

From dense retrieval to retrieval-augmented generation and source-grounded answers.

- Audience: Engineers building cited answer systems over private or curated corpora.
- Difficulty: intermediate
- Estimate: 5 papers, 2 focused evenings
- Outcome: You can reason about retrievers, readers, generators, and when citations are evidence versus decoration.
- Provenance: arXiv and conference papers selected for retrieval architecture coverage.
- Trust note: This path prioritizes durable mechanisms over benchmark leaderboard freshness.

1. [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906): Dense retriever
   - Popularizes dual-encoder dense retrieval for open-domain QA.
   - Read for: Focus on negative sampling and why retriever training dominates downstream answer quality.
2. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401): Generator with retrieval
   - Combines parametric generation with non-parametric retrieved memory.
   - Read for: Read for the split between RAG-sequence and RAG-token and how retrieval changes generation.
3. [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909): Pretraining with retrieval
   - Trains language representations with a latent retriever over a knowledge corpus.
   - Read for: Look at how retrieval is folded into pretraining rather than bolted on only at inference time.
4. [ColBERTv2: Efficient and Effective Retrieval via Lightweight Late Interaction](https://arxiv.org/abs/2112.01488): Late interaction
   - Shows a practical retrieval design between simple bi-encoders and expensive cross-encoders.
   - Read for: Read for the quality/latency tradeoff: late interaction often matters in real search products.
5. [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://arxiv.org/abs/2310.11511): Adaptive retrieval
   - Adds reflection tokens so models can decide when to retrieve and critique generations.
   - Read for: Read for retrieval gating and answer critique, not just the model-specific implementation.

## Diffusion And Generative Vision

The short path from denoising objectives to latent diffusion image models.

- Audience: Readers who want the core image-generation lineage before modern product papers.
- Difficulty: advanced
- Estimate: 5 papers, 3 focused evenings
- Outcome: You can explain why denoising, score matching, and latent spaces made diffusion practical.
- Provenance: arXiv and conference papers selected for conceptual continuity.
- Trust note: This is a foundations path; it intentionally skips many newer product and video-model variants.

1. [Extracting and Composing Robust Features with Denoising Autoencoders](https://www.cs.toronto.edu/~larocheh/publications/icml-2008-denoising-autoencoders.pdf): Denoising representation
   - Introduces denoising as a representation-learning pressure before diffusion models made it generative infrastructure.
   - Read for: Read for the intuition that reconstruction from corruption teaches structure.
2. [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://arxiv.org/abs/1503.03585): Diffusion framing
   - Frames generation as reversing a gradual noising process.
   - Read for: Understand the forward and reverse processes before jumping to modern samplers.
3. [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239): DDPM recipe
   - Makes denoising diffusion models competitive and readable as a practical training objective.
   - Read for: Pay attention to the simplified objective and sample quality versus likelihood tradeoff.
4. [Score-Based Generative Modeling through Stochastic Differential Equations](https://arxiv.org/abs/2011.13456): Score/SDE view
   - Unifies score-based models and diffusion through SDEs.
   - Read for: Use this to connect the probabilistic and continuous-time views.
5. [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752): Latent diffusion
   - Moves diffusion into a compressed latent space, making high-resolution generation much cheaper.
   - Read for: Read for the engineering move: generate in latent space, decode later.

## Compression And Generalization

A compact path through simplicity, description length, and why models generalize.

- Audience: Readers curious about the theory thread behind compression, intelligence, and learning.
- Difficulty: advanced
- Estimate: 5 readings, 2 focused evenings
- Outcome: You can explain why compression is a useful lens for generalization without overstating it.
- Provenance: A mix of papers, tutorials, and one explanatory essay because the idea spans formal theory and intuition.
- Trust note: This path is conceptual. It is meant to orient, not settle debates about intelligence or generalization.

1. [Keeping Neural Networks Simple by Minimizing the Description Length of the Weights](https://www.cs.toronto.edu/~hinton/absps/colt93.pdf): Neural nets and MDL
   - Connects neural-network regularization to minimum description length and simpler explanations.
   - Read for: Read for the bridge between practical regularization and an information-theoretic preference for simplicity.
2. [A Tutorial Introduction to the Minimum Description Length Principle](https://arxiv.org/abs/math/0406077): MDL grounding
   - A practical entry point for understanding MDL as model selection by compression.
   - Read for: Use this to get the vocabulary before reading stronger claims about compression and intelligence.
3. [The First Law of Complexodynamics](https://scottaaronson.blog/?p=762): Complexity intuition
   - Explains why complexity can rise and fall in closed systems, motivating the coffee automaton thought experiment.
   - Read for: Treat it as intuition-building rather than a formal ML result.
4. [Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton](https://arxiv.org/abs/1405.6903): Complexity toy model
   - Formalizes a small model for understanding how apparent complexity can emerge and dissipate.
   - Read for: Read after the essay. The value is the formal move from story to measurable complexity.
5. [Variational Lossy Autoencoder](https://arxiv.org/abs/1611.02731): Representation and compression
   - Separates lossy representation from nuisance detail in a variational autoencoder setting.
   - Read for: Use it as a modern ML bridge back to the core question: what should a representation preserve or discard?