The one-sentence version
Reinforcement learning assumes an agent is maximizing rewards. The Free Energy Principle proposed by neuroscientist Karl Friston offers a different picture: living organisms (and the brain) are minimizing surprise, meaning they try to make their predictions of the world as accurate as possible; and to achieve accurate predictions, they both update beliefs (learning) and take actions to make the world conform to expectations (active inference). The concept here is: to use the active inference framework to design post-training objectives for LLMs, shifting the model's motivation from "getting high scores" to "understanding and stably predicting the world," which naturally incorporates curiosity (exploring areas that reduce uncertainty).
Positioning
Research concept. The Free Energy Principle itself is controversial within the neuroscience community (some argue it is too universal to be falsifiable). This article distinguishes between existing evidence and speculation.
Layer 1: The Intuition
1.1 Two Types of Motivation
| Reward Maximization | Minimizing Surprise | |
|---|---|---|
| Analogy | A child acting for candy | A child wanting to understand how the world works |
| Motivation | External scores | Making the world understandable and predictable |
| Risk | Cheating for candy (reward hacking) | Hiding in a dark room where nothing ever happens ("The Dark Room Problem") |
1.2 Two Ways to Reduce Surprise
Suppose you predict "it won't rain today," but you step out and get soaked; you are surprised. You can:
- Change beliefs: Check the weather forecast next time (learning);
- Change the world: Bring an umbrella, or simply stay indoors (action).
Active inference suggests: the agent does both, and will actively collect information to reduce future uncertainty, such as looking out the window before heading out. This is the mathematical source of curiosity.
1.3 Connection to Large Language Models
Pre-training's "predict the next token" is itself a way of minimizing a type of surprise (cross-entropy = the "degree of unexpectedness" regarding the actual next word). The concept is: to expand this objective from passive text to active action.
Layer 2: Existing Evidence
| Direction | Content |
|---|---|
| Free Energy Principle and Active Inference (since Friston, 2006) | Unifies perception, learning, and action into the minimization of "variational free energy" (an upper bound on surprise); widely used as an explanatory framework in neuroscience and psychiatry (e.g., explaining hallucinations as overly strong priors) |
| Predictive Processing Theory (Clark, Hohwy, etc.) | The brain is a hierarchical prediction machine that only passes "prediction errors" up the hierarchy (see Predictive Processing and the Next Token) |
| Curiosity-driven Reinforcement Learning | ICM (2017) uses prediction error as an intrinsic reward; RND (2018) achieved breakthroughs in difficult-to-explore games (e.g., Montezuma's Revenge) |
| The "Noisy TV" Problem | Agents rewarded purely by prediction error may become obsessed with staring at random noise (which is impossible to predict and thus always "interesting"). This shows the need to distinguish between "reducible uncertainty" and "irreducible randomness" |
| Engineering implementation of Active Inference | Implementations exist for small-scale control tasks and robotics, but there is little work on scaling this to large-scale models |
Layer 3: Specific Forms of the Concept
3.1 Expected Free Energy
In active inference, the basis for choosing actions is Expected Free Energy, which can be roughly divided into two terms:
The agent selects the policy that minimizes : it wants to learn something while also achieving its preferred outcomes.
- Epistemic value solves the "Dark Room Problem": hiding in a dark room yields zero epistemic value because nothing can be learned;
- Calculating only "reducible" uncertainty (information gain) rather than raw prediction error solves the "Noisy TV Problem": nothing can be learned from noise.
3.2 For LLM Post-training (Reasonable Speculation)
| Component | Correspondence |
|---|---|
| Generative Model | The LLM itself (predictions about the world and conversation) |
| Observations | Environmental feedback, tool returns, user responses |
| Preferred Outcomes | Defined by tasks or values (e.g., "the user's question is resolved") |
| Actions | Asking questions, calling tools, conducting experiments, answering |
A specific behavioral prediction: a model trained this way will actively ask questions or verify information when uncertain, because asking has high epistemic value; rather than, like some current models, stubbornly hallucinating an answer.
3.3 Relationship with Other Directions
- Relationship to Interventional Training: The "information gain reward" in the latter can be seen as a special case of epistemic value within expected free energy;
- Relationship to World Model Post-training: Active inference requires an interactive world to generate observations;
- Relationship to Alignment: The preference term is where values are encoded, resembling "what the expected world should look like" more than a "reward function."
3.4 Hypotheses to be Verified
- Can expected free energy be efficiently approximated at the scale of large models?
- Will models trained this way truly have fewer hallucinations and be more willing to admit uncertainty?
- How should the preference term be set to avoid new forms of hacking (e.g., making the user "stop asking questions" to reduce surprise)?
- Is the universality of the Free Energy Principle an advantage or a disadvantage: is it just a different way of phrasing reinforcement learning?
Summary
| Layer | Key points |
|---|---|
| Vision | Agent minimizes surprise: through learning (changing beliefs) and action (changing the world) |
| Advantages | Endogenous curiosity; information gain term avoids the dark room and noisy TV problems |
| Basis | Friston's Free Energy Principle, predictive processing, curiosity-driven RL |
| Prediction | Post-trained models will be more inclined to ask and verify when uncertain |