Beyond Verifiable Rewards: How to Do Reinforcement Learning in Domains Without Standard Answers

Layer 1: The Intuition

1.1 Exercises with Answers vs. Exercises without Answers

• Practicing math: After finishing, check the answer; if wrong, redo it. One person can practice ten thousand problems alone;
• Practicing writing: After writing an article, who grades it? The tastes of teachers, readers, and editors all differ, and grading is very slow.

Reinforcement learning requires large-scale, fast, and reliable scoring. Verifiable domains have it; open domains do not.

1.2 Problems with Existing Approaches

Layer 2: Existing Evidence

Layer 3: Possible Solutions

3.1 Decompose: Break Open-Ended Tasks into Verifiable Parts

It is hard to verify if a research review is "well-written," but many of its components can be:

The remaining truly "subjective" parts (writing style, insight) are then left to humans or models for judgment. The larger the proportion of verifiable parts, the more reliable the reward.

3.2 Using World Models as Verifiers (Concept)

For tasks like strategic planning or business decision-making, plans can be executed in a simulated environment to observe the results:

• A marketing plan → Run it among a simulated user group;
• A negotiation strategy → Negotiate with a simulated opponent;
• An engineering plan → Run it in a simulation.

Simulators are imperfect, but they provide a feedback source that cannot be pleased (see "World Models for Post-training").

3.3 Combining Multiple Weak Verifiers

A single verifier is easy to exploit, but it is much harder to simultaneously deceive multiple independent verifiers (similar to multi-source cross-validation):

$$r = \min\big(r_1, r_2, \dots, r_k\big) \quad \text{or} \quad r = \text{weighted average, but with lower weights for samples with high disagreement}.$$

Using the minimum value instead of the average means the model must satisfy all verifiers; exploiting a loophole in a single verifier becomes meaningless.

3.4 Debate and Scalable Supervision

When a task exceeds human evaluative capacity (such as reviewing a highly professional paper), let two models debate: one defends the conclusion, and the other looks for flaws. Humans only need to judge whether the specific issues pointed out by both sides hold true. Theoretically, the side that lies is more likely to be caught by the opponent.

3.5 Delayed Real-World Feedback (Concept)

The true results of some tasks are only known after a long time: an investment advice, a medical plan, a product decision. "Delayed rewards" can be designed:

• First, train using proxy rewards;
• Once the real result comes out (weeks, months later), go back and calibrate the reward model;
• Turn predictive tasks into "prediction market" style training: the model provides probabilities for future events, and scores are assigned based on calibration accuracy once the event occurs (Brier Score).

Predicting the future is a naturally verifiable task that cannot be cheated in advance.

3.6 Hypotheses to be Verified

1. If the decomposed verifiable sub-tasks are done well, does it mean the whole is done well? Or will there be a case of "every item meets the standard, but the whole is mediocre"?
2. Will combining verifiers make the model too conservative (producing only mediocre outputs that no verifier can find fault with)?
3. Can predictive rewards be transferred into more general judgment capabilities?

Summary