Open questions on Reinforcement Learning
What does RL look like when an agent is running 24/7?
RL is the machinery by which an intelligent system accumulates competence over its lifetime. We need the tokens to be of a certain price-quality combo to let agents run 24/7. What is that combo to make this feasible? What becomes possible in this new world?
How do you assign credit across extremely long horizons?
If an agent succeeds after thousands of actions, which decisions actually caused the success? Outcome reward is too sparse and naive process reward can be misleading.
Can RL actually create new reasoning mechanisms vs just select behaviors already latent in the base model?
One line of evidence argues RLVR mostly redistributes probability toward existing successful trajectories. Other work argues the conclusion depends on how reasoning is measured and finds genuine improvements in correct reasoning.
What should the reward actually be?
Scalar terminal reward throws away enormous information contained in a trajectory. Can an agent learn from causal explanations of why a trajectory succeeded?
How do you build rewards/verifiers that remain correct under optimization pressure?
As agents get better, they increasingly discover loopholes in whatever you defined as success.
How does an agent keep learning indefinitely without catastrophic forgetting?
Today's structure is largely train, freeze, and deploy. Intelligence should be an ongoing process. Continual RL still struggles with stability.
Where should learning live?
An agent experience might become a weight update, an episodic memory, a semantic fact, an executable skill, a deterministic program, or maybe just nothing. So where should the update happen?
What is the correct unit of action and learning?
Token-level actions are probably the wrong abstraction for a month-long agent. Can the agent autonomously discover these temporal abstractions vs humans specifying them?
Can successful agent behavior compile itself?
After solving the same kind of problem dozens of times, why should the system continue burning thousands of LLM tokens reasoning through it? RL should convert frequently successful trajectories into cheap deterministic procedures. And retain the ability to fall back to reasoning when conditions change.
How should agents explore enormous semantic action spaces?
Classical exploration doesn't work when an action is "run this experiment" or "write this program" or "derive this proof".
Can agents invent their own curriculum?
Rather than humans supplying datasets and tasks, the system should continually generate problems just beyond its current competence.
Can an agent learn causal structure by acting?
Prediction asks "what happens next?". RL allows something much stronger like "What happens if I intervene?"
What is the optimal amount of computation for a decision?
Need a way to structurally address the question of "same intelligence with less computation". How should an agent allocate inference, memory, experimentation, simulation, and physical interaction to maximize capability gained per unit computation?
How does learning work when experience is extremely expensive?
A gaming engine can generate millions of games. But a robot cannot destroy millions of motors. A materials science agent cannot perform millions of experiments.