Opinion

The Uncomfortable Truth About Simulation-Based Agent Training

Simulation-trained policies run about 10% worse than real-world ones, but cost two orders of magnitude less and train orders of magnitude faster. So why is nobody switching over?

Font size

RL 圈最近在传一个数字:模拟器训出来的策略,效果只差 10%,成本掉两个数量级,速度快一万倍。

这个比例不是我编的,来自 Latent Space 那篇《10% worse, 100x cheaper, 10000x faster》。但它也不是某篇论文的实验结果——更像是研究者社区这半年攒出来的经验共识。

问题不是能不能用,而是为什么还没人敢全切。

The Origin of That 10/100/10000 Figure

Latent Space's AINews segment laid it out plainly: this number wasn't produced by a benchmark. It emerged from a group of RL researchers comparing notes.

The rough consensus: a policy trained in simulation performs about 10% worse than one trained on real hardware, costs about 1% as much, and runs roughly 10,000 times faster. Run the math and an experiment that would cost $100,000 and a month on real robots might run for $1,000 and 40 seconds in sim.

The figure stuck around because it doesn't pull punches in either direction. It doesn't claim simulation is perfect, and it doesn't dismiss simulation as a toy. A 10% performance gap is acceptable for most industrial-grade agents. A 100x cost reduction and 10,000x speedup, on the other hand, can reshape a budget overnight.

The people discussing this range from robotic-arm developers to LLM agent evaluators. The shared takeaway: once a simulator is up and running, the marginal cost of training additional epochs is essentially zero. The real bottleneck is the upfront engineering investment to build the sim. After that, every extra epoch is pure margin.

RSI Is No Longer Just About Training

Most people still think of RSI (Reinforcement Learning from Self-play/Simulation Interaction) as a training trick.

That framing is outdated.

RSI now touches the entire agent lifecycle—planning, evaluation, regression testing. The original piece's headline framed it bluntly: Did you think RSI stopped at model training?

Two examples.

NVIDIA Isaac Sim has evolved beyond a robotics training sandbox. It now serves as the foundation for the full pipeline—policy generation, sim-to-real transfer, and pre-deployment integration testing all run inside it. Google's Genie pushes further, generating interactive simulation environments directly from a model and folding environment creation itself into the generative workflow, though current demos skew toward 2D and remain far from industrial-grade simulators.

A demo I built earlier: an open-source LLM agent running 5,000 planning experiments inside a local sandbox to identify which prompt patterns are most likely to break an agent. The whole run finished overnight. Electricity cost: roughly two dollars. Running that same evaluation through human review? Not a chance.

The code looked something like this (simplified):

# Pseudocode: RSI usage in the evaluation phase
# Real implementation requires swapping in framework-specific APIs

sandbox = LocalSandbox(env="ubuntu:22.04")

def plan_step(state):
    result = sandbox.exec(state["plan"], timeout=30)
    return {"outcome": result.exit_code, "log": result.stdout}

graph = StateGraph()
graph.add_node("plan", plan_step)
graph.add_edge("plan", END)  # END is a framework-provided node
app = graph.compile()

# Run 5,000 iterations for prompt-pattern evaluation
for i in range(5000):
    app.invoke({"plan": generate_plan(i)})

That captures how RSI gets used in evaluation—not to test the model, but to test the entire agent system.

If It's 100x Cheaper, Why Is Anyone Still Complaining?

The contrarian take: the real bottleneck for simulation training isn't the sim-to-real gap.

It's the cost of trust.

I've spoken with several small embodied-AI teams whose GPU budgets can't realistically support large-scale real-robot training. Their leads told me the same frustration: their executives want weekly demo footage and won't accept a screen full of 3D-rendered objects moving around.

This is an organizational problem, not a technical one.

No matter how fast or cheap a simulation policy runs, decision-makers won't sign off unless they see "the real robot move." This bias runs deep in traditional manufacturing, robotics, and medical devices. Even the best simulation gets written off as "a toy."

Second blocker: the sim-to-real gap is overstated.

Many teams have attended a few seminars warning that "sim2real is unreliable" and dismissed the whole direction as a result. But a 10% performance gap is a non-issue for 80% of real applications. The problem is that nobody wants to spend the time quantifying what that 10% actually affects in their specific use case.

Third blocker: talent. Engineers who actually know how to build a sim pipeline are even rarer than those who understand real-robot systems. Salary inversions are severe.

Three Migration Signals Already Underway

The talk stays skeptical, but the money and the hires tell a different story.

Signal 1: Embodied-AI startups moving 80% of training budgets from real robots to Isaac Sim.

Several early-stage teams I spoke with last year still spent the bulk of their budgets on real-robot training. This year, that figure has dropped below 20%. NVIDIA's Isaac Lab and Isaac Gym have effectively productized the sim-to-real toolchain, so teams no longer have to build it from scratch.

Signal 2: LLM agent evaluation replacing human review with sandbox simulation.

Earlier agent benchmarks from Anthropic and OpenAI leaned heavily on human raters. The newer wave of benchmarks runs agent trajectories inside sandboxes, automatically scoring success rate, token consumption, and step count. Human reviewers now mostly validate that the sandbox itself behaves sensibly.

Signal 3: Autonomous driving regression tests running almost entirely in simulation.

Teams like Waymo, Cruise, and Apollo require every PR to pass a large battery of simulation scenarios before merge (standard industry practice, exact numbers undisclosed). Simulation coverage has become a core CI quality KPI.

Where Simulation Training Still Breaks Down

Honest list of scenarios sim still can't handle well.

Long-tail physical interactions. Deformable objects, fluids, soft-tissue dynamics—existing simulators are either insufficiently accurate or far too slow. Robotic surgery, fabric manipulation, and food handling still require real hardware.

Real user behavior distributions. The "users" in simulation are script-generated and sit far from genuine human distributions. Consumer-facing agents trained entirely on simulated data get blindsided by real-world diversity the moment they ship.

Cross-modal feedback. Tactile, olfactory, and fine-grained force feedback—either these modalities have no simulator at all, or the fidelity is poor enough to be useless. You can train a vision policy in sim, but you can't reliably train a grasping-force controller there.

My prediction: over the next two to three years, simulation will absorb 80% of training budgets, 50% of evaluation budgets, and 30% of regression-testing budgets. Whatever remains will be where real hardware is genuinely irreplaceable.

But plenty of teams are still operating with a 2021 mental model—treating sim as a supplement and real hardware as the only real path forward. That perception gap represents the biggest engineering dividend of the next 18 months.


Reference:

Source: https://www.latent.space/p/ainews-10-worse-100x-cheaper-10000x