New benchmark runs coding agents on 1,090 embodied tasks

Manling Li has released the Embodied First Exam, a benchmark that she says runs coding agents across 1,090 embodied tasks in 31 simulation suites on 44 robots, from a single line of code.

Manling Li

@ManlingLi_

X

Real-time eval, where the world won't pause

That line is a harbor run command where you drop in your own harness and your own model. The pitch is that you should not have to wire up thirty different simulators to find out whether your agent can do anything in a body.

What is in it

Li's post lists the categories. Puzzles with spatial constraints. Olympic sports with multi-robot collaboration. Dynamic, agile, whole-body control. AutoResearch with sim2sim (testing whether something that works in one simulator transfers to another). And real-time evaluation, which is the one worth pausing on.

In most robot benchmarks the simulator politely waits while the model thinks. Li says this one does not. Her phrasing is that the world will not pause. That turns thinking time into a cost, which is the thing that separates a planner that works on paper from one that works while a box is falling.

The framing is aimed squarely at the gap between pick-and-place and everything else. Her post asks whether your agent can go far beyond pick-and-place, and the task list is built to answer that rather than to produce another tidy table of grasp success rates.

No scores yet

This is the exam, not the results. Li did not publish numbers for any agent, so nobody knows yet how current coding agents perform on the 1,090 tasks, or how much of the suite is already saturated versus hopeless. Benchmarks also only matter once people run them, and a one-line command is a bet that they will.

Separately, Chris Paxton posted a line that reads like the mood around work like this.

Chris Paxton

@chris_j_paxton

X

Robotics is finally just another sub area of ai