RoboColosseum runs 20,000 real robot trials to rank the models
Hugging Face, BitRobot Network, EAIC and Jiafei Duan put 100+ robots and 30 full-time evaluators behind an open benchmark run in the wild.
A new open evaluation effort called RoboColosseum has already run more than 20,000 real robot trials, and its first result is that the model you would pick out of the box is not always the model you would pick after fine-tuning.
The project comes from Hugging Face, BitRobot Network, EAIC and Jiafei Duan. It offers access to 100+ robots, including Unitree G1s, LeRobot SO-101s, YAMs and Franka arms, across 140+ in-the-wild scenes, with open rollout data and a pledge of 10,000 free evaluation hours for academic partners.
Thirty people resetting scenes
The part that makes this different from a leaderboard is the labor behind it. Duan says 30 full-time evaluators work on site in shifts across the global fleet, resetting scenes, maintaining robots and keeping trials running around the clock. That is the bottleneck nobody wants to pay for, and it is why most robot policy claims get compared on video rather than on numbers.
There are two tracks. In the fine-tune track, shared demonstrations and repeatable setups let the team compare how models adapt to the same data. In the open track, models go head to head on new tasks in the wild, testing generalization. Duan says the trials so far cover four robot platforms, more than 6,800 tasks and over 150 real-world scenes.
What the first paper measured
Yi Ru (Helen) Wang, who posted the paper thread, says the study ran 5,699 real-world robot comparisons across 124 sites, evaluating 5 VLA policies (models that turn camera images and text into robot actions) and 3 frontier AI agents. The headline finding from Duan's thread is the ranking flip, out-of-the-box performance and post-fine-tune performance do not line up.
Which specific models won, and by how much, is in the paper rather than the threads. The sources do not name the five policies or the three agents, so treat the ranking as unreleased in public summary form for now.
Duan frames the ambition bluntly, arguing physical AI needs shared infrastructure the way astronomy and particle physics have it, because every week brings another paper and another claimed best model. Whether a community benchmark actually settles those arguments depends on labs sending their policies in. The team is taking sign-ups and says it wants to expand beyond the four platforms it tests today.
Robotics needs its own Hubble Space Telescope or CERN, a scientific megaproject to cut through the noise in physical AI.
Another paper, another launch, another “best” model. Do those gains hold up on real robots, at scale, in the wild?
We ran 5,699 real-world robot comparisons across 124 sites, evaluating 5 VLA policies and 3 frontier AI agents.
Sound like you were at the demo
One email a day. Under three minutes. The demos, the deployments and the part the press release left out.