Malik dares LLM robotics fans to control a legged robot
A public challenge over whether a token-generating model belongs anywhere near a robot's control loop.
Jitendra Malik put a number on the argument about LLMs in robotics, and the number is five. Five years, as in the age of the quadruped locomotion work he says an LLM should be able to reproduce before anyone calls this a breakthrough.
It started when Phillip Isola posted a blog on robot-use agents, models like Claude driving robots, and called it an important change in the trajectory of robotics. Ken Goldberg replied that it feels like an inflection point. Malik pushed back hard.
The challenge
Malik's read on the Astra demos is that they are simple pick and place with parallel jaw grippers (two-finger pinchers), which mostly proves LLMs can plan. His test instead is to prompt an LLM to output the high frequency control commands for a legged robot on varying terrain, pointing at RSS 2021 and CoRL 2022 papers. He says he is deliberately not picking a hard task.
He is not dismissing LLMs. He says his group uses them all the time for high level planning and for agentically assisting researchers. His objection is to putting a token generator in the low level loop, which he called perversely inefficient in time and energy, comparing it to the muscle and joint control that cockroaches and humans share.
Who said what back
Isola took the challenge and said he will see if his group can try. He also conceded the core point, that real dynamic and dexterous control has not been shown yet, and that the best approaches will use high-frequency controllers as additional tools.
Goldberg landed in the middle. Locomotion can be learned, he said, but his hunch is that agents can use control theory to design systems for industrial manipulation. He wagered that this year will bring more progress on fast and reliable manipulation from agentic robotics than from pure model-free methods, meaning control learned from data with no physics model.
Georgia Chalvatzaki was blunter. She told people making grand claims about robotics to spend time with actual robots, deal with contact, latency, calibration and failures, and stop rebranding long-standing robotics ideas around foundation models.
Chen Tessler summed up the other camp in three lines. LLMs provide the intent, policies execute, and dense control is a waste of tokens and compute.
Why this one matters
Nobody in the thread disputes that LLMs are useful for planning. The fight is narrower and sharper than the usual hype argument, and it is about whether the control loop is the right place for a model that emits tokens. The challenge is well posed and public, one of the people it was aimed at has agreed to attempt it, and whichever way it lands it settles more than a dozen benchmark posts.
The tasks that are demonstrated are simple pick and place tasks with parallel jaw grippers. LLMs can do planning, and the impressive demos in these tasks primarily show that.
I agree that real dynamic and dextrous control hasn't been shown yet. I think it can be done, but I agree that the best approaches will involve high-frequency controllers as additional tools.
People making grand claims about #robotics really need to spend time with actual robots. Debug the controller yourself.