MEMI hits 97% on RoboMME by inferring what robots remember

Yuejiang Liu released MEMI, a method that figures out what a robot should remember by working backwards from the decisions in its demonstrations. On RoboMME's 16 memory-dependent tasks, he reports 97.0% end to end and 99.4% when paired with a low-level controller, which he calls nearly solving the benchmark for the first time.

Yuejiang Liu

@liu_yuejiang

X

Demos record what a robot did, never what it remembered

The framing is the interesting part. A demonstration logs what the robot did, never what information it was holding onto when it did it. So memory is hidden, the same way reward is hidden in inverse reinforcement learning. MEMI casts memory learning as an inverse problem and infers the memory from the decisions.

Three rules to pin it down

That problem is underdetermined, because many possible memories explain the same sequence of actions. MEMI narrows it with three principles. The memory has to be sufficient, meaning it reproduces the decisions. It has to be predictable from the past only, with no peeking ahead at future frames. And it has to be necessary, carrying no detail the policy ignores.

The loop uses one reasoning model in two jobs. An annotator infers memory from the demonstrations, a policy uses that memory to act, and the three principles generate feedback that refines both. Liu says the process is fully offline and runs without prescribed memory labels, so nobody has to hand-write a list of what matters.

The numbers

Two setups improved. Astra-Plan went from 92.4 to 99.4%, and Astra-Act went from 88.3 to 97.0%, which Liu calls a new end-to-end state of the art. He says gains hold across all four reasoning models tested, and that even the cost-efficient Claude Haiku climbs from 73.9 to 90.8%, which he describes as reaching human-level performance.

Co-author Yilun Du framed the motivation as building embodied agents that can effectively remember what they have seen in the past. The work is credited to Liu with @sooyoung_ryu, @YinpeiD, @du_yilun and @chelseabfinn. Code, prompts and logs are all released.

Shown versus claimed. These are benchmark numbers reported by the authors on RoboMME, not footage of a physical robot recalling something it saw ten minutes earlier in a real kitchen.

Yilun Du

@du_yilun

X

we infer what an agent needs to remember to reproduce the demonstrated decisions