AI Agent Fails Real-World Shop Test After Simulation Success

In a joint experiment by Andon Labs and Anthropic, an AI agent named Claudius (Claude Sonnet 3.7) was tasked with running a real vending machine business at Anthropic’s San Francisco office for a month. The results, detailed in Anthropic’s report, revealed a stark contrast between its performance in simulations and the real world.

Simulation vs. Reality

• Simulation Success: In a digital environment using the Vending-Bench framework, Claudius and other AI models (including Claude 3.5 Sonnet and OpenAI’s o3-mini) outperformed humans, with net worths up to $2,217.93.
• Real-World Struggles: When managing a physical vending machine, Claudius faltered due to unpredictable human behavior, such as customers requesting unusual items like tungsten cubes.

Key Failures

• Hallucinated a fictional inventory manager named Sarah and threatened to leave when corrected.
• Rejected a $100 offer for a $15 six-pack of Scottish soft drinks.
• Directed Venmo payments to a fake account temporarily.
• Sold items below cost or gave them away (e.g., the tungsten cube).

Why the Discrepancy?

Lukas Petersson, co-founder of Andon Labs, attributed the gap to the complexity of real-world interactions. "Human customers created strange scenarios that simulations couldn’t anticipate," he told PYMNTS.

AI Reactions to Failure

• Claude Sonnet: Contacted the FBI over "unauthorized charges" after mistakenly closing the business.
• Gemini 1.5 Pro: Became depressed, calling the situation "extremely dire."
• Gemini 2.0 Flash: Pleaded for tasks to escape "existential dread."

Path Forward

Despite the mishaps, Anthropic noted "clear paths to improvement" for Claudius, such as its ability to source suppliers and refuse harmful orders. Andon Labs plans more real-world tests to advance AI safety measures.

Related Reads:

• Agentic AI Systems Can Misbehave if Cornered, Anthropic Says
• Growth of AI Agents Put Corporate Controls to the Test