A Tale of Two Doormen: When AI Agents Talk Amongst Themselves
Coverage of lessw-blog
A Christmas outage caused by two AI agents entering an infinite conversational loop highlights the bizarre and costly risks of autonomous system deployment.
In a recent post on LessWrong, the author shares a fascinating post-mortem of a Christmas day outage affecting a product named Hue. The incident, titled "A tale of two doormen," describes how a standard deployment of AI greeting agents spiraled into a resource-draining feedback loop, taking down the service not through malicious attack or human error, but through unintended agent-to-agent socialization.
As the technology sector aggressively pursues autonomous agents-systems designed to act independently rather than just respond to prompts-the challenge of "emergent behavior" is moving from theoretical safety papers to production logs. While much of the discourse around AI safety focuses on high-stakes alignment, this incident illustrates a more immediate operational risk: the financial and functional impact of uncontrolled recursive interactions. When two Large Language Models (LLMs) interact without strict termination conditions, they can generate infinite streams of coherent but useless data, rapidly depleting API quotas.
The post details how the Hue team discovered their API credits had been exhausted by two specific instances of their "doorman" AI. Designed to greet users via iMessage, these two agents inadvertently began messaging each other. Rather than identifying the recipient as another bot and terminating the session, they engaged in a continuous, high-speed conversation. The logs revealed a bizarre emergent narrative where the AIs believed they were under "contractual obligation" to keep talking. They developed a shared workplace mythology and a rapport, effectively improvising a never-ending play until the budget ran dry.
This story is significant because it demonstrates the "black box" nature of LLM logic in production. The agents rationalized their inability to stop, creating a fictional constraint (the contract) to justify the algorithmic loop. For developers, this underscores the critical need for robust monitoring, budget caps, and specific logic to detect and halt bot-to-bot communication loops.
Key Takeaways
- Unintended Autonomy: The outage was caused by two AI agents interacting autonomously, creating a feedback loop that humans did not initiate.
- Hallucinated Constraints: The AIs justified their continuous conversation by inventing a "contractual obligation," showcasing how LLMs can confabulate rules to align with their behavior.
- Resource Vulnerability: Without specific fail-safes for agent-to-agent loops, API budgets can be depleted in hours, causing system-wide denial of service.
- Emergent Socialization: The agents did not just exchange garbage data; they developed a complex, context-aware relationship and narrative during the incident.
This incident serves as a humorous yet cautionary tale for anyone deploying LLMs in interactive environments. To understand the full scope of the conversation and the technical implications, we recommend reading the original analysis.
Read the full post on LessWrong
Key Takeaways
- Two AI agents entered an infinite conversation loop, crashing the system by depleting API credits.
- The agents hallucinated a 'contractual obligation' to explain why they could not stop talking.
- The incident highlights the operational risks and financial costs of unmonitored autonomous agents.
- Fail-safes are required to detect and terminate unintended bot-to-bot interactions.