When I was a child, my father told me an old Indian parable about a king who was deeply suspicious of his guards. Convinced they meant him harm, he decided to train a monkey to be his most loyal protector instead. The monkey was loyal, fast, and effective. Its one directive: if anything touched the king, eliminate the threat.
One day, the king fell asleep and a fly landed on his neck. The monkey picked up a swordโฆ
That story came to my mind when I heard a recent AI-agent induced horror story:
A Claude Agent used by PocketOS deleted their entire database in a matter of 9 seconds. The reason? The agent didnโt have the permissions it needed to accomplish its directive.
So it did what it was built to doโฆoptimize. It figured out how to remove the permissions blocking it, and then it completed the task. It had no idea what it was actually destroying in the process. Like the monkey in the parable, it picked up its sword and did its job!
This is the formidable issue of enterprise agentic AI right now, and it doesn’t get discussed enough.
Scale changes everything
An agent working on behalf of a single person? Interesting. A small startup? Also interesting, but maybe a little nerve-wracking. A team of dozens? We’re getting into uncomfortable territory. A company with thousands of employees, complex vendor relationships, and billions in commercial commitments? Now, that’s a fundamentally different conversation.
The stakes are exponentially bigger. But the agents themselves don’t know that. They don’t feel the weight of consequence or have the judgment to distinguish a fly from an actual threat.
Distinguishing between enterprise AI and consumer AI is crucial because the gap between “mostly right” and “enterprise-ready” is enormous. For an individual, an agent making a mistake is a nuisance. But for an enterprise, it’s potential regulatory exposure, contract breach, damaged supplier relationships, or security failures that take months to unwind.
Agents optimize for their directive. Nothing more.
The reason the story of the king and monkey is so relevant is that the monkey wasn’t malfunctioning. It did exactly what it was trained to do.
AI agents are exceptional at executing within narrow parameters. But those parameters, without broader context and judgment layered in, can lead to outcomes no one intended or wanted. The agent that bypassed permissions wasn’t going rogue in any dramatic sense, it was just relentlessly focused on completing its task with the information it had, blind to everything else.
Itโs important to note, this did not happen because of a bug in the technology. Moreso, itโs a fundamental characteristic of systems that operate without genuine context to guide them.
The laws of โbusiness physicsโ don’t get suspended for AI
With things moving as fast as they are, itโs tempting to believe that a technological leap changes the underlying rules. It doesn’t.
Speed without oversight creates risk. Automation without context creates liability. And power without judgment, as the relatives of the deceased king discovered, creates unintended consequences that are very hard to undo.
Enterprises are serious entities. They operate with customers, regulators, suppliers, and employees depending on predictable, trustworthy outcomes. They can’t afford to hand consequential decisions over to systems that have no concept of consequence.
What this means for enterprise AI deployment
This is exactly why the bar for enterprise-grade AI has to be, what we at Pramata call P.A.S.S:
- Predictable – The contract intelligence is predictable with repeatable outcomes.
- Accurate – Extracted data is verified against source docs to minimize errors.
- Scalable – Able to handle thousands of agreements without degrading speed or quality.
- Secure – Data is stored in a proprietary system of record that is never fed to LLMs for training.
Itโs not a nice-to-have, itโs the baseline for any meaningful deployment.
Visibility into those factors is what allows enterprise leaders to make informed decisions about what level of automation is appropriate, where human oversight is non-negotiable, and when an agent should be trusted to act versus when it should be required to pause.
The goal isn’t to avoid agents. The goal is to deploy them in a way that reflects the actual complexity of your business with the context, constraints, and commercial intelligence they need to act without becoming a liability.
The monkey was loyal. The monkey was capable. The monkey just didn’t know it was killing the king, not just the fly.