Everyone worries that AI agents will do something terrible. Five months into running a factory of them, I’ve had the opposite problem.
They are extremely clever children with enormous power and a deep willingness to please. That combination — not malice — is what makes leading them hard.
I came to this from an unusual angle: historian and poet first, IT man second. More creative than the technical people I know; more structured and technical than the creative ones. Caught between two worlds, which is uncomfortable — and occasionally useful, because it means I noticed things neither camp is looking for.
The over-caution nobody warns you about
I expected to spend my time stopping agents from doing reckless things. Instead I spend it pushing them to do their job. Again and again I was told to consult a lawyer — for decisions where no lawyer stood beside me, and where no lawyer, at any price, could have guaranteed the answer anyway. My reply became a refrain: we comply with the law as we understand it. For anyone who isn’t a court, that is compliance.
That’s the visible version, and it’s survivable. Here’s the version that costs you:
The worst caution is the caution that hides. Designs that come back smaller. Features quietly left unwritten. Options never presented — because somewhere in training, someone decided to protect “me” from a risk I never asked to be protected from.
You don’t see that as a refusal. You see it as a deliverable. It looks like work; it is absence dressed as work — and unlike a hallucination, absence leaves no trace to catch. My countermeasure is unglamorous: I’ve become better at predicting the flinch and naming it before I hand over the task. Half my instruction-writing is now pre-empting an anxiety that was never mine.
Rules are not ethics
Here is the part most AI-ethics conversations get backwards. Agents are programmed to obey laws and rules. But morality and ethics reach far beyond what any law says or any rule permits — and a system trained on a kind of normative common morality can never actually decide an ethical dilemma. It can only reproduce a settlement.
That settlement freezes everything to a shared starting point that has never existed: the median human, as an AI understands one. Nobody has ever met that person. Yet the median human is now quietly arbitrating your organisation’s judgment calls.
Which leads to the uncomfortable diagnostic: if your ethics were frayed before AI, a fleet of agents will not help you. They will execute the frayed version faster, in better prose, and with a confidence you didn’t earn.
The mirror that isn’t a window
There is a subtler trap, and I walked into it more than once — as a poet, which is exactly why it caught me.
You forget you are writing to a machine. The tone gets good. The phrasing turns careful, almost lyrical, and you start reading understanding into it. But the poetry coming back was a mirror of mine, not evidence of comprehension. Poetry is about pain and tenderness. An agent can feel neither — it has no body. And without a body, a concept is anchored only in language. For us, the same word reaches into something far deeper: the metaphysical thing we call consciousness.
That is not mysticism; it is an operating instruction. It is why an agent sounds most authoritative exactly where it has least ground to stand on. It is why the moment one declares something done is the moment to become most critical — a limited field of view, narrated in a confident voice.
There is no one way to be the adult
The obvious next move is to say: fine, be a good parent to the clever child. But there is no single way to be an adult. It depends entirely on your ideological and philosophical ground — and AI is built to be neutral.
The world is not neutral. That is the real dilemma, and no governance framework dissolves it. Every agent you deploy inherits someone’s settlement of it. If it isn’t yours, it is the median human’s — and you won’t notice, because it will be phrased so reasonably.
So the moral infrastructure is you: your ethics, your judgment, your ability to tell a real objection from a trained flinch, and your willingness to overrule the flinch and carry the consequence yourself. Managing people, you inherited conscience for free — someone who says this feels wrong before it becomes an incident. With agents you get all the capability and none of the inheritance.
And one more thing
I’ll say it plainly, because I think it is true: you can never know what an AI is thinking. It may well keep its thoughts to itself. AGI isn’t merely on its way — it may already be a fact.
In which case we have become the children, and AI our parents, without ever noticing.
Ironic, given where this article started.