I built a new search tool for my agents and set up a two-week trial to compare it with the old one. Day one would record a set of searches, day fourteen would run the same set again, and the difference between the two would tell me whether the new tool was worth keeping.
Everything I checked before the start said yes. Every agent's file named the new tool. The routing notes told them to prefer it and to fall back to the old one if they had to. I called the service directly, with the kind of question an agent would ask, and it answered correctly. I'd already said, more than once, that every search in my system now went through it.
Then, on day one, I ran one job the way my jobs really run: not from my own session, but as a scheduled run with only the permissions a scheduled run has. The agent asked for the new tool and was told permission hadn't been granted. The tool was in no allow list anywhere. The other tools built alongside it had been added as they were built; this one had shipped last and never been added. The old tool had been allowed all along.
What made that worse than a missing permission was the fallback. The agents had a documented second choice, and that one was allowed. Every one of them would have asked for the new tool, been refused, done exactly what their instructions said and used the old one, and the output would have come back looking fine. The only trace would have been one line in each transcript, and nobody reads the transcript when the output is fine. For the whole fortnight the trial would have run on the old tool, and on day fourteen the comparison of the new tool against the old would have been the old tool against itself. The two would have matched perfectly, and I'd have been quite pleased with that. There might even have been a post about it.
The fix took a minute: one line in the permissions, and the same job run again came back with results from the new tool. What took longer was working out why every check had passed. The service answered when I called it directly because a direct call doesn't pass through the permission layer at all. The agent files named the tool, but naming a tool and being allowed to call it are decided in different places. Everything I had asked had said yes, and in hindsight I'd only asked the things that couldn't say no.
So the check I now run before any trial is made by the caller, not by me. Not "does the tool work" but "does this agent, started the way it is really started, with the permissions it really has, reach this tool and get an answer from it". An interactive session, a subagent and a scheduled job are three different surfaces with three different sets of permissions, and a pass in one says nothing about the other two.
The enterprise version of this is the pilot. Most evaluations of an AI tool I've seen include my direct call in some form: someone puts a question to the new service and it answers well. Far fewer include a check that the agent, running as it will run in production, under the permissions it actually has, called that service rather than whatever it was allowed to call instead. A sensible fallback is good engineering, and it turns that missing check into a convincing success. A trial that only proves the tool works can succeed without the tool ever being used.
Sources
- First-hand: the search tool trial in the multi-agent system I run at home, September 2026. On day one the new tool was in no allow list and the old tool, the documented fallback, was allowed; found by running one scheduled job the way it really runs, fixed the same day and re-proven by the same run. No tool, server, agent or job names.