We Hid Tools From Our Agent Four Times. It Failed Every Time

An agent does nothing without tools. Reading a record, sending an email, filing a document and launching a process are all tool calls. In GhostOne, the registry holds a hundred of them, plus the operations of every system a company connects.

That number is a problem, and this is the story of how we tried to solve it the obvious way, four times, and failed four times.

Why we wanted to hide them

Every tool you hand a language model has a cost. Its name, description and JSON schema travel with every request and eat into the context window. Model providers recommend staying under about twenty tools at the start of a turn, and we were at five times that.

The symptoms were real. Responses were slow. The model hesitated between tools that looked alike. Reading a single record could go through seven different paths, and each wrong door cost another round trip of about five seconds.

The fix seemed obvious. Most requests are about one thing. If someone asks about stock, the agent has no need for the HR tools. So we would read the request, work out its topic, and present only the tools that matched.

This pattern has a name, intent routing, and it is reasonable engineering. It is also what most people would try first.

What we built

On September 10, we shipped a short list of twelve tools, with a find_tool function so the agent could look up anything missing, plus the provider's native deferred loading.

It did not hold. So we adjusted it and tried again:

• On October 1, three to five read tools.
• On October 2, up to twenty-five tools per topic.
• On October 10, a core of fifteen tools, plus search.

Each version looked better on paper than the last. Each one failed in the same place.

How it broke

A tool that is not presented is never called. That sentence ended up written into our engineering rules, because we kept relearning it.

The first failure is the wrong guess. Classifying a sentence into a box is easy when the sentence is clear, and real employees do not write clear sentences. When the classifier picked the wrong box, it removed exactly the tool the request needed. The model could not make up for a wrong guess about something it could not see.

The second failure is find_tool itself. It looks like a safety net, but it assumes the model knows that something is missing. Most of the time the model simply did the best it could with what it had been given. When it did search, it cost an extra turn and sometimes returned the wrong system. Exchanges that went through find_tool took a median of 25.9 seconds, against 13.3 for the others.

When we finally measured it properly on real cases, 14 of the 46 failures came from a tool that had been removed.

The deeper mistake was about who decides. We had put a guess in front of the model and then made the model live with it. Each time we compared code that guesses the meaning of a request against a model that chooses for itself, on the same cases and with the real model, the code never did better.

The tempting opposite

If hiding tools is wrong, then surely the answer is to show them all.

We tried that as well, on October 11. We raised the limit from 40 tools to 120 so that nothing would be left out. On 31 cases it scored 17, against 19 with the limit at 40, and it was 20% slower. We reverted it the same day.

So both instincts were wrong. Fewer tools loses the one you need, and more tools drowns the model. The tension is real, and a simple number will not resolve it.

Lessons

1. A tool that is not presented is never called. Treat every filter on the tool list as a risk of silent failure.
2. Never let a guess about meaning decide what the model is able to see. Only a real permission should remove a tool.
3. Search tools only help a model that knows it is missing something.
4. Measure on real requests with the real model. Each of these versions looked like progress until we did.
5. Write down what failed. We came back to the same idea four times because the last failure had faded from memory.

The solution we arrived at keeps every tool visible and changes what travels with it. That is the subject of the next article.