The Redesign That Broke the Assistant

On October 30, we put a redesign of our product's interface live. Within hours, I had taken it down again. The agent was talking nonsense, as if none of the work had been done.

Nothing about the model had changed. All our tests were green.

A screen change that was not one

The redesign included a batch that placed cards in the conversation from the code. It was filed under screens, and that was the problem, because it also changed the engine. A new instruction told the model that a card had been placed and not to repeat it, on every channel. Only the cards from the last pass were kept, and they disappeared from the history the model could read.

So the agent could no longer see what it had just shown. From its point of view, nothing had happened.

Why the tests were green

Our test suite is large, and it uses a fake model. A fake model proves your tooling and your rules work. It cannot notice that a real model has become confused, because it was never confused.

A change in behaviour can only be seen by the thing that behaves.

What we changed

The rule is now simple. Before any release that touches what the model receives, its tools or the way an exchange ends, we run the bench with the real model on the beta version, and compare.

A pure screen change does not touch the domain. If it does, it has to say so and be measured.

The bench costs real money, so we had to be pragmatic about it: one pass over all the cases, then three passes over the failures only. Measuring has a budget too.

Lessons

1. The label on a release says nothing about what it changes. Check what the model receives.
2. A green test suite with a fake model is no evidence about an agent's behaviour.
3. Keep a comparison on the real model before every switch.
4. Be able to take a release down within minutes.