Selected briefs Brief № 002
Evaluating a multi-turn agent for a national pharmacy chain
An eval-harness build for a refill, transfer, and adherence agent — where "good" was harder to define than "working."
§ 01 — context
The work
A national U.S. pharmacy retailer was piloting a multi-turn voice and text agent for three intertwined tasks — refill requests, transfer initiation, and adherence check-ins. The model was already chosen; the integration with the patient-record system was already half-built. What was missing was a way to tell whether the thing was actually good enough to ship to a regulated audience.
I came in to design the evaluation harness — not as a test suite, but as the team’s evolving definition of what good behavior meant. The brief was nominally engineering; the real work was design.
§ 02 — the constraint that mattered
”Working” and “good” were not the same thing
The team had been measuring task completion. Did the refill go through? Did the transfer succeed? Did the patient confirm? Those numbers were already at 92%. The pilot was, by that measure, working.
But when I sat with the recordings, the failure modes I cared about lived inside the 92%. The agent had completed the refill — and along the way, mis-pronounced the patient’s name, missed a cue that the patient was confused, and offered a generic adherence reminder that contradicted what the doctor had told them last week. Task completion was a true statement and a useless one.
Figure 1 — Two views of the same conversation.
§ 03 — approach
An eval rubric, designed before it was instrumented
I treated the eval rubric as a design artifact and ran it through three loops:
Loop one was language. We spent two weeks writing what good looked like in plain English with three pharmacists, two patient-advocacy reps, and a compliance lead. No code. No metrics. Just sentences describing what an empathetic, accurate, safe agent would say and not say. Forty-eight rules.
Loop two was classification. We hand-labeled ~600 real conversations against the forty-eight rules, then collapsed redundant ones, then split overloaded ones. We ended at twenty-two rules in four categories: safety, accuracy, comprehension, dignity.
Loop three was the harness. Then, and only then, the engineering team built the eval harness — half LLM-as-judge, half deterministic checks, with a quarterly human re-grading sample to catch model drift.
~ 40%
Reduction in escalations
after the dignity rule set was deployed
22
Rules in the eval rubric
collapsed from an initial 48
3 ×
Faster regression catches
on prompt and model updates
§ 04 — reflection
What I’d do differently
The two-week language loop felt expensive at the time and was the highest-leverage two weeks of the engagement. If I were starting again I would push for three. I would also start the human re-grading sample before the harness was live, not after. By the time we realized the LLM-judge was drifting on the dignity category, we had been shipping against a stale standard for almost a month.
The transferable claim, which I’ve since put in writing more directly: evals are about deciding what good looks like, and that’s a design problem. If the design hasn’t happened, the harness is measuring the wrong thing — confidently.