--- title: "Evaluating a multi-turn agent for a national pharmacy chain" sector: "U.S. national pharmacy chain" role: "Principal Experience Architect" duration: "~ 4 months, 2025–2026" pillar: "Evaluation as a design discipline" sub: "An eval-harness build for a refill, transfer, and adherence agent — where \"good\" was harder to define than \"working.\"" --- ## § 01 — context ### The work A national U.S. pharmacy retailer was piloting a multi-turn voice and text agent for three intertwined tasks — refill requests, transfer initiation, and adherence check-ins. The model was already chosen; the integration with the patient-record system was already half-built. What was missing was a way to tell whether the thing was actually *good enough to ship to a regulated audience*. I came in to design the evaluation harness — not as a test suite, but as **the team's evolving definition of what good behavior meant**. The brief was nominally engineering; the real work was design. --- ## § 02 — the constraint that mattered ### "Working" and "good" were not the same thing The team had been measuring task completion. Did the refill go through? Did the transfer succeed? Did the patient confirm? Those numbers were already at 92%. The pilot was, by that measure, working. But when I sat with the recordings, the failure modes I cared about lived *inside* the 92%. The agent had completed the refill — and along the way, mis-pronounced the patient's name, missed a cue that the patient was confused, and offered a generic adherence reminder that contradicted what the doctor had told them last week. Task completion was a true statement and a useless one.
How the team was measuring it Task completion · 92% ✓ refill ✓ transfer ✓ confirm ✓ log ✓ close What was happening inside the 92% name mispronunciation missed confusion cue contradicted clinician advice tone slip on sensitive topic the things task-completion didn't see

Figure 1 — Two views of the same conversation.

## § 03 — approach ### An eval rubric, designed before it was instrumented I treated the eval rubric as a design artifact and ran it through three loops: **Loop one** was language. We spent two weeks *writing* what good looked like in plain English with three pharmacists, two patient-advocacy reps, and a compliance lead. No code. No metrics. Just sentences describing what an empathetic, accurate, safe agent would say and not say. Forty-eight rules. **Loop two** was classification. We hand-labeled ~600 real conversations against the forty-eight rules, then collapsed redundant ones, then split overloaded ones. We ended at twenty-two rules in four categories: *safety, accuracy, comprehension, dignity*. **Loop three** was the harness. *Then*, and only then, the engineering team built the eval harness — half LLM-as-judge, half deterministic checks, with a quarterly human re-grading sample to catch model drift.

~ 40%

Reduction in escalations

after the dignity rule set was deployed

22

Rules in the eval rubric

collapsed from an initial 48

3 ×

Faster regression catches

on prompt and model updates

## § 04 — reflection ### What I'd do differently The two-week language loop felt expensive at the time and was the highest-leverage two weeks of the engagement. If I were starting again I would push for three. I would also start the human re-grading sample *before* the harness was live, not after. By the time we realized the LLM-judge was drifting on the dignity category, we had been shipping against a stale standard for almost a month. The transferable claim, which I've since put in writing more directly: *evals are about deciding what good looks like, and that's a design problem.* If the design hasn't happened, the harness is measuring the wrong thing — confidently.