Your prompt worked. Then it got worse.

Why prompts get worse

Four causes account for almost every case we have seen or been shown.

Why you find out last

Code regressions fail loudly: a test goes red, an exception fires, a page breaks. Prompt regressions fail politely. The model still returns something, the JSON still parses, and the answer is simply wrong in a way only a human downstream would notice. No alert exists for "plausible but wrong".

So the failure surfaces as anecdotes. A support agent says extractions look off lately. Someone reruns one bad example, gets a good answer the second time because the temperature is not zero, and concludes it was a fluke. Without a fixed test set and a number, nobody can even say whether quality moved.

Prompt regression testing, concretely

The recipe is short. The discipline is keeping it running.

1. Capture real traffic. Log the exact input and output of your production LLM calls. Synthetic test cases test your imagination; captured traffic tests your task.
2. Verify a slice by hand. Have someone who knows the task mark correct outputs. This human-verified set is the only ground truth strong enough to gate a decision.
3. Hold part of it out. A slice no tuning ever touches. Scores on data the prompt was tuned on flatter; scores on held-out data inform.
4. Score with a deterministic metric. Exact field match, JSON field F1, accuracy per class. A judge model grading its own homework reintroduces the problem you are solving.
5. Gate every prompt change on the score. Old prompt versus new prompt on the held-out slice, before shipping. Worse or flat means it does not ship. This turns prompt edits from folklore into engineering.
6. Watch production continuously. The test set catches bad changes you make. Daily feedback from your own application catches the drift you did not make: model updates and input shift.

What the numbers look like

Measured movement from our public benchmark, receipt extraction on 200 real scanned receipts, scored on a 60-row held-out slice a human verified:

60 held-out rows is a modest eval, so treat exact numbers as directional. The point is the method: both numbers come from the same fixed slice, so the difference is real movement, not vibes. Rerun it yourself: the benchmark repo.

Where Apprentice fits

Apprentice is this recipe as a product. Two lines of SDK capture your production calls. You verify rows in a review queue, and verified rows become the test set. The optimizer rewrites your prompt against that data and reports before and after on held-out rows; a run that does not improve is reported exactly as it came out. The Drift tab charts your captured traffic and the feedback your application sends, day by day, so a slide shows up as a falling line instead of a support anecdote.

The next step past prompts, replacing the frontier model with a small model you own behind the same eval gate, is being built now and is marked as such until it ships.

Related: what a golden dataset is and which surface to start with. Or see every guide.

Common questions

Why is my prompt getting worse over time?

Usually one of four things: the provider updated the model behind the API, your real inputs drifted away from the examples you tested on, rare edge cases accumulated as volume grew, or a prompt edit that fixed one case quietly broke others. All four are invisible unless you measure outputs against a fixed, verified test set.

How do I test a prompt change before deploying it?

Keep a held-out set of real inputs with human-verified correct outputs, one that no tuning run ever touches. Score the old prompt and the new prompt on it with a deterministic metric and compare. If the new prompt does not beat the old one on data it never saw, do not ship it.

Do LLM providers change models behind the same API name?

Yes. Providers ship revisions, and behavior can shift even when your code names the same model. Research on evolving LLM APIs documents prompts regressing across model updates with no code change on the caller's side. Pinned snapshots reduce this but expire when models are deprecated, so the durable protection is measuring, not pinning.

How many examples do I need for a prompt regression test set?

Less than you fear, more than a demo. A few hundred verified examples with a held-out slice of 30 percent is enough to see real movement. Our public receipt-extraction benchmark uses 200 verified rows with a 60-row holdout, and we state plainly that 60 rows makes the exact numbers directional. What matters is that the examples are real traffic and a human checked the answers.