The set of examples that decides what ships

What it actually is

Two properties, and both of them do work. The inputs are real, taken from traffic your system already handled, because inputs you write yourself test the cases you thought of. The outputs are human-verified, because the set exists to judge a model and a model cannot be the judge of itself.

That second property is the one teams quietly drop, and it is the one that matters. The useful way to think about a row is to ask what confirmed it. That answer sorts every row you have into three tiers, and each tier can carry a different weight of decision.

The names are not ours. The silver-to-gold promotion pattern, where machine-produced rows wait in a lower tier until a subject-matter expert signs off, is roughly where the evaluation tooling ecosystem landed. What differs between products is how strictly the line is held. Ours is held in code: the training endpoint reads gold rows only, and a silver row cannot reach it no matter how confident anyone is about the file it came in on.

What a verified set is worth

One number makes the case better than any argument. In our public benchmark, receipt extraction on 200 real scanned receipts scored by field-level F1 on a 60-row held-out slice, the same 4B open model scores twice as well with a verified set behind it as without one.

First and last rows are the same model. The difference between them is 140 verified examples. The middle pair is the same model twice with a different prompt, where the verified rows bought +11.3 without touching the model at all. Sixty held-out rows is a modest eval, so treat the exact figures as directional and rerun them yourself from the benchmark repo. The point is not the decimal places. It is that the leverage sat in the data, not in the model choice.

This is the part that surprises people. The instinct when quality is short is to reach for a bigger model. The measured answer on a repeatable task is usually that a small model plus verified examples beats a large model plus a hand-written prompt, and costs a fraction of it.

The features that genuinely do not need one

Almost every article on this subject tells you everyone needs a golden dataset. That is not true, and the exceptions are worth naming because trying to build one for the wrong feature wastes a quarter.

• Anything with no correct answer. Creative writing, brainstorming, open-ended chat, tone rewriting. There is no output a reviewer could mark "correct", so a set of correct outputs cannot exist. These need preference data instead: a human choosing between two outputs. That is a different instrument measuring a different thing, and calling it a golden dataset confuses both.
• Genuinely one-off work. A prompt you will run twice does not justify the review effort. The threshold is repetition, not importance.
• Features still being explored. Before you know what the task is, a verified set locks in a definition of correct that you are about to change. Explore first, then capture.

Everything else needs one. Classification, extraction, routing, moderation, triage, tagging, data cleaning, structured generation: all of them have a right answer, all of them run repeatedly, and for all of them the alternative to a verified set is somebody reading a handful of outputs and forming an impression. That is the loop the industry has taken to calling a vibe check, and it fails in a specific way. It does not miss the outputs that are obviously broken. It misses the ones that are confident, well-formed, parseable, and wrong.

How many rows

The honest answer is that it depends on what the set has to decide, and the range is narrower than most teams fear.

• To move a prompt: a few dozen. Enough to see whether a change helped. A hosted Apprentice run requires at least 20 verified rows, and that floor is deliberately low so a team can get a first real number in an afternoon rather than a quarter.
• To trust a comparison: one to five hundred. Published guidance clusters here. Below about a hundred the confidence interval is wide enough that small differences mean nothing; above five hundred each extra row buys less than it costs to maintain.
• To let something else serve traffic: hundreds, all gold. Hosted training requires at least 500 human-verified gold rows, and the promotion gate reads gold rows only. Silver never counts here. The gap between the two floors, 20 and 500, is the difference between improving something and replacing it.

Those two floors are what the hosted API enforces. The keyless local CLI is looser on purpose so you can try the loop in ten minutes: optimize --local runs on as few as two rows. It will give you a number. Two rows cannot give you a trustworthy one, so read the floors above as the honest guidance rather than as the only thing stopping you.

Coverage matters more than count. A hundred rows spanning every failure mode you know about is worth more than a thousand rows of the same easy case, and it is cheaper to review.

Building one without stopping the world

The reason most teams do not have a golden dataset is not that they disagree with any of the above. It is that building one sounds like a labeling project, and nobody has a quarter for a labeling project. It is not one. Five steps, and the third is the one that makes it possible.

1. Capture real traffic, not test cases. Log the exact input and output of the LLM calls you already run in production. This costs nothing in judgment: no one has to decide anything yet. Invented test cases test your imagination; captured traffic tests your task.
2. Have the right person verify, not the available person. The reviewer has to be someone who can tell a correct output from a plausible one. Often that is the support lead or the analyst downstream, not the engineer who wrote the prompt. The engineer knows what the prompt was supposed to do, which is exactly the wrong knowledge for this job.
3. Verify a few dozen rows, then stop and measure. Do not book a labeling project. Verify enough to run one scored comparison, look at the result, and let the result tell you whether more rows are worth it. Teams that plan a thousand-row effort up front usually deliver zero rows.
4. Hold a slice out and lock it. Split before you tune, not after. The held-out slice is the only part of the set that can honestly answer whether a change helped, and it stops being able to answer the moment anything trains on it.
5. Top it up from the failures. The rows worth adding next are the ones your current setup got wrong. A verified set that grows along its own failure modes gets sharper per row; one that grows randomly mostly gets bigger.

Four ways teams fool themselves

Each of these produces a number. That is what makes them dangerous: a number that feels like measurement but is not.

Where Apprentice fits

Apprentice is the loop above as a product. Two lines of SDK capture your production calls, and captured rows land as raw. Your team verifies rows in a review queue, and a verified row becomes gold. You can invite the person who actually knows the task, a support lead or an analyst, to review without giving them anything else. If you are starting from nothing, Apprentice can generate a starter set from your task description and prompt, and those rows land as raw like everything else: generated data never counts until a human confirms it.

From there, verified rows drive the two things worth doing with them. Prompt optimization scores rewrites against gold and silver and reports the before and after on a held-out slice. Training reads gold only, and the promotion gate reads gold only. A run that does not improve the score is reported exactly as it came out.

Two things we are careful to state plainly. Promotion is currently recorded rather than routed: the gateway that moves live traffic to a promoted model is being built, and every surface says so until it ships. And if you run the CLI locally with your own CSV, those rows are treated as gold because you asserted they are. That escape hatch exists so you can try the loop in ten minutes without an account, and it moves the responsibility for the word "verified" onto you.

Next: which surface to start with, and why prompts get worse over time. Or see every guide.

Common questions