The last hand-tuned prompt

Software tunes prompts better than I do. I did not want that to be true, so I tested it: I took a prompt of the kind I would write for reading scanned receipts, let an optimizer rewrite it, and scored both on sixty examples the tuning never saw. My kind of prompt scored 72.9 out of 100. The rewritten one scored 84.2. Eleven points, and nobody touched it by hand.

I know the by-hand loop well: paste the prompt into a chat window, change one sentence, run it again, decide the outputs look better. It grows one small fix at a time. The AI feature ships, mostly works, and then the unusual cases arrive, and each fix is one more sentence tested by eye. Many teams running AI on real traffic have some version of this loop. Its cost is hard to see.

Why the guessing loop never ends

The problem was never the person tuning. The problem is that hand-tuning has no memory and no yardstick.

No yardstick: without testing against the same set of examples every time, a new prompt can look better on the cases at hand and still get worse elsewhere. Better stays a feeling, not a number.

No memory: the reasons behind each fix live in one person's head. When they change teams, the next person starts over. And when the AI provider updates its model, which happens without asking you, the whole loop restarts.

Because each fix looks small and reasonable on its own, the missing measurement is easy to overlook.

What replaced it

In the last two years, prompt tuning became something software does. The idea is simple enough to explain in one paragraph.

You collect examples of the task: real inputs, and the outputs you consider correct. The software then writes many versions of the prompt, scores each one, and keeps the version that scores highest. The winner is then scored once more on a slice of your examples held back from the whole run, and that is the number you see. Think of an A/B test, except a machine runs hundreds of rounds of it in minutes, and reads its own failures to decide what to try next.

The output changes kind, not just quality. The guessing loop produces a prompt someone likes. The measured loop produces a prompt plus a score, before and after, on examples the tuning never saw.

Which left me with the question that bothered me more than losing: how does software write a better prompt than the person who understands the task? It took me two ideas to see it.

DSPy, in one idea

Nobody has written machine code by hand for decades. You write what you mean, and a compiler produces the code. DSPy, an open-source framework from Stanford, applies that idea to prompts.

Instead of writing the prompt, you declare the task: what goes in, what should come out, and examples of both. The framework writes the prompt, and rewrites it whenever something better is found. This is the part I resisted longest: it says the wording, the craft I thought I was testing, is not where the value lives.

The prompt stops being a carefully guarded text file and becomes generated output, like compiled code. What your team versions and improves is the left column: the task, the examples, and the grader. The wording takes care of itself.

How GEPA improves a prompt

GEPA is the optimizer inside DSPy that produced my numbers. Its cycle is short enough to draw:

Step two is the idea that changed my mind. Older optimizers only saw a score go up or down, so they mutated prompts mostly blind. GEPA reads the mistakes the way you would: it sees that the model returned the wrong field name, or dropped a currency symbol, and writes the next revision aimed at exactly that. It was not smarter than me about the task. It was faster at the part I skip: actually studying every mistake before rewriting.

The pool in step four is the part people miss. A revision that fixes one kind of mistake often loses on another, so GEPA keeps several versions of the prompt alive at once:

The labels are illustrative. The mechanism is the point: instead of betting on one prompt getting everything right, the pool holds prompts that are each right about something, and later revisions borrow across them. The winner is still picked the boring way, by score rather than by taste, and the score it ships with is measured on examples the tuning never saw.

What GEPA needs from you

Two things, and both are ordinary work rather than research.

Examples with verified outputs. Twenty is enough to start; a few hundred is comfortable. They come from your real traffic, checked by someone who knows the task.

A grader whose feedback says what was wrong, in words. This is the piece practitioners call the metric. A score alone teaches little. "Missing field: invoice_id" teaches exactly the right thing. In my runs, the largest single improvement came not from more examples or a bigger model, but from making the feedback name the mistake. The optimizer is a fast student, and like any student it learns from the quality of the correction.

The numbers

The research behind this is public. Across GEPA's published tasks it scored 6 points higher on average than a heavier training method, up to 20 points on some tasks, while using as little as one thirty-fifth of the test runs. It also beat the previous leading prompt optimizer by more than 10 points.

The run from the opening of this essay, on public data anyone can check:

Scores measure how much of each document the model got right (field-level F1, 0 to 100), on examples kept out of the tuning: 200 real scanned receipts, 60 held for the test. One honest limit: 60 examples is a modest test, so treat the figures as an early result, not a guarantee. The run can be repeated from the public benchmark.

And so the eleven points do not stand alone: the same optimizer ran on every task in the benchmark. The gains vary, which is the honest picture.

Each line is one task, scored before and after optimization on its own held-out examples. Tasks use different data and difficulty, so scores compare within a line, never across lines. The benchmark's fourth task, contract-clause extraction, sits on a far lower scale (34.0 to 36.3) and would flatten this chart; it is in the same repo.

What quietly becomes valuable

Once tuning is measured, the valuable asset moves. It is not the prompt anymore. It is the set of verified examples: real inputs paired with outputs a person confirmed are correct. Prompts become cheap to regenerate. Good examples do not, and they belong to the team in a way no vendor's model ever will.

Three questions worth asking about any AI feature, whoever builds or funds it:

1. When we last changed the prompt, how did the score change? If the answer is a feeling instead of a number, that is the guessing loop.
2. How many verified examples of this task do we own? A useful answer includes the count, what those examples cover, and which ones were kept out of tuning for the final score.
3. What happens when the model provider ships an update? The right answer involves rerunning a measurement, not re-reading outputs until they look fine.

The tools, plainly

DSPy is the open-source framework where most of this lives; GEPA is the optimizer inside it that produced the research numbers above. Both are open source; running them still costs compute or model fees, usually small ones.

Apprentice is the product version of the measured loop, and for honesty: I am building it. It captures your real traffic, helps a person verify the examples, runs the tuning, and reports the before and after. When a run does not improve the score, it says so.

The quickest look is inside the tools you already use: an agent skill for Claude Code, Codex, and Copilot CLI that spots repeatable LLM calls in your code and walks through this loop.

npx skills add singhabhishekkk/apprentice-skill

There are two other ways in, a Python package and a web console, and which one to open first depends on whether the person who can tell a right answer from a wrong one is you. Either way it starts with the set of examples that decides what ships. Or see every guide.

Software tunes prompts better than I do. Once I understood why, I stopped minding: the skill was never the wording, it was owning the task, the examples, and the grader. A hand-tuned prompt without a score is a measurement still waiting to be taken, and a first measurement costs an afternoon, not a quarter.