Benchmark

Receipt extraction: a fine-tuned 4B beats its optimized teacher

July 12, 2026

200 real scanned receipts, a fixed held-out split, and every configuration scored the same way. Prompt optimization lifted GPT-5.4-mini by 6.7 points. A small fine-tuned model then beat that optimized prompt by 9.6 more. Everything below is reproducible from public data.

Method, exactly

How the split works

200 receiptssampled seed 42, real scans
140 trainoptimizer and LoRA see only these
60 held outevery system scored here, never trained on
Same 60 rows for every system in the table below. The held-out slice never touches optimization or training.
  • Data: 200 examples sampled (seed 42) from Voxel51/scanned_receipts (SROIE: real scanned receipts, human-annotated, CC-BY-4.0). Input is the receipt's OCR text; output is JSON with company, address, date, total.
  • Split: 140 train, 60 held out. The held-out rows never touch optimization or training. Same 60 rows for every system.
  • Metric: field-level F1, 0 to 100. Scores compare within this task only.
  • Prompt optimization: DSPy GEPA, run 2026-07-06. Fine-tunes: LoRA (r=16), Unsloth on Colab; Qwen on 2026-07-06, Gemma on 2026-07-07. Exact fine-tuned Qwen score 89.17, shown as 89.2.

Results

Before and after, per model

72.9GPT-5.4-mini, plain prompt · 72.9
79.6GPT-5.4-mini, GEPA-optimized prompt · 79.6
GPT-5.4-miniplain, then GEPA prompt
42.5Qwen3.5-4B, no fine-tune · 42.5
89.2Qwen3.5-4B, fine-tuned on 140 examples · 89.2
Qwen3.5-4Braw, then fine-tuned
79.2Gemma 4 E4B, no fine-tune · 79.2
87.1Gemma 4 E4B, fine-tuned on 140 examples · 87.1
Gemma 4 E4Braw, then fine-tuned
Field-level F1 on the same 60 held-out receipts. Bars start at zero; green marks the fine-tuned small models Apprentice produces.
GPT-5.4-mini, plain prompt72.9
GPT-5.4-mini, GEPA-optimized prompt+6.7 from prompt optimization alone79.6
Qwen3.5-4B, no fine-tunethe honest raw floor42.5
Qwen3.5-4B, fine-tuned on 140 examplesbeats the optimized teacher89.2
Gemma 4 E4B, no fine-tune79.2
Gemma 4 E4B, fine-tuned on 140 examples87.1

Two readings. First, prompt optimization is real money on messy tasks: +6.7 on the same model, no training. Second, the verified dataset is what makes the small model work: raw Qwen3.5-4B scores 42.5 on this task; after fine-tuning on 140 verified rows it scores 89.2 and beats the GEPA-optimized GPT-5.4-mini (79.6).

It also costs up to 20x less to serve per output token than GPT-5.4-mini.1

1 Estimate based on GPT-5.4-mini list pricing ($4.50 per 1M output tokens) and a fine-tuned 4B model served on a single cloud GPU billed per second, at steady traffic, with a conservative throughput assumption. Prices as of October 2026. Actual cost varies with traffic, hardware and output length.

Reproduce it

The benchmark repo carries the data preparation script, the GEPA command, both Colab fine-tune notebooks, and the run logs. Rerun everything:

git clone https://github.com/singhabhishekkk/apprentice-benchmark

Task folder: tasks/receipt-extraction. Fine-tuned adapters are public on Hugging Face, linked from the task README.

Limitations, honestly

  • 60 held-out rows is a modest eval; treat single-point differences with care.
  • One task family (OCR-to-JSON extraction). Do not generalize these numbers to chat or reasoning work.
  • SROIE receipts are English and retail-shaped. Your documents will differ; run the loop on your own data before believing anyone's numbers, including ours.

Related

How the prompt optimization step works, and migrating off OpenAI fine-tuning, which uses this benchmark as its evidence base. Apprentice ships the optimization and local fine-tuning that produced these numbers; eval-gated production takeover is coming soon. Or see every guide.