Khizar Seo
| Data annotation services should hand over seven records alongside the labels: a versioned protocol, a blind gold set score, per class agreement, an error taxonomy, traceability, a rights chain, and rework terms. Score each record from 0 to 2. Under 8 out of 14, the labels are unverifiable and the eval risk sits with you. |
Here is the pattern I keep seeing.
A team ships a training run. Eval comes back soft on one class. Two weeks go into the model. Learning rate, augmentation, checkpoint selection, all of it. The model was fine. Batch 31 had run under a revised guideline that nobody wrote down, and a boundary rule had flipped halfway through delivery.
Nobody could prove it, because the delivery was labels and an invoice. That is the failure this test catches. Not bad annotators. Missing records.
The seven records, in order
- A versioned annotation protocol with edge case rulings
- A blind gold set scored on every batch
- Inter annotator agreement reported per class and per batch
- An error taxonomy that separates systematic bias from random noise
- Traceability records that survive an audit
- A documented rights and consent chain for the source data
- Rework terms and a defect threshold agreed before batch one
Score each one as you go. Two points, one point or zero. The bands sit after record seven.
1. A versioned annotation protocol with edge case rulings
Ask for the written protocol before the first batch ships, carrying a version number and a dated log of every edge case ruling.
Most label disagreement starts in the guidelines, not in the annotator. Two qualified people read the same rule, split on the same ambiguous frame, and both are following instructions. That split lands in your training data as noise with no traceable cause.
A usable protocol names the taxonomy, defines every class boundary, and carries a changelog. Version 1.3 ruled that a partly occluded object still gets a box. Version 1.4 reversed it. Knowing which batches ran under which rule turns a 400,000 item redo into a 4,000 item re run. That is the entire value of a version number, and it is why serious data labeling work starts with a document rather than a chat thread.
| Ask this: Send me the current protocol with its version history, plus three edge case rulings you made in the last month.Red flag answer: We work from your guidelines and clarify anything unclear over chat. |
| Why it matters: You pay once to change a rule and twice to find out which of your data it touched. |
2. A blind gold set scored on every batch
Hold back a gold set the vendor never sees, and score every delivered batch against it.
Vendor accuracy figures are self reported. Treat them as a claim, not a measurement.
The case for measuring it yourself is on the record. Northcutt, Athalye and Mueller audited the test sets of ten of the most used vision, language and audio datasets and estimated an average 3.4 percent label error rate, including 2,916 errors in the ImageNet validation set, roughly 6 percent of it. Those are the benchmarks the field trusts most. Nobody holds your vendor to a higher bar by default, which is why ground truth has to be something you own rather than something you accept.
A few hundred items, labeled by your own team, never shared, gives you an independent read. Score every batch. Not at project close, when the money is spent and the calendar is gone.
| Ask this: We will hold back 400 items and score every batch against them. Is that in scope, and will you accept our number over yours?Red flag answer: Our internal QA already reports accuracy on every batch. |
| Why it matters: A blind gold set turns quality from a vendor claim into a number you own. |
3. Inter annotator agreement reported per class and per batch
Ask for agreement scores broken out by class and by batch, never as one project average.
A project average of 0.86 can hide a class sitting at 0.41. The weak class is usually a rare one, which is often the exact behavior you bought the data to teach.
Krippendorff gives the usual reference points. Rely on data at an alpha of 0.800 or above. Treat 0.667 to 0.800 as tentative. Discard below 0.667. Those bands are stricter than the kappa ranges most vendors quote, which is precisely why buyers should use them.
So ask for the matrix. Class by class, batch by batch, with the item count behind each score. An alpha computed on nine items is not a score, it is a rounding artifact. Any team running real data annotation services produces this as pipeline output, which is why Humyn Labs annotation solutions ship agreement metrics with every delivery rather than on request.
| Ask this: Show me alpha per class for your last delivery, with the item count behind each class.Red flag answer: Overall agreement on that project was above 90 percent. |
| Why it matters: A per class breakdown tells you which part of your taxonomy to fix before you fund another batch. |
4. An error taxonomy that separates systematic bias from random noise
Demand an error breakdown by type, because random noise and systematic bias damage a model in completely different ways.
Random noise averages out as volume grows. Systematic error does not. It teaches the model a wrong rule, consistently, and it survives every increase in dataset size you throw at it.
So 98.2 percent accuracy tells you almost nothing. On a 400,000 item set that figure still leaves 7,200 wrong labels. Scattered at random, you absorb most of it. Clustered in two confusable classes, you have taught the model a rule and you will spend a quarter unteaching it. Same headline number. Completely different outcome.
Ask for counts by error class. Missed object, wrong class, boundary drift, ambiguous case, protocol misread. The shape names the fix. Boundary drift inside one reviewer group is a tooling problem. The same two classes swapped repeatedly is a protocol problem. A cluster of ambiguous cases is a taxonomy problem.
| Ask this: Break your last error report into counts by error class rather than one accuracy figure.Red flag answer: We hit the 98 percent accuracy target in the SLA. |
| Why it matters: You need to know if you bought noise you can absorb or bias you are about to ship. |
5. Traceability records that survive an audit
Every delivered item should carry the protocol version, the qualification tier of the reviewer, a timestamp, and which QC pass cleared it.
This stopped being optional this month. Article 53(1)(d) of the EU AI Act has required providers of general purpose AI models to publish a summary of training content since 2 August 2025. From 2 August 2026 the AI Office can request documents, evaluate models, order corrective action, and issue fines up to the greater of EUR 15 million or 3 percent of worldwide turnover.
You cannot summarize what you never traced. Note the framing though. What you need is proof the work was done to spec at network level, not the identity of the person who did it. Protocol version, qualification tier, QC pass, timestamp. Defensible without exposing anyone, and a natural output of a pipeline that already tracks its own stages. See how the pipeline runs end to end.
| Ask this: Pull the complete provenance record for one item in your most recent delivery and send it to me today.Red flag answer: We can put a summary together for you if the auditors ask. |
| Why it matters: Reconstructing provenance after delivery costs more than requiring it up front, and often it cannot be done at all. |
6. A documented rights and consent chain for the source data
Ask who holds rights to the source data and how consent was captured, before any of it gets labeled.
Annotation inherits every licensing problem underneath it. A clean label on contaminated data is still contaminated, and now it is contaminated and expensive. Humyn Labs treats the chain as part of the deliverable rather than a follow up email, and most frontier buyers now expect the same.
Ask for four things per source. Where the item came from. The license or consent basis. Restrictions on derivative use. How rights holder opt outs under the EU text and data mining exception were handled. If your vendor cannot produce that chain, you cannot complete your own filing, and no amount of label quality fixes it.
| Ask this: For three items I pick at random, show me the rights basis and any restriction on derivative use.Red flag answer: Everything we source is licensed. It is covered in the master agreement. |
| Why it matters: Legal exposure travels with a dataset long after the training run finishes. |
7. Rework terms and a defect threshold agreed before batch one
Set the defect threshold, the measurement method, and who pays for rework before the first batch ships.
Your negotiating position peaks before the first invoice. Use it.
Demand is not softening either, and that shapes the room. Mordor Intelligence puts the data annotation tools market at USD 3.07 billion in 2026, heading to USD 12.42 billion by 2031 at a 32.27 percent compound rate. Vendors have options. So write it down early. Acceptable defect rate per class. The measurement method, meaning your gold set and not their sample. Rework turnaround in working days. And the error rate at which a batch is rejected outright instead of patched. our case studies show how those thresholds behave once volume climbs.
A defect caught at batch three costs one re run. The same defect caught at eval costs the re run, the compute, and the quarter.
| Ask this: What defect rate triggers free rework, who measures it, and what is the turnaround?Red flag answer: We handle any quality issues case by case as they come up. |
| Why it matters: Rework you did not price is rework you absorb, in schedule and in budget. |
Score it. Out of 14.
Two points if the vendor produces the record on request, without a kickoff call and without conditions. One point if they clearly could produce it but have not. Zero if the answer is a reassurance rather than an artifact.
| Score | What it means | What to do next |
|---|---|---|
| 12 to 14 | The records already exist as pipeline output | Proceed. Write all seven into the statement of work anyway, because staff change and memory does not survive them. |
| 8 to 11 | The capability exists, the discipline does not | Negotiate. Make the four weakest records contractual before batch one, with the measurement method named. |
| 4 to 7 | You are buying labels, not verified labels | Run one paid pilot batch, score it against your own gold set, then rescore the vendor before committing volume. |
| 0 to 3 | Nothing in the delivery is independently verifiable | Walk. Every unit of eval risk transfers to you, and none of it is visible until the training run is finished. |
Run the test on every shortlisted vendor in the same week, using the same wording. Comparable answers are the whole point. A vendor who scores 13 and quotes 20 percent higher is usually the cheaper option once you price a single avoidable re run.
What most vendors hand over, and what to require instead
Same seven records. Two very different contracts. This is what separates data annotation services that run a pipeline from those that run a labeling floor.
| Record | Common default | What to require | What it protects |
|---|---|---|---|
| Annotation protocol | Guidelines in a shared doc, edited in place | Versioned document with dated edge case rulings | Traceable rework scope |
| Accuracy measurement | Vendor reported accuracy on their own sample | Blind gold set you control, scored every batch | An honest quality number |
| Agreement scores | One project level figure, if any | Alpha per class and per batch with item counts | Early warning on weak classes |
| Error reporting | A single accuracy percentage | Counts by error class, systematic split from random | The right fix, first time |
| Provenance | A delivery file and an invoice | Protocol version, qualification tier, QC pass, timestamp | Audit readiness under the AI Act |
| Rights and consent | Assurance in an email | Source, license basis, use restrictions, opt out handling | Your legal position after launch |
| Rework | Discussed after the first bad batch | Defect threshold, method, turnaround, rejection point | Budget and schedule |
Where a full pipeline vendor fits
Every record on that list is a pipeline question, not a labeling question. Sourcing, validation, multi layer QC, annotation, and human in the loop review either produce these records as a by product, or the records do not exist and someone rebuilds them later under pressure.
We built humynlabs.ai around that order of operations. Every annotation passes two independent checks, peer review by fellow domain experts and then a centralized QC team, and every label is checked rather than sampled. Deliveries carry metadata, provenance documentation and audit trails, plus quality reports with agreement scores and error breakdowns. Output arrives in COCO JSON, Pascal VOC, YOLO, VoTT CSV, or a custom schema, and evaluation runs through BRIDGE rather than a self scored spreadsheet.
Run the seven record test on us and on everyone else on your list. Comparing data annotation services on records rather than rate cards is the fastest way to find out who is actually running a pipeline. Tell us what you need and you get a proposal scoped to your modality, volume and timeline within 48 hours.
Frequently asked questions
What should data annotation services deliver besides the labels?
Data annotation services should deliver seven records: a versioned protocol, a blind gold set score, per class agreement, an error taxonomy, traceability, a rights and consent chain, and rework terms. Labels without those records cannot be verified, audited, or fixed efficiently when eval goes soft.
How do I evaluate a data annotation vendor before signing?
Run the seven record test in a single call. Ask for each record and score it 0, 1 or 2. A vendor scoring 12 or above already produces these as pipeline output. Below 8, you are buying labels with no independent way to verify them.
What inter annotator agreement score is good enough?
Krippendorff suggests relying on data at an alpha of 0.800 or above, treating 0.667 to 0.800 as tentative, and discarding below 0.667. Apply that per class rather than project wide, since a healthy average routinely hides one class that fails outright.
How much label error is acceptable in training data?
There is no universal number, so set a threshold per class against your eval target. For reference, an audit of ten widely used benchmark test sets estimated an average 3.4 percent label error rate, with roughly 6 percent in the ImageNet validation set.
Do I need provenance records for annotated training data?
Yes if your model reaches the EU market. Article 53(1)(d) of the EU AI Act requires a public summary of training content, and the AI Office gained enforcement powers on 2 August 2026. You cannot summarize sources your vendor never recorded in the first place.
Is sampling based QC enough for a production training run?
Sampling catches common errors and misses rare ones, which is backwards for training data, since rare classes usually carry the behavior you are paying for. Full review costs more per item and far less per rework cycle. Ask which model your vendor runs.
Before you sign anything
The labels are the cheapest thing in the delivery. The seven records around them are what let you prove a training run worked, and fix it fast when it did not. Send your modality, volume and timeline to Talk to Humyn Labs and we will come back with a scoped proposal inside 48 hours.
