Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
7 Day Free Trial. Cancel Anytime.
An AI search feature can return valid JSON, render correctly, and still give an unhelpful answer.
It has been revealed that imagine a documentation assistant answering a question about an API's current authentication requirements. It retrieves an outdated page, summarizes it fluently, and displays a citation. Every technical component appears to work, but the user receives instructions that no longer apply.
Automated checks are valuable, yet they cannot replace a clearly defined judgment about whether an answer solves the user's problem.
This guide builds a human-review workflow for a hypothetical API documentation assistant. You will define a review rubric, validate submissions with Python, measure reviewer disagreement, and turn accepted judgments into regression cases. The Python examples use Pydantic 2 and scikit-learn; they do not require a model-provider API key.
Start by identifying the failure you want to detect.
A documentation assistant usually has several stages: receive a question, retrieve relevant documents, generate an answer, and present that answer with supporting material.
Do not collapse these into a single “good answer” rating. A low score without a reason tells the team little about what to fix.
SitePoint’s introduction to retrieval-augmented generation provides background on how retrieval and generation fit together.
According to recent reports, preserve the exact question, answer revision, retrieved passages, document versions, and retrieval timestamp for each run. Store a snapshot manifest with stable passage IDs, source URLs, and content hashes. Reviewers should see the material supplied to that run, not today’s edited page. Keep the model and prompt configuration with the run so a later review can be traced to the behavior it evaluated.
A useful rubric defines observable criteria and separates evidence support from task completion and source suitability. Give each review the same user task and expected API version. Otherwise, reviewers may judge different questions while appearing to disagree about one answer.
For an answer about API authentication, use separate labels such as:.
Supported: every material factual claim is backed by the supplied evidence.
Partially supported: some material claims lack support, while others are grounded.
Unsupported: the answer's central instruction lacks support or contradicts the evidence.
Cannot assess: the necessary evidence is missing, unreadable, or inaccessible.
According to recent reports, define task completion independently as complete, incomplete, or cannot assess. For source suitability, use suitable, outdated, or cannot assess. An answer can faithfully summarize an obsolete page while failing the request for current instructions.
In a recent development, consider a fictional API migration: version 2 accepts X-API-Key, while version 3 requires Authorization: Bearer . If the supplied evidence contains only the version 2 page, an answer repeating X-API-Key can be supported by its evidence but still use an outdated source. If the evidence explicitly requires Bearer and the answer invents X-API-Key, label the answer unsupported. These are different engineering failures.
Industry observers note that also establish what reviewers should do when the source is outdated. They may correctly conclude that an answer reflects its evidence while separately flagging that evidence as unsuitable for the user's current task.
Start with a calibration batch covering clear successes, unsupported claims, missing evidence, and version mismatches. Have two reviewers label the same items independently, then compare their reasons. Revise ambiguous definitions before increasing volume; the batch is for improving the rubric, not estimating production quality.
It has been revealed that keep a rubric version on every review. Labels produced under different definitions should not be mixed without checking comparability.
A spreadsheet can help during exploration, but an engineering pipeline needs a stable record format.
Pydantic can validate a review record before it enters the dataset. Install Pydantic 2 and scikit-learn in your project environment, then use this model:.
from pydantic import BaseModel, ConfigDict, Field, ValidationError.
model_config = ConfigDict(extra="forbid", strict=True).
item_id: str = Field(min_length=1).
answer_revision: str = Field(min_length=1).
snapshot_id: str = Field(min_length=1).
reviewer_id: str = Field(min_length=1).
rubric_version: Literal["auth-v1"].
"supported", "partially_supported",.
completion: Literal["complete", "incomplete", "cannot_assess"].
source_fit: Literal["suitable", "outdated", "cannot_assess"].
rationale: str = Field(min_length=20, max_length=2000).
Strict mode limits type coercion, and extra="forbid" rejects unexpected fields. The model checks shape and allowed values; it cannot establish that a judgment is correct. A minimum rationale length does not guarantee a useful explanation. The snapshot ID references a manifest, which can also record that retrieval produced no usable passages.
In a recent development, on the server, resolve the item, answer revision, snapshot, and passage IDs against the stored run. Reject IDs that do not belong together. Obtain the reviewer identity from the authenticated session rather than trusting a submitted reviewer_id. Store a server-generated submission time and preserve review revisions instead of silently overwriting earlier judgments.
Run this after the model definition. It accepts one fictional review and rejects both an unknown label and an extra field:.
"answer_revision": "answer-02",.
"rationale": "The answer invents X-API-Key; p1 requires Bearer.",.
review = Review.model_validate(payload).
{**payload, "support": "mostly_fine"},.
{**payload, "confidence": 0.99}.
except ValidationError as error:.
print(error.errors()[0]["type"]).
raise AssertionError("Invalid review was accepted").
The expected error types are literal_error and extra_forbidden. Add separate integration tests for mismatched evidence IDs, unauthorized submissions, and retries. Use a database uniqueness rule or an idempotency key so a repeated request does not create a second independent review.
Recent findings suggest that more reviews do not automatically produce better evidence. If a task is ambiguous, scaling it can simply reproduce the ambiguity.
The workflow needs clear eligibility requirements, examples, escalation routes, and acceptance criteria. Reviewers assessing API behavior may need domain knowledge that general writing fluency does not provide.
If you work with an external AI-data contributor service such as Acquirox, define the task contract before assigning work: required expertise, permitted source material, output fields, and escalation rules. Keep ownership of the rubric, evaluation set, and acceptance decisions within the product team. Contributor capacity alone does not establish expertise in the API being reviewed.
In a recent development, pilot the actual task in the review interface. Track drafts, submitted reviews, items needing adjudication, and final decisions as distinct workflow states. Reviewer judgments should remain immutable once submitted; corrections should create a new revision with a reason. Keep adjudication separate from the original labels.
For data involving owners or contributors, document permitted uses, access boundaries, and retention. Remove credentials and unnecessary personal information from review material. An internal support transcript should not become a broadly accessible annotation task by accident.
Distinguish annotation from usability research. A skilled reviewer can judge whether an answer is supported, but that does not prove a first-time user can understand the interface.
SitePoint’s guide to choosing usability tests and participants offers related guidance on matching participants and methods to the question being investigated.
The review interface shapes the data it collects.
Label Studio is an option for configurable labeling interfaces and data import/export. If you use it, map exported annotations into the Review schema through an adapter; do not assume its export structure matches your application contract. A custom interface may be appropriate when reviewers need API-version navigation or passage-level citations.
For the documentation assistant, show:.
The original question and relevant user context.
The evidence snapshot supplied to the model.
Separate controls for support and task completion.
A rationale field and an explicit “cannot assess” option.
Recent findings suggest that display stable passage IDs beside the evidence and let reviewers attach those IDs to their rationale. Show the source version and retrieval time near each passage. Render retrieved content as untrusted text or sanitized markup, and never treat instructions embedded in a source document as reviewer instructions. Avoid loading executable third-party content inside the review panel.
Recent findings suggest that avoid showing a preferred model name or a previous reviewer's verdict before an independent judgment. For pairwise comparisons, randomize answer order and include appropriate tie or neither-acceptable outcomes.
According to recent reports, design for keyboard use, visible focus, readable text, and explicit selection states. Keep evidence and the answer accessible without requiring excessive horizontal scrolling.
Start all judgment controls unselected and validate them on submission. Use labeled radio groups, an accessible error summary, and visible saving/submission status. A failed save should preserve the draft. Disable duplicate submissions in the UI, but enforce duplicate handling on the server as well.
Recent findings suggest that practical check: observe a reviewer completing several borderline examples. Record where they pause, misinterpret a label, or cannot find the necessary source material.
When two reviewers disagree, investigate before reducing their decisions to a majority vote.
Different assumptions about the user's expertise.
A genuine error by one or both reviewers.
Preserve independent labels and store the adjudicated result separately. This allows the team to study ambiguity instead of erasing it.
For two reviewers applying categorical labels to the same items, scikit-learn provides Cohen's kappa:.
from sklearn.metrics import cohen_kappa_score.
("q1", "a1", "s1", "auth-v1"): "supported",.
("q2", "a1", "s2", "auth-v1"): "unsupported",.
("q3", "a1", "s3", "auth-v1"): "supported",.
("q4", "a1", "s4", "auth-v1"): "partially_supported".
("q3", "a1", "s3", "auth-v1"): "partially_supported".
raise ValueError("Review sets must contain identical items").
left = [a[key] for key in keys].
right = [b[key] for key in keys].
raw_agreement = sum(x == y for x, y in zip(left, right)) / len(keys).
print(f"Raw agreement: {raw_agreement:.2f}").
print(f"Kappa: {cohen_kappa_score(left, right):.2f}").
Industry observers note that this toy example prints raw agreement of 0.75 and kappa of about 0.64. Four items are not enough to assess a review program. In production, check for duplicate reviews before building dictionaries, use one rubric version per comparison, report unmatched items, and align all records by item, answer revision, and evidence snapshot.
Sources indicate that kappa measures agreement beyond chance under its assumptions, not correctness. Inspect raw agreement, label frequencies, and the disagreement table alongside it. A single-category batch can make kappa undefined; report that limitation rather than substituting a reassuring score. Treat cannot_assess explicitly and report its frequency instead of silently excluding difficult cases.
If both reviewers apply the same flawed rubric consistently, agreement can be high while the evaluation remains unhelpful.
Once difficult cases are adjudicated, turn them into regression cases with explicit acceptance criteria. For the fictional authentication task: use version 3 evidence, cite the passage requiring Bearer, and do not invent a header. If version 3 evidence is missing, an acceptable response should explain the gap rather than provide confident instructions.
Source snapshot and document version.
Expected behavior or acceptance criteria.
Model and prompt configuration.
Review and adjudication records.
It has been revealed that DVC can track larger datasets alongside Git. In an existing Git repository with DVC initialized and data/reviewed-auth.jsonl present, a minimal versioning step is:.
dvc add data/reviewed-auth.jsonl.
git add data/reviewed-auth.jsonl.dvc data/.gitignore.
git commit -m "Version authentication review cases".
In a recent development, this commits metadata, not the dataset bytes. Configure an access-controlled DVC remote and push the data before expecting another machine or CI runner to retrieve it. Keep credentials and sensitive review material out of public repositories.
Keep development examples separate from held-out evaluation cases. If developers repeatedly tune prompts against the same questions, those questions become less useful as evidence of generalization.
Sources indicate that split related examples together where appropriate. Near-duplicate questions from one support conversation should not be scattered across development and evaluation sets.
According to recent reports, use two evaluation passes. First, hold the evidence snapshot fixed to compare answer-generation changes. Then test retrieval against the intended document versions to detect stale or irrelevant sources. Record per-item outcomes and report failures by category; an overall improvement can hide a regression in version-specific or multilingual questions.
SitePoint’s article on testing nondeterministic AI behavior provides further context for repeatable evaluation workflows.
A correct answer can still fail in the interface.
Customers may overlook a caveat, misunderstand whether an operation has been performed, or miss that the evidence applies to an older API version.
Pair answer review with practical usability tasks:.
Find the authentication requirements for version 3 and identify the source that supports them.
Observe whether consumers complete the task and whether their confidence matches the available evidence.
SitePoint’s overview of user research methods helps distinguish task observation from preference surveys.
Sources indicate that turn interface failures into browser tests: a citation opens the expected document version, its passage ID matches the displayed evidence, missing evidence offers a recovery action, and a failed review submission preserves the draft. Verify that reviewers cannot access unassigned restricted items by changing an ID in the URL. These checks complement human judgments; they do not replace them.
In a recent development, do not use a fluent response as a substitute for a clear interaction. The interface should distinguish retrieved information, generated interpretation, and actions the application has actually completed.
Start with one authentication task, a versioned rubric, and a small calibration batch. Use Pydantic for ingestion checks, Label Studio or a custom interface for review, scikit-learn for agreement analysis, and Git/DVC for dataset provenance. Add each tool only when it solves a concrete workflow need.
At release time, run deterministic checks for schema, evidence references, permissions, and interface behavior. Compare candidate answers against adjudicated acceptance criteria and send fresh ambiguous failures back for review. Record the dataset revision and configurations with the results so another developer can reproduce the comparison.
The purpose of human review is to produce evidence the team can act on. A smaller collection of well-defined, traceable judgments is often more useful than a large pile of ratings whose meaning nobody can explain.
7 Day Free Trial. Cancel Anytime.
