A
AI output review
Human review of the answers an AI assistant or agent gives real users, checked against the company's policies and facts. Reviewers sample outputs, escalate wrong or risky answers and report the patterns behind them.
Why it matters: Most AI failures are only visible in live answers. Review turns them into fixes.
Related service →C
Calibration
The process of aligning reviewers with each other and with the client's own decisions before live work. Reviewers score the same items, compare results and discuss disagreements until their judgments match.
Why it matters: Without calibration, two reviewers apply the same guideline differently and the data becomes noisy.
Related service →Content moderation
The review of user posts, comments, reviews, listings, ads and AI-generated content against a platform's policies, deciding what stays up, what comes down and what needs escalation.
Why it matters: Moderation protects users and brands, and is where policy meets real volume.
Related service →CX quality assurance
The review of customer support conversations against a company's own standards to find where agents, bots and policies fall short. Also called customer service QA or contact center QA.
Why it matters: QA shows what customers actually experience, beyond satisfaction scores.
Related service →D
Data annotation
Adding labels, tags or richer markup to raw data such as text, conversations, documents, images or audio, so a model can learn from it or be tested against it. Often used interchangeably with data labeling.
Why it matters: Models learn the patterns in their labels, including the mistakes.
Related service →Data labeling
Assigning categories or values to raw data according to written guidelines, for example tagging a support conversation with its intent, sentiment and outcome.
Why it matters: Consistent labels are the foundation of reliable training and evaluation data.
Related service →E
Edge case
An unusual or ambiguous example that the guidelines do not clearly cover. Edge cases are where reviewers disagree most and where models fail most often.
Why it matters: Collecting and resolving edge cases is how guidelines and models improve.
Related service →Escalation
Passing an item that needs a decision the reviewer cannot or should not make, such as a policy grey area or a high-risk case, to the client's team.
Why it matters: Clear escalation rules keep humans in charge of the hardest calls.
Related service →G
Gold set
A set of items with known correct answers, agreed in advance, mixed into normal work to measure each reviewer's accuracy. Also called golden set or gold items.
Why it matters: Gold items show accuracy per reviewer without reviewing everything twice.
Related service →H
Human-in-the-loop
A setup where people check, correct or approve some of an AI system's outputs, before or after they reach users, and their corrections feed back into the system.
Why it matters: It lets AI handle volume while people handle judgment and risk.
Related service →I
Inter-annotator agreement
A measure of how often different reviewers give the same label to the same item. Common statistics include percent agreement, Cohen's kappa and Krippendorff's alpha.
Why it matters: Low agreement usually means unclear guidelines, not careless reviewers.
Related service →L
LLM evaluation
Checking whether a large language model or AI agent gives correct, safe and on-policy answers, using written scoring guides, side-by-side comparisons, graded agent runs and test sets.
Why it matters: Evaluation tells you whether a change made the model better or worse.
Related service →P
Preference ranking
Asking reviewers to compare two or more model responses to the same prompt and choose the better one, usually with a written reason. Preference data is used to evaluate and improve models.
Why it matters: Side-by-side judgments are often more consistent than absolute scores.
Related service →R
Red teaming
Deliberately trying to make an AI system fail, using realistic and adversarial inputs, so weaknesses are found and fixed before real users hit them.
Why it matters: It finds the failures normal testing misses.
Related service →S
Scoring guide
A written rubric that tells reviewers how to rate an output, for example on correctness, helpfulness, safety and policy, with examples for each score. Also called a rubric.
Why it matters: A clear guide is what makes human scores repeatable.
Related service →T
Test set
A fixed collection of real cases with agreed expected answers that a model or agent must pass. It is rerun before each release to catch regressions.
Why it matters: A test set turns 'does it work?' into a number you can track.
Related service →Trust and safety
The team and practices that keep a platform's users safe and its content within policy: moderation, platform integrity, fraud and abuse review, and review of AI output.
Why it matters: Trust and safety is what lets a platform grow without losing its users' confidence.
Related service →