The Measurement Problem Inside Human-in-the-Loop AI
If the same person makes hundreds of judgments over time, do we learn something stable about that person, or mostly about the tasks we happened to give them?
AI systems increasingly rely on people to label, rank, score, forecast, compare, annotate and adjudicate. The output is often reduced to a clean label or a contributor quality score. That makes the pipeline easier to operate. It can also hide a measurement problem.
When a reviewer is right, wrong, calibrated, inconsistent or unusually useful, what exactly produced that result? Their underlying capability? The task? Domain familiarity? Language? The rubric? Motivation? Context? A lucky guess? A bad question?
We keep encountering this problem while building Pukot. It appears in forecasting, evaluation, annotation, multilingual delivery, question design and contributor routing. The more we try to measure useful human judgment, the less defensible a single context-free quality score looks.
A contributor score is a mixture
A useful first approximation is not a scoring formula. It is a variance problem.
Any persistent score assigned to a person is making an implicit claim that the person component is large enough, stable enough and general enough to justify carrying that score forward.
That claim needs evidence.
Industrial-organizational psychology and psychometrics have dealt with versions of this problem for decades. SIOP's guidance for AI-based employee assessments emphasizes that scores used for decisions should support the intended interpretation, predict relevant outcomes, show appropriate consistency, be evaluated for fairness, and be documented so the scoring process can be verified and audited. Technology changes the instrument. It does not remove the measurement burden.
The scalar score trap
Imagine three contributors.
- Ana scores 0.84 across 200 relatively easy tasks.
- Ben scores 0.73 across 45 difficult specialist tasks.
- Carla disagrees with the majority more often than either of them, but her minority forecasts later resolve correctly at an unusual rate.
Who is the best contributor?
There may be no defensible scalar answer. A ranking can be computed, but computation is not the same thing as measurement.
A single score can collapse accuracy, agreement, calibration, consistency, expertise and downstream usefulness into one number even though those properties are not interchangeable. It can also reward easy task exposure, penalize productive dissent and create false precision around small samples.
The longitudinal opportunity
The interesting possibility is not that every person has one hidden quality number waiting to be discovered. It is that repeated observations may let us estimate where a person is useful, under which conditions, and with what uncertainty.
That changes the research questions.
- Stability. Does performance persist over weeks or months?
- Generalization. Does strong performance transfer across task types or domains?
- Calibration. When people express confidence, does that confidence correspond to realized outcomes?
- Learning. When scores change over time, are we observing noise, practice effects or genuine skill development?
- Disagreement. When is divergence evidence of error, and when is it information that consensus would destroy?
- Measurement invariance. Does the same task measure the same thing across language, cohort and context?
- Incremental validity. Does a longitudinal profile tell us anything useful beyond obvious variables such as experience and prior task exposure?
Question quality is part of contributor measurement
A weak item can make a strong contributor look weak. A vague rubric can manufacture disagreement. A translation can preserve the nominal answer space while changing the cognitive burden. An item that appears difficult before launch may turn out to discriminate almost nothing once humans answer it.
This is why we currently treat Pukot's universal-core difficulty bands as author priors rather than calibrated item parameters. Adaptive routing is bounded and heuristic. Empirical difficulty, discrimination, reliability, fairness and differential item functioning require response data. Full CAT or multidimensional IRT claims would be premature.
The same caution applies to contributor scores. We separate universal reasoning evidence, specialist-domain performance, language accessibility and task-specific outcomes rather than treating them as one permanent rank.
Resolved outcomes give us one unusually clean test
Forecasting provides a useful case because some questions eventually resolve against external ground truth. A forecaster does not become correct because they agreed with another model or with the crowd. Their probability can be scored against what actually happened.
That gives us a criterion that many human-evaluation tasks lack. It also exposes a broader problem: when there is no later ground truth, what should validate the judgment system? Expert adjudication? Inter-rater agreement? Downstream model improvement? Better decisions? Repeated consistency? Different use cases may require different criteria.
What we are instrumenting
Pukot is being built so that the answer does not have to be separated from the conditions that produced it. Where relevant, we can retain task and rubric versions, domain, language rendition, independent response references, timestamps, confidence, QC state, disagreement and later ground truth.
That does not validate a construct. It creates the possibility of testing one.
Our internal design history already contains repeated examples of scoring assumptions breaking when we change the task, domain, question quality or outcome definition. What we do not yet have is enough mature longitudinal human response data to claim that Pukot has discovered a stable latent property called "judgment quality." That is precisely the research problem.
Our research agenda
Over time we want to test whether longitudinal human judgment profiles can support better routing, evaluation and decision systems without pretending that activity itself is ability.
- Estimate reliability and stability over time.
- Measure cross-domain and cross-task generalization.
- Separate calibration from raw accuracy.
- Estimate rater, task, rubric and interaction effects.
- Measure question quality rather than treating it as fixed.
- Test measurement invariance across language and cohort.
- Study human and synthetic judgment interactions.
- Validate scores against independent outcomes where possible.
- Test when disagreement predicts later value.
- Compare multidimensional reputation with scalar reputation.
The question we want challenged
If you were asked to establish that a longitudinal contributor-quality profile represented something real and useful rather than an elaborate platform metric, what evidence would you require before trusting it?
That is the standard we want to build toward.
References and further reading
Society for Industrial and Organizational Psychology: Recommendations for AI-Based Assessments