Human judgment is becoming infrastructure for AI. We want to know when it can be trusted.
Pukot studies how human judgments can be measured, calibrated, combined and routed in AI, research and decision systems.
Q(person)
Q(person, domain, task, time)
A clean score can hide a messy measurement system.
Human-in-the-loop systems routinely turn judgments into labels, rankings and quality scores. Those outputs can mix person effects with task difficulty, domain familiarity, language, rubric quality, context, learning and error.
What are we measuring?
Accuracy, agreement, calibration, consistency, expertise and usefulness are different constructs.
What generalizes?
Strong performance in one task or domain may not transfer to another. Persistent reputation needs evidence of stability and transfer.
What should survive?
Disagreement, uncertainty and context can be signal. Collapsing them too early can erase the reason a judgment matters.
Keep the conditions around the answer.
Pukot is designed so that a judgment can remain connected to the conditions that produced it, where those conditions are relevant to interpretation.
Reliability over time
How much measured performance persists, and how much is noise or learning?
Cross-domain generalization
Where does capability transfer, and where must it remain task-conditional?
Calibration vs accuracy
When confidence is available, does it improve prediction of future correctness?
Rater and task effects
How much observed variation belongs to the person versus the item, rubric or interaction?
Measurement invariance
Do scores retain meaning across language, cohort and presentation conditions?
Disagreement as signal
When does minority judgment predict later value rather than error?
Bring us a measurement problem.
If you are building a scoring, evaluation, annotation, assessment, forecasting or human-in-the-loop system and are unsure whether the resulting measurements are defensible, start with the problem. We can explore a scoped research collaboration, evaluation design, audit, pilot or advisory engagement without assuming the commercial format in advance.
Work on the problem with us.
We are building a distributed expert network around statistical science, forecasting science, prediction markets and research methodology. Adjacent expertise in I-O psychology, psychometrics, quantitative psychology, measurement theory, behavioral science, computational social science and AI evaluation is directly relevant.
We care more about demonstrated work and methodological criticism than titles. Founding Fellows can contribute within a domain before an accountable Domain Lead emerges.
We are building the evidence before making the claim.
Pukot currently uses bounded heuristics and task-specific evidence. We do not claim a validated universal judgment construct, calibrated universal item bank, deployed full CAT/MIRT system, or proven collective intelligence. Those boundaries are part of the research design.