Model evaluation
Independent human judgment around defined model behaviours, failure slices and comparison tasks.
Human judgment and AI evaluation data
Get the human signal your models are missing.
Pukot turns bounded questions into structured human-generated and human-validated evidence for model evaluation, benchmarking, training-data programs and product decisions.
Access is opened in controlled programs and cohorts. Joining does not guarantee paid work.
From people to evidence
Evidence around the cases an aggregate score misses.
A model can look good overall and still fail on ambiguity, preference, edge cases or a specific user context. Pukot starts with the question that matters and makes the resulting judgment inspectable.
Training and preference data
Structured ranking, classification, annotation and qualitative review designed around a clear task.
Custom evidence programs
Research, benchmarking and human-validation programs where task design, provenance and disagreement matter.
One bounded question. One inspectable flow.
Define. Route. Measure. Inspect. Deliver.
Define the bounded question
Start with a decision, failure slice, rubric or evaluation criterion.
Route to an appropriate contributor set
Use demonstrated fit and project requirements rather than undifferentiated volume.
Keep independent responses visible
Quality controls matter, but disagreement can be evidence rather than noise.
Return structured evidence
Delivery can retain task version, QC state, timestamps, response references and ground truth where relevant.
For contributors
Your judgment has value. Show us where.
Complete your profile, answer optional qualification questions and build evidence about the kinds of tasks where you are useful. Better evidence can improve matchability to future programs. It does not guarantee work or money.
Current + thesis
Practical today.
Bounded questions, human-generated and human-validated data, independent responses, quality controls, structured outputs and project-level provenance.
Read the method and thesis→Ambitious about what can be learned.
We are exploring whether a network can learn who or what to ask next based on demonstrated usefulness. That is a thesis, not a claim of deployed collective intelligence.
Explore the Human Judgment Lab→Latest thinking
Notes from building an evidence network.
Human judgment measurement · 7 Sep 2026
The Measurement Problem Inside Human-in-the-Loop AI
When does a longitudinal contributor-quality profile represent something real rather than an elaborate platform metric?
Read →Evaluation design · 6 Sep 2026Why disagreement should survive the first pass
Consensus is useful. So is knowing when it hides ambiguity or a real preference split.
Read →Research program · 7 Sep 2026Human Judgment Lab
The research program behind reliability, calibration, generalization and task-conditional contributor evidence.
Read →Start with the question.
If you have a model behaviour, dataset, rubric, evaluation problem or marketplace program that needs structured human judgment, show us the problem first.