We gave three AI judges 264 image pairs already scored by humans. The best panel got 18% wrong on the easiest comparisons. Here is why that happened.
Three AI judges looked at 264 pairs of upscaled images that our human reviewers had already scored. On about 70% of those comparisons, the two strongest judges independently picked the same winner, which felt like a reasonable definition of confidence. That agreement was wrong 30% of the time.
Then we filtered down to the easy cases. Not the ambiguous ones, not the hairline calls, but the comparisons where all three human reviewers independently picked the same winner without hesitation. On that subset, the best AI panel still got 18% wrong.
The whole experiment cost about $12 in API credits and an afternoon. It ended a project we had been quietly excited about for weeks.
Why anyone would want this in the first place
If you want to know whether one image upscaler is better than another, you can’t just run a metric on the output. Sharpness scores reward halos. Perceptual metrics disagree with each other. The only thing that reliably answers “which of these two looks better to a person” is asking a person.
So that’s what we do. Two versions of the same image, side by side, in randomized order, with the original available for reference, are 100% crops so you can actually see the texture. A reviewer picks A, B, or a tie across several separate qualities: recovered detail, edge sharpness, and naturalness. Multiply that by 264 image pairs and three reviewers each, and you get 730 human judgments for a single head-to-head test.
It works. It is also slow, expensive, and genuinely tedious. Which is why the idea keeps coming up: let the AI handle the easy pairs and send only the hard ones to humans. If a model could clear the obvious cases, reviewers would spend their attention where it matters.
That idea has a formal name and fifty-five years of literature behind it.
The reject option, and the part everyone gets wrong
In machine learning, a system that is allowed to say “I don’t know, ask someone else” is called selective prediction, or classification with a reject option. C. K. Chow [formalized it in 1970](https://doi.org/10.1109/TIT.1970.1054406), and the modern deep-learning version arrived with [SelectiveNet](https://arxiv.org/abs/1901.09192) at ICML 2019.
The framing that matters is this: such a system has no single accuracy number. It has a curve. On one axis, coverage, meaning what share of cases the machine handles by itself. On the other hand, selective risk, meaning how often it’s wrong on the cases it chose to handle. Push coverage up and risk goes up with it. Any goal of the form “AI handles 60% of the work at under 10% error” is one point on that curve, and the only honest question is whether that point exists.
There’s also a well-developed way to pick the cutoff with an actual guarantee attached. [Conformal risk control](https://arxiv.org/abs/2208.02814), from Angelopoulos, Bates, and colleagues, gives you a threshold with a distribution-free, finite-sample promise that error on the accepted set stays under whatever bound you set. A hand-tuned threshold promises nothing.
So the machinery exists, and the question was what signal to feed it.
The intuitive answer is to ask the model. Have it rate its own confidence, high or low, and defer the low ones. This is the single weakest option available, and it’s worth understanding why, because it’s the design most people reach for first.
Language models are systematically overconfident about their own verbalized certainty, and that overconfidence doesn’t transfer cleanly between tasks. Xiong and colleagues [measured this directly at ICLR 2024](https://arxiv.org/abs/2306.13063), and Anthropic’s [Language Models (Mostly) Know What They Know](https://arxiv.org/abs/2207.05221) maps where the self-report holds and where it falls apart. A model saying “I’m confident” is a token prediction, not an introspection.
The signal that actually works is consistency. Not what the model says about itself, but whether independent checks land in the same place. That’s the logic behind [self-consistency sampling](https://arxiv.org/abs/2203.11171) and behind [semantic entropy](https://www.nature.com/articles/s41586-024-07421-0), the hallucination detector Farquhar and colleagues published in *Nature* in 2024. Run it several ways. See if the answers converge.
For our setup, the cleanest version of consistency was cross-judge agreement. Three models, same protocol, same crops, same randomized order. When two of them independently pick the same winner, that’s evidence. When they split, that’s a case for a human.
What the numbers said
We ran Gemini 3.5 Flash, Gemini 3.1 Pro, and GPT-5.5 through the identical protocol our human reviewers used on the identical images.
The reference wasn’t a simple majority vote. When human reviewers disagree with each other, treating the majority as ground truth throws away real information about how contested a case is. Barbara Plank’s [EMNLP 2022 position paper on human label variation](https://arxiv.org/abs/2211.02570) makes the argument better than we could. So we ran [Dawid-Skene](https://doi.org/10.2307/2346806), a 1979 EM algorithm that estimates each rater’s reliability and produces a soft probability distribution over the true answer rather than a hard label. Every judge was scored against that.
| Judge | Agreement with human labels |
| Gemini 3.5 Flash | 59% |
| Gemini 3.1 Pro | 63% |
| GPT-5.5 | 46% |
Our internal bar for letting an AI do an unsupervised first pass is 85%. Nothing came close. GPT-5.5, the only non-Google model in the lineup, was the worst judge in the room by a wide margin, and that detail matters more than it looks.
Individual accuracy was never the point, though. The gate was open. Here is what happens when you auto-resolve every pair where two judges agree and send the rest to humans:
| Panel | Auto-resolved | Error on auto-resolved |
| Both Gemini models | 70% | 30% |
| Gemini Pro + GPT-5.5 | 56% | 35% |
| All three unanimous | 46% | 32% |
Thirty percent error on the cases the system was most sure about. And no threshold fixes this, which is the part worth sitting with. Conformal risk control can guarantee you a 10% error ceiling, but it delivers that guarantee by shrinking coverage until the promise holds. When the most confident slice of your predictions is already 18 to 20% wrong, the threshold that hits 10% keeps almost nothing. You get your guarantee and an empty auto-queue.
The twist: the diversity hypothesis was right and it didn’t help
Going in, we had a specific hypothesis, and it’s the standard one. Two models from the same family make the same mistakes. Add a judge from a different lab and the errors should decorrelate, so agreement between them means more. This is the reasoning behind [Panel of LLM Evaluators](https://arxiv.org/abs/2404.18796), where Verga and colleagues showed that a jury of smaller, diverse models can beat a single large judge.
We measured it instead of assuming it, because Jitkrittum and colleagues spent a [NeurIPS 2023 paper](https://arxiv.org/abs/2307.02764) mapping the conditions under which confidence-based deferral stops being reliable, and correlated errors are one of them. Correlation between two judges’ error patterns, where lower means more independent:
| Pair | Error correlation |
| Flash × Pro (same family) | +0.48 |
| Pro × GPT-5.5 (cross-family) | +0.30 |
| Flash × GPT-5.5 (cross-family) | +0.47 |
Cross-family really did decorrelate: 0.30 against 0.48. Translated into effective votes, the Kish measure of how many independent opinions you actually have, two Gemini models are worth 1.35 votes, while Gemini Pro plus GPT-5.5 is worth 1.54. The hypothesis held.
And the cross-family panel performed worse anyway. Lower coverage, higher error, worse on both axes at once.
The reason is unglamorous. Decorrelated errors only help if the judge you added is individually competent. GPT-5.5 agreed with humans 46% of the time against Gemini Pro’s 63%. Its agreement doesn’t certify that a call is right; it just drags the panel toward its own mistakes while cutting coverage. Diversity multiplies whatever competence is already there. It doesn’t manufacture any.
Also worth noting: “different brand” is not a synonym for “independent.” Flash and GPT-5.5 came from different labs and correlated at 0.47, nearly the same as two models from the same family. Whatever these models share, it isn’t only training pipelines.
The wall is the task, not the model
The finding that actually explains everything else has nothing to do with which model we picked.
Our human reviewers were unanimous on 48% of the comparisons. On the other 52%, they split 2-to-1. More than half of these image pairs are genuinely contested among trained people looking at the same crops under the same conditions.
That sets a ceiling. Human-to-human agreement on this task runs around 80%, and a judge has no realistic path to being more consistent with people than people are with each other. Our best model hit 63%. The gap between 63 and 80 is a model problem worth attacking. The gap between 80 and 100 is not a model problem at all.
It also reframes what “error” means here. On a pair where reviewers split 2-to-1, calling the AI wrong is partly an artifact of a noisy reference. Those cases should go to humans by design, not because the machine failed but because the question has no clean answer. Which is exactly why the easy-case number was the one that ended the project. On pairs where all three humans agreed, where the answer genuinely is clean, the best gate was still wrong 18% of the time.
There’s a more sophisticated version of this whole idea, called learning to defer, where the system routes a case based on whether the model or the human is more likely to be right on that specific input ([Madras et al., NeurIPS 2018](https://arxiv.org/abs/1711.06664); [Mozannar and Sontag, ICML 2020](https://arxiv.org/abs/2006.01862)). It needs a per-case estimate of human reliability, which three votes and no gold standard don’t give you. It’s the right frame. We can’t populate it yet.
What we changed instead
Humans do the evaluation. That’s the verdict, and it wasn’t the one we wanted.
The AI judge stays in the pipeline as an optional preview. It gives a rough read on which model is leading before the human session starts, and it’s a useful calibration signal when a result looks strange. It never issues a verdict, and its output is kept structurally separate from human statistics so it can’t contaminate them.
The experiment did pay for itself in an unexpected way. We were scoring four qualities per pair, and color fidelity was the one where AI judges agreed with humans least, 43 to 48%. Looking closer, it was also the scale where human reviewers called a tie about 45% of the time and the one that overlapped most with the others. It wasn’t measuring much. We dropped it, which cut about 25% off reviewer time per session with no loss of signal. The remaining three scales are not interchangeable, with a maximum overlap of 75% between any two.
The AI judges also came with well-known failure modes we had to control for anyway, all catalogued in [Judging LLM-as-a-Judge](https://arxiv.org/abs/2306.05685) from NeurIPS 2023: position bias toward whichever image appears first, a preference for longer or more elaborate output, and self-preference when a model evaluates its own family. Randomizing presentation order handles the first. The others are structural.
The question this leaves
We spent $12 to learn we couldn’t do the thing. That was a good trade, and the reason it was a good trade is that we found out before building any of the infrastructure that would have depended on it. A deferral queue, a threshold dashboard, a calibration set. All of it would have worked perfectly and delegated to the wrong 30%.
The uncomfortable implication is broader than one benchmark. Evaluation is the layer everything else rests on. If you can’t trust your evaluator, you can’t trust any ranking built on top of it, including the ones in the model cards you’re reading right now. LLM-as-a-judge is standard practice for scoring model comparisons at this point, so here’s the number worth looking for before you trust one of those scores: how often the human raters disagreed with each other on the same task. If that isn’t reported, the score doesn’t mean what it appears to mean.
So the question isn’t whether models will eventually judge image quality as well as people do. On the contested half of the pairs, “as well as people” means agreeing with a coin flip.
The question is what we do about the tasks where there is no ground truth to converge on, only a distribution of reasonable opinions. We’ve been treating those as measurement problems. They might taste.