Google Found a Way to Test Gemini Without Seeing the Questions
Imagine a kid who already saw the exam. A perfect score does not mean they learned anything. That is the mess with a lot of AI scores.
The technical name is benchmark contamination. The model already saw the test questions while it was training—copied off GitHub, a blog, a paper, a forum—so the number on the chart is partly memorization. Researchers checked 31 open models in 2024 and found signs of that leak in about half of them. ([paper](https://arxiv.org/abs/2404.18824))
Google DeepMind says it ran a double-blind evaluation of Gemini. Double-blind means Google never saw the questions, and the testers never saw Gemini's model weights (the inner workings, the thing you would need to copy it). They are calling it the first test like this on a closed, top-tier model. They did not publish a new Gemini score. They published the method. ([DeepMind](https://deepmind.google/blog/piloting-the-worlds-first-double-blind-ai-evaluations/), [The New Stack](https://thenewstack.io/google-double-blind-evaluation/))
What they actually built
Usually you get two bad options:
DeepMind's workaround is Confidential Space, part of Google Cloud's confidential computing. In plain English: a locked box in the cloud. Both sides put their secrets in. The test runs inside. Google cannot peek at the questions. The testers cannot peek at the weights. When it is done, the testers get the results and the box gets thrown away.
They ran this with outside groups (Singapore's AI Safety Institute, OpenMined, AVERI, MLCommons) on Gemini Flash Lite, a smaller Gemini, not the flagship. It ran on one NVIDIA H100 Confidential GPU (a chip that keeps data encrypted even while it is being used), with OpenMined's PySyft so the code inside the box cannot phone home and leak the test.
Before anyone sends secrets in, they use remote attestation. That is a signed receipt saying "this box is running the software we agreed on, not a sneaky extra program."
What this is not
It is a pilot. One smaller model. One chip. No public score.
And a locked box does not mean "trust nobody." Google still signs that attestation receipt. You still have to trust Google Cloud and the chip maker are not cheating. MLCommons says you still have to take care of the benchmark itself. Hiding the questions is not the same as a good test.
So if a vendor waves "double-blind" at you, that is not an independent audit. It is a better process than "please don't look." Defaults still lie. Ask who locked the box, and who signed the receipt.
What you should actually do
Do not pick a model off a marketing table. That number is theirs unless you controlled the test.
Ask:
- Could the company see the questions?
- Who attested the environment, and do they work for the company?
- Did anyone test your real work—leaked secrets, bad patches, weird edge cases—not a public quiz the internet has been arguing about for two years? Chris's take: A cleaner SAT score still will not tell you if the assistant dumps a customer file or rubber-stamps a bad patch. Keep running tests on your actual work. Pay attention.
- [DeepMind's post](https://deepmind.google/blog/piloting-the-worlds-first-double-blind-ai-evaluations/)
- [The New Stack writeup](https://thenewstack.io/google-double-blind-evaluation/)
- [2024 leakage study, 31 models](https://arxiv.org/abs/2404.18824)