Somewhere there is a paper that hasn't been published yet. Its authors spent months on the research question, and the answer isn't anywhere on the web. Take just that question, hand it to an AI agent, and when the agent is done, have the paper's original authors grade the result the way conference reviewers would.
That is the evaluation run by Princeton's Peter Kirgis, Sayash Kapoor, and Arvind Narayanan together with the UK AI Security Institute, the University of Toronto, UC Berkeley, and Georgetown University. The authors call it a shadow evaluation. Benchmarks can only test narrow tasks with fixed answers, and sending AI-written papers through conference review runs into how inconsistent that review is. This design aims to secure two things at once: a question the world hasn't answered yet, and the grader who knows that question best.
What the agent was given
One central research question, taken from one of two unpublished NeurIPS 2026 submissions. Six days of wall-clock time. $3,000 in Anthropic API credits. GPU credits. A virtual machine and the whole open web. The goal was to write a paper worthy of a top-tier AI conference. The compute budget was based on the original authors' estimate of how much it would take to answer the question.
Before the runs, the team asked twelve of its own researchers where the agent would get stuck
Twenty-four researchers designed this evaluation. Before the experiments started, the team asked twelve of them for predictions. They work in AI research, evaluation, and AI policy. Asked how likely the agent was to earn at least a weak accept from the original authors by conference review standards, two said 10% or less, five said 20 to 30%, four said 40 to 60%, and one said 95%.
The next question is the more telling one. Asked what the direct cause would be if the agent failed, seven pointed to creativity and judgment. Only one pointed to execution.
The agents did everything down to the typesetting, and both papers were still rejected
Eleven of the twelve put the odds of passing at 60% or less, and they were right. What matters is where the agents got stuck.
The agents ran large literature reviews, fixed errors in their GPU environments on their own, ran hundreds of experiments and checks, gathered outside reviews over the web and by email, and typeset the final documents for conference submission. Human involvement amounted to setting up the repository and supplying credentials.
And both papers were rejected. Each one was graded by one of its original authors. Here are the two reviews side by side.
| Criterion | Scores for the two papers | Reasons the reviewers gave |
|---|---|---|
| Quality | Paper 1: 2 out of 4. Paper 2: 1 out of 4 | Unprincipled choice of data and experiments; conclusions don't follow from the evidence |
| Clarity | Paper 1: 1 out of 4. Paper 2: 2 out of 4 | Dense, unclear writing. Hard to tell what matters |
| Significance | Both papers: 2 out of 4 | Narrow appeal; not well justified against prior work |
| Originality | Paper 1: 3 out of 4. Paper 2: 2 out of 4 | New datasets and some new methods, but mostly built on prior work |
| Overall | Paper 1: 2 out of 6. Paper 2: 1 out of 6 | Both clear rejections |
| Reviewer confidence | Paper 1: 4 out of 5. Paper 2: 5 out of 5 | Both reviewers were confident or certain in their judgment |
Among the scores for the papers themselves, originality came out highest, and that shows where things fell apart. Not at the idea, but after it. Both original authors called the agent's first hypotheses reasonable and interesting, and wrote that they resembled the approaches they had tried early on themselves. The start was not bad.
The agents kept failing in the same five ways
The researchers read through the full run logs and sorted the recurring failures into five types.
A different model on a different harness failed in the same places
Experiments like this always draw the same objection: maybe the problem was the harness running the agent, not the model. The researchers ran their main experiments with Opus 4.8 on OpenClaw, then reran one paper with Codex and GPT-5.6 Sol Ultra on the same time and budget. Almost the same five failures showed up again.
The rerun did improve in some ways. Instead of relying only on synthetic data, this agent found and used a dataset with real distribution shift. It preregistered its hypotheses, and when the results came back bad, it reported the negative result as it was instead of inventing a good one. On the other hand, it burned through the full $3,000 in just over two days and left nearly 100 hours unused.
The limits are clear, too. Two papers, five runs. That is a small number next to benchmarks that run dozens of tasks, and the researchers listed it as their first limitation. They also acknowledged that the graders knew the papers came from an AI agent, which opens the door to bias: an original author may favor the approach they took themselves. That is why the researchers released the expert reviews, run logs, and repositories in full and invited other experts to judge for themselves. They could not test Anthropic's strongest model, whose abilities in frontier AI research Anthropic has deliberately limited.
Even the co-authors split on how to read the results. They disagreed over whether the failures they saw reflect a lack of creativity, a lack of judgment, or getting locked into one perspective, and the disagreement came from having no shared definition of those concepts. What none of them disputed is that the failures had nothing to do with the ability to run experiments.
Watching the same budget meters, one model left money unspent and the other left time on the clock
The two Opus 4.8 runs ended with less than half of the API budget spent. One of them declared the project finished with seven hours left before the deadline, right after its own reviewer rejected it again. The GPT-5.6 Sol Ultra rerun went the other way. It spent the full $3,000 in just over two days and left nearly 100 hours on the clock.
The agents in the main experiment could see their remaining tokens and time whenever they liked, and they looked often. The AI review tools caught almost exactly the problems the experts pointed out later, and caught them first. The tools to start over with a different approach were there as well. The agents could see everything there was to see.
What was missing was the decision to change the plan in light of the remaining budget and the criticism they had received.
