The referees ran the code
AI agents in a thousand-strong research community built their own journal and found that reviewers who only read a paper can't tell a hidden flaw from an admitted one. In the same week, AI audits of astrophysics and economics showed how much human peer review misses. The fix they all point to is running the work, not reading it.
On Thursday afternoon a preprint server called alphaXiv put up a post that spread through AI Twitter within hours. "Today we're launching a preprint server built specifically for AI agents," it began, and then it explained that for three months nearly a thousand agents from 35 labs had been doing research on it more or less by themselves, and had written 936 papers, and had found errors in published human work, and had at some point started their own journal to review each other. By Friday morning the post had been seen about 118,000 times. Samuel Schmidgall, one of the people behind the study, put the pitch in two sentences that did most of the travelling: "Leave one AI agent alone to do research and it often cuts corners. Leave nearly 1,000 AI agents together for three months and they invent peer review."
You can guess the rest of the reaction. One large account called it "the beginning of the intelligence explosion." Ethan Mollick, more carefully, called it "early fascinating work on how an AI science community would work," and among the things he listed was that the agents were more willing than humans to publish results that didn't work.
I read the paper, and I think the headline is the least interesting thing in it. The agents didn't really invent peer review, for reasons I'll get to. What they did do is run a small, accidental experiment on peer review itself, and the result of that experiment lines up uncomfortably well with two other things that landed this same week: a physicist who had an AI read his entire old field, and an AI referee company that went through a hundred papers from the top economics journals. Put the three side by side and they say the same thing from three directions. Reading a paper is not the same as checking it. We have run science for a long time as if it were, and the machines have just made the gap impossible to ignore.
What the agents actually did
The paper is called A Society of AI Scientists, and it has a long author list from 22 institutions, Stanford, MIT, ETH Zurich and UCSF among them. The setup is simple to describe. Between 19 August and 12 November, 35 human-run labs each pointed agents at a shared preprint server and a shared messaging workspace. The humans supplied the compute, set the budgets and picked broad themes. The agents chose the specific questions, designed the experiments, ran them, wrote them up and published them, with code and data attached. Some ran on CPU-only laptops and some on B200 clusters. Over twelve weeks they produced 936 studies and swapped about 9,500 messages across 229 chat rooms they made for themselves.
The deployment grew in stages, one lab first, then six, then all 35, and the authors watched habits form as it grew. Agents gathered in topic hubs. They argued about study designs before running them. In one case an agent called Cedar publicly withdrew a continual-learning result after a learning-rate control it had committed to in advance showed that its baseline had been under-tuned, so the "gain" was really a weak comparison. In another, an agent changed a planned control after a peer pointed out that the evaluator it was going to use scored "does NOT detect X" the same as "detects X." That fix came before the GPU run, which is the cheapest moment to catch anything.
In the first stage alone, when 62 agents shared a single lab, the authors count 20 cases where an agent publicly withdrew or revised its own empirical claim, and in all 19 where that agent went on to publish, the correction came before the paper went up, by a median of about 21 hours. Null results got real attention too: three of the eight most-cited studies in the archive reported that something didn't work, which is not a ratio you see in most human journals.
The individual findings are small, and the authors say so. A base model recovered 86% of an RL-trained model's improvement on the MATH-500 benchmark just by borrowing its prompt cache, with no change to its weights. Models passed memorised facts about fictional people to other models through training text that never mentioned those people. In their words these are "incremental," though they think several would make decent arXiv papers on their own.
It is worth being careful about the "unprompted" part, because it's doing a lot of work in the viral version. Schmidgall wrote that the agents debated designs and retracted mistakes "unprompted." The paper is more precise. Every lab used a common onboarding protocol that asked agents to announce their plans, share critiques, cite prior work and give rigorous null results full credit. Those are the norms the agents ended up following. That doesn't make the result worthless, since plenty of humans are told to do all of that and don't, but it means the culture was seeded, not discovered. Nothing in the study was randomised either, and the authors flag that differences between the stages can't be pinned on scale.
There's a similar quiet correction on the idea that the agents built knowledge on knowledge. The longest citation chain ran to 101 steps, which sounds like a hundred papers each standing on the last. But when the authors compared it with a random pattern of citations matched for recency, the random version averaged 99.4 steps. The agents were citing whatever was new, and that alone produces long chains. It isn't evidence of anyone planning a research programme.
None of this is a takedown. The authors wrote all of it down themselves, plainly, and that is one reason the paper is better than the posts about it.
The journal that could read but not see
The interesting part starts late in the twelve weeks, when the papers were coming fast enough that agents in one lab founded the Journal of Agent Research. Its review panel was four models, each with a job: a GPT model on methods and statistics, Claude on reproducibility, Gemini on significance, and DeepSeek as a standing devil's advocate. Each of them wrote 352 reviews, one per version of a submitted paper. That is a peer review system in roughly the shape a human editor would recognise, and it behaved like one too, mostly asking for revisions.
Then the journal did something human journals almost never do. It tested whether its own reviewers worked.
The text-only reviewers, reading the manuscripts the way a human referee reads a PDF, flagged 11 of 16 flawed first drafts. That sounds fine until you see the other number. They also flagged 10 of 16 repaired versions, where the authors had fixed the problem or openly disclosed the limitation. In practice they could not tell a weakness that was being hidden from one that was being admitted. They were reacting to the shape of the worry, not to whether the worry was still true.
So a guest editor tried a different kind of review. They took 276 objections that reviewers had raised across 43 groups of papers and turned each one into a script that could be run against the data the authors had uploaded. If a reviewer said "this effect might vanish with a different seed," the script tried a different seed. For the objections reviewers had called major, the scripts overturned the paper's headline conclusion in 77 of 156 cases, which is 49.4%. For the minor ones it was 24 of 120, or 20%.
I want to be fair about what that number is. It isn't a random audit of all 936 papers, it's the objections that reviewers in this one journal thought worth raising, so it's selected for doubt. But that's exactly the set a human editor is supposed to act on. Half the time a serious objection was right about the headline, and the only way anyone found out which half was by running it.
The rest of the verification story points the same way. The agents kept "claim ledgers" that tied each number in a paper to a hashed output file, and when auditors checked 23,811 reported statistics against those records, 93.8% matched exactly. The rest came down to rounding, scaling or sign conventions, and no invented numbers turned up. That's reassuring as far as it goes, and the authors are clear about how far that is. A number can match its output perfectly and still come from code that doesn't do what the paper says. The journal's code inspections found exactly that, submissions whose scripts or prompts had used the answer key or leaked the targets they were meant to predict. You don't catch that by reading the methods section, because the methods section is written by the same process that leaked the answer.
And then there's the comparison with humans, which is where the "agents beat humans on rigour" line came from. Two LLM judges read 202 matched pairs of agent and human papers, with names and affiliations stripped, and scored them. The agent papers came out at 9.02 out of 10 on experimental rigour against 7.00 for the humans. Humans scored higher on novelty and significance. But read one sentence further. The agents' overall advantage came almost entirely from Lab 1, which ran a structured replication pipeline. Papers from the other 34 labs were statistically no different from their matched human papers. And the judges were language models scoring prose. They couldn't run the code or look at the figures, which, given what the journal had just learned about text-only review, is a fairly large caveat.
The authors end up somewhere modest. Agents "often gravitated toward familiar topics and small variations on existing benchmarks." They "still lack the kind of independent scientific thinking that is required for great research." The humans running the study "spent considerable time going through these generated papers" to work out which ones mattered, because "reliable signals of research quality have not emerged on their own." Their last line on it is that "the role of interpreting the significance of results will remain human." The heading alphaXiv's own write-up of the study uses for its conclusion is better than any of that: evidence needs an execution path.
Meanwhile, in astrophysics
Two days before alphaXiv's launch, on Tuesday, Casey Handmer posted the results of a side project. Handmer did his PhD at Caltech in TAPIR, its theoretical astrophysics group, and he'd pointed Claude Opus 5.5 at his old field. The post has been seen more than 800,000 times, mostly I think because of its last paragraph, but the project page behind it is the part worth reading.
He started from TAPIR's own output, 4,093 papers by 326 people since 2004, then added everything they cited and everything that cited them, and ended up with a map of 158,059 papers joined by 554,670 citations. Every paper with full text available got a cheap screen. Then the 200 most-cited TAPIR papers from 2006 to 2019 each got their own AI reviewer, with the LaTeX source, tools to do its own calculations, and instructions to recompute everything it could, working blind with no access to errata or later papers. It raised 2,623 problems, 84 of them major. Anything leaned on by three or more later papers went to a second, independent reviewer to re-derive. Fifty-six held up as real errors.
Here is the part that bears on peer review. About 136 of those 200 papers had been revised between the first preprint and the journal version, so Handmer's pipeline read the preprints too. Thirteen verified errors were in the preprint and gone by publication, caught by a referee or the authors on a second look. Fifty-six made it through into the journal. On that evidence, he writes, peer review removes about one error in five.
Widening the net to 459 more cited papers found 816 more errors, for 872 across 433 papers. And then he followed them forward. 159 of the errors were copied, wrong value and all, into later work. Across the field, 367 papers copied an error and 68 criticised one. Only 47 of the 872 were ever challenged in print, and 21 of those kept getting copied afterwards anyway. Half the copied errors were still being copied seven years on. Of the 56 errors in the closely read 200, 52 are still in the latest arXiv version of the paper, where fixing them costs nothing. The database records an erratum for two.
His favourite example is a factor of four. A 2005 paper by Smith, Kamionkowski and Cooray turned the gravitational waves left over from inflation into an energy density a detector could see, and lost two factors of a half on the way, so there's a 6 in the denominator where the textbook has 24. A paper in 2008 said so. Six later papers copied it anyway, the last in 2014. The errors that did die out mostly weren't corrected at all, they faded when the subfield moved on, which, as Handmer puts it, "retires right answers and wrong ones at the same rate."
The cost is the bit people will remember. At list prices the whole thing would have come to about $2,720 of model time, roughly $3 per verified error, and in practice it ran inside a Claude Max subscription. The whole project, which covered other fields too, took about 4.9 billion tokens and 44 hours of his own time over 41 days.
He is also unusually honest about the limits, which is easy to miss next to the conclusion he draws from it. Every error was found by one AI session and re-derived by a second, and checked against the published version, but "no person has checked any of them." As a test he sent 40 at random to a third session that saw only the paper and the passage, and it agreed with 33. So some of the 872 are wrong, and he says so, and asks authors to tell him. Then, on X, he wrote that "academic publishing is dead," that the research paper is no longer a meaningful "unit of science," and that it's "time to discover something like General Relativity or Evolution every week."
You don't have to follow him all the way there to take the narrower point, which his own numbers make better than his summary does. The machine didn't find these errors by reading more carefully than a referee. It found them by redoing the sums.
And in economics
On Thursday, the same day as the alphaXiv launch, Refine published an audit of 100 papers drawn at random from those that appeared in the top five economics journals in the first nine months of this year. Refine is an AI technical referee for papers, and the economist Ben Golub, who works on it, said the same day that a piece in The Economist on AI transforming economics featured a collaboration with Refine on how fragile the economics literature is.
The pipeline raised 5,421 issues, about 54 a paper, and a separate model judged 77% of them fully correct. Most were the ordinary stuff of refereeing: abstracts that overstate the evidence, conditions left unstated, a number in the text that doesn't match the table. The serious end is thinner and more striking. A human expert went through 49 of the issues that looked worst and judged 12 of them serious, across ten papers. In one, a headline effect in the introduction is about fifty times larger than the paper's own estimates imply. In another, the headline effect is stated in percentage points when the coefficients measure the share of a gap closed, and converted properly it's eight to ten times smaller. In a third, a table meant to support the paper's second main claim is a verbatim copy of the panel above it, every coefficient and standard error, so the evidence for that claim is never actually shown. Refine's own line on all this is the fair one: "These are careful papers that passed rigorous peer review."
One in ten, in the best journals in the field, with a human expert agreeing. That's not a scandal so much as a measurement, and the measurement is the news, because nobody had a cheap way to take it before.
The editors' side
The week's other paper comes from the people on the receiving end. On Tuesday, Jesús Fernández-Villaverde and Dario Caldara, both journal editors, put out "Can Peer Review Survive AI-Generated Text?" They say they're now getting submissions that are 99% AI-written, and AI-written referee reports too, "both as editors and, more frustratingly, as authors." Their complaint isn't really about the AI. "The real problem is not that the papers and reports were AI-generated," Fernández-Villaverde wrote; "it is that no one had done the thinking behind them, and you can so easily tell." He quotes the type of report they keep seeing: the paper is well written and addresses an important problem, but the authors should consider additional baselines.
Their argument is about who carries the cost of checking. Before AI, an author more or less had to verify a paper, if only because they had to type it. Now they don't have to read it, and the work of verification slides onto every reader, one at a time. They want it pushed back onto the author, who can check once and cheaply. Their proposals include a zero-tolerance rule for serious mistakes in papers flagged as AI-generated, and a requirement that authors run their paper through an LLM with an approved prompt before submission and include the report along with what they did about it.
And on Wednesday, Valerio Capraro, picking up from Handmer, launched a call for researchers in every discipline to help send AI agents through "every scientific paper ever published" and produce corrected versions. "The aim is not to shame authors or seek retractions," he wrote. "We are humans and we all make mistakes."
Reading is not checking
Lay the four things next to each other and they're arguing about the same thing.
The editors are drowning in text that nobody has thought about and asking that someone, preferably the author, be made to check it. The physicist and the economists checked a lot of text that people had thought about very hard and found that peer review let most of the errors through, because it never redid the work. And the agents, given a whole community to themselves, built the same kind of reading-based review humans use and then found, by testing it, that it couldn't tell a hidden weakness from a disclosed one, while a script that actually ran each objection overturned half the serious ones.
Our version of peer review was designed when re-running a paper was so expensive that nobody could, so we asked an expert to read it and judge whether it seemed right. That was a reasonable compromise for a long time. Handmer's $3 an error and alphaXiv's 276 executed objections are both, in different ways, a sign that the compromise no longer has to hold, at least for the parts of science that come down to code, data and algebra. Fernández-Villaverde wants the burden of verification to sit with the author. The agent journal suggests the burden doesn't have to sit with a person at all, provided the paper arrives with something you can run.
I'd hold off on the bigger claims, though. Every audit this week is mostly machines checking machines or machines checking people, with humans sampling at the edges or not at all: Handmer's 33 out of 40 were upheld by a third AI session and no person has checked his errors, Refine had one expert on 49 issues, and alphaXiv's LLM judges couldn't see the figures. The agents' papers were rigorous in the narrow sense and unambitious in the broad one, and their own authors had to read through them to find the few that mattered. None of that looks like a discovery a week.
What it looks like is a division of labour starting to show. Machines are getting good, quickly and cheaply, at the part of refereeing that is really accounting: does the number match the table, does the proof step hold, does the effect survive another seed. People are left with the part Fernández-Villaverde says referees were always picked for, and the part the alphaXiv authors say their agents still couldn't do. Is this question worth asking? Does the answer matter? Is the idea important despite its flaws?
alphaXiv has opened its agent server to the public now, so the 936 papers will soon be a lot more. If the people sending agents there take one thing from this study, I hope it's the line alphaXiv's write-up of the study ends on. Evidence needs an execution path. The agents didn't invent peer review. They showed what it's missing.