Half of it is true. The AI can guess which half. Probably.

You know the feeling when you find out that a fact you've repeated at dinner parties for ten years isn't really a fact. It's a bit like reaching for a step in the dark that isn't there. Your foot keeps going. Your stomach does something odd.

I got that feeling again this week. Not because of brand-new results, but because two new papers sent me back to a couple from April. One is a counter-audit posted on 24 October. The other is a preprint from 28 October. That's either a sign of a healthy field or a sign I should go to fewer dinner parties. Perhaps both.

1About half of it holds up (and a new challenge to that "half")

First, the April result. A years-long project called SCORE, run through the Center for Open Science with DARPA money, went back and tried to replicate 274 positive claims from 164 social and behavioural science papers published between 2009 and 2018, across 54 journals.^1,2 These were proper attempts, mind you: high-powered, and reviewed in advance, so nobody could fudge it afterwards.¹ The main paper, by Andrew Tyner and a large team, came out in Nature on 1 April 2026 (Tyner et al., Nature).

For 151 of the 274 claims (about 55%), the replication found a significant result in the original direction. Weighted by paper, it was about 49%.¹ So that's about half of a large 2009–2018 social-science sample, not half of everything ever published. But the number that's stuck with me is the shrinkage. Among claims with comparable effect sizes, the median went from r = 0.25 in the originals to r = 0.10 in the replications. That's an 82% drop in shared variance.¹ Even the findings that survived came back looking like they'd been through a hot wash.

Now the new bit, and in fairness, because the honest person inside me insists. A counter-audit preprint posted on 24 October (PsyArXiv) went back through six SCORE replications where a claim had "surprisingly" failed: the original had p < .01 and the replication came out significant in the opposite direction.³ Of the 12 claims tested across those six, the author argues five were miscategorised as failures because of analysis and interpretation problems. Six replications were invalid because of poor data quality and an error in the experiment code.³ It's one preprint, it's narrow, and I haven't seen a formal response yet (as of 3 Nov). I'll keep an eye on it. I'd also note, with some pleasure, that we now have replications of the replications, and that's the ride we're all on now.

What do you do with this if you're not an academic? My own rule of thumb, not a finding: treat any single published behavioural result as a coin flip until someone's repeated it. If you're planning around it, halve the effect size, maybe more. That isn't cynicism. It's just pricing in the base rate.

2The AI that reads the plan and guesses the result

This one's a new preprint from the economist Eva Vivalt, posted on 28 October (arXiv), and it's clever in a sneaky sort of way.

The problem with testing whether AI can "predict" research results is that the model has probably already read the results. So Vivalt gave large language models the pre-registrations and pre-analysis plans, the written plans researchers lodge before running a study, from the AEA RCT Registry and OSF. She used only economics studies whose results were first released in 2026, with models whose knowledge cut-offs came before those papers appeared. That gave 889 treatment effects from 39 papers, and 673 from 33 papers after screening.⁴

The headline numbers need some care, so here they are carefully. On the most recent subset, 339 forecasts from 20 papers released since May 2026, the newest model had a median absolute error of 0.054 standard deviations.⁴ The comparison is a benchmark that predicts each paper using the average of the other papers from the same period, and it had a median error of 0.092.⁴

Vivalt also reports that the AI beat simply forecasting zero effect.⁴ The models' 10th–90th percentile intervals had roughly their stated coverage, and wider intervals did tend to flag the effects that were harder to predict.⁴

The caveats are real, and most of them are hers. It's a single-author preprint. It's a backward-looking test on papers that were already out. It covers economics RCTs with well-documented plans, and she notes those "may be precisely the easiest to forecast".⁴ So this is about well-documented economics trials, not "AI can predict whether your idea works".

Still, put this next to item 1 and something slightly vertiginous happens. About half of a big sample of published findings didn't replicate. Meanwhile a machine reading a careful plan can give a decent guess at the effect size before the results come out. I don't quite know what to do with that yet, if I'm honest. My instinct is the humble version: write your plan down like a pre-registration, ask for an estimate with an interval, and if the interval is wide, pilot it before you bet the budget on it. A wide interval is the model's way of clearing its throat.

3Framing survives the full menu

You probably know the classic Tversky and Kahneman disease problem from 1981.⁷ "200 people will be saved" sounds better than "400 people will die", even though they describe the same outcome. A long-running explanation says this happens partly because the options are described incompletely. Tell people both halves, saved and die, and the effect should go away. In earlier studies, which were mostly small, it typically did (DeKay, 2026).⁵

Then, in 2024, Michael DeKay and Shiyu Dou at Ohio State found a framing effect even when both options were completely described (DeKay & Dou, 2024).⁶ It was a surprising result, and it rested on only 97 people.⁵ In April this year, DeKay replicated it in a large, pre-registered sample of 1,697 US adults matched to census data, with 477 people choosing between completely described options (Psychonomic Bulletin & Review).⁵ A good chunk of the classic framing effect does come from mismatched or incomplete wording. Fine. But with matched, complete descriptions, framing still nudged choices, with an odds ratio of about 1.84.⁵

My favourite part is the paper's epigraph, a quote about having six theories on raising children before having six children and no theories. It's attributed (and probably wrongly) to John Wilmot, "perhaps".⁵ Data, like children, wreck your theories.

So "just give people all the numbers" isn't a cure. If you're putting options in front of a board, the more reliable habit is deliberate reframing: state every option in both gain and loss terms, out loud, before anyone chooses. It feels clunky. It's meant to.

Putting it together

Read side by side, these are sort of one story. What we know about how people decide is shakier than the airport books suggest. The tools for guessing what's real are getting better, at least for well-documented trials. And some of the old effects really are solid, sturdy enough to survive the very fix that was supposed to kill them. Which half is which is now the interesting question.

That's it for issue one. If this goes the way these things usually go, at least half of it will hold up. I'll let you know which half.