argmin gravitas

Transnormalism

Fri, 20 Feb 2026 00:00:00 +0000

Society is unlikely to fall suddenly under the spell of the transhumanist worldview. But it is very possible that we will nibble at biotechnology’s tempting offerings without realizing that they come at a frightful moral cost… Modifying any one of our key characteristics inevitably entails modifying a complex, interlinked package of traits, and we will never be able to anticipate the ultimate outcome… we may unwittingly invite the transhumanists to deface humanity with their genetic bulldozers and psychotropic shopping malls.

— Fukuyama (2004)

the great artisan… made man a creature of indeterminate nature, and… said to him ‘Adam, we give you no fixed place to live, no form forever peculiar to you, no function that is yours alone. According to your desires and judgment, you will have and possess whatever place to live, whatever form, and whatever functions you yourself choose… To you is granted the power of degrading yourself into the lower forms of life, the beasts, and to you is granted the power, contained in your intellect and judgment, to be reborn into the higher forms, the divine.’… To us it was given to be whatever we choose to be, and so that is what we want.

— Pico della Mirandola (1486)

Around 2003 there was a big, rancorous human enhancement debate 3.

The reigning “bioconservatives” predicted that new biotech would lead to moral or political catastrophe, and the loss of human dignity, and maybe wouldn’t even boost welfare; the “transhumanists” 2 argued that nuh uh.

Transhumanism?

The 1998 declaration:

1. ...broadening human potential by overcoming aging, cognitive shortcomings, involuntary suffering, and our confinement to planet Earth.
2. humanity’s potential is still mostly unrealized.
3. humanity faces serious risks, especially from the misuse of new technologies. There are possible realistic scenarios that lead to the loss of most, or even all, of what we hold valuable... not all change is progress.
4. Research effort needs to be invested into understanding these prospects.
5. Reduction of existential risks, and development of means for the preservation of life and health, the alleviation of grave suffering, and the improvement of human foresight and wisdom should be pursued as urgent priorities
6. Policy making ought to be guided by... respecting autonomy and individual rights, and showing solidarity with and concern for the interests and dignity of all people
7. We advocate the well-being of... humans, non-human animals, and any future artificial intellects, modified life forms, or other intelligences
8. We favour allowing individuals wide personal choice over how they enable their lives ["morphological freedom"]

Clearly this is a tame and cuddly ideology compared to stuff like posthumanism and accelerationism, but for some reason the critics settled on attacking all biotech-curious ideologies under the name "transhumanism".

Dickey–Wicker

The big policy move from the debate, banning federal funding for embryonic stem cell research, was actually passed in 1996 under Clinton. Bush's 2001 executive order further restricted NIH funding to pre-existing cell lines, but was rescinded by Obama in 2009.

We routed around some of the damage: in 2006, the invention of induced pluripotency allowed for the creation of (second-rate) stem cells without touching embryos.

Some of the technologies they fought over (cloning, germline editing) haven’t happened at scale yet. But others (GLPs, hormones, hair tech, embryo selection) did, and were rapidly adopted by millions of people who had no interest in either ideology.

And so: a new era of mass chemical enhancement and healthy polypharmacy 1:

(link, link)

(link)

(link - a huge decline but still 1% of US pop! And trans is up to another 1%)

As so often, Fukuyama (quoted above) looks wrong but is not wrong: society indeed did not become transhumanist - that is, not in belief. Few of the biotech users endorse the philosophy of technological transcendence. But society is heading there in deed. Conservative forces (religion, disgust, precaution) were in this case grossly outgunned by the force of sheer desire.

We got, not transhumanism (as deliberate, informed, rational decision to self-consciously go beyond natural human capacity), but surreptitious transhuman behaviour, without the weird philosophy or the new aesthetics. Technology by default, without conscious ideology. Playing god - but using these new, unfathomable powers to… become more normal. So call it transnormalism. 5.

What of the bioconservative prediction of a reckoning for civilisation? So far none arrived. Either

the current techs are not powerful enough yet; or
the corrosive effects (on say fairness, authenticity, self-concept) are lagged or hard to measure; or
mass enhancement is just fine.

Ancient enhancement

A weak reason to think it’s fine is that we’ve been doing it for all of history and prehistory. Enhancement is older than humanity. What’s new is just the size of the enhancements and the biological, internal, invisible nature of the modifications.

Classic:

New:

Old School	New School
Alcohol (prehuman)	Antidepressants
Caffeine (ancient)	Adderall (1996)
Dexedrine (1937)	Adderall (1996)
Glasses (1300)	Intraocular lens (1999)
Makeup (ancient)	Tretinoin (1971), botox (1989), etc
Fluoride (1945)	Hydroxyapatite (1980)
Aspirin (1899)	COX-2 inhibitors (1998)
Antacids (1852)	Proton pump inhibitors (1989)
Statins (1989)	PCSK9 inhibitors (2015)
Dental fillings (ancient)	Osseointegrated (c. 1970s)
Anabolics (CDMT / Turinabol, 1968)	HGH (1985), rhEPO (1993), cardarine (2001), ACP-105 (2009), bimagrumab (2013)
The Pill (1960)	IUDs (2000), subdermals (2006)
Sunscreen (1940)	nanoparticle (2000s)
Alarm clocks (c. 1904)	light alarms (2010s)
Hearing aids (c. 1800)	Cochlear implants (1984)
Inactivated vaccines (1796)	mRNA (2020)

Other tech

I've been pretty focussed on chemical and biochemical enhancement in the above. There's a lot more:

Genetic

Not prevalent yet.

Surgery

Only 38 million cosmetic surgeries a year? Surprising!

It's hard to say what fraction of the 310 million major surgeries are "enhancers" but not many.

Electronics

The above omits a very early and widespread kind: electronic aids and implants.

1936: first wearable hearing aid. Now 300 or 400 million
1958: internal pacemaker. Now around 30? million people.
1977: cochlear implants. Maybe 2 million.

Nonmedical use isn't mainstream yet. The grinders (people who do DIY surgery to implant electronics for nonmedical use) are roughly as strange as they were 15 years ago.

Normal exceptionalism: the escape from morphological freedom

With the exception of the bodybuilding and trans communities 6, we don’t see much weirding at mass scale. We aren’t expressing our morphological freedom to look more different. It seems to me that the result of power over our appearance is not deviance and weirdness but heightened normative normalcy. Bigger biceps, fewer wrinkles, and nerds suddenly getting normative gender presentation. Not many cyborgs.

I’m not fit to do any comparative analysis of the history of cosmesis and beauty standards. This essay is great. But it seems to me that the beauty queens and movie stars of the 1960s no longer look as exceptional as they once did, because cosmetic technology (and image editing ig) has shifted the (top decile of the) distribution upwards so much. I can’t say what all of this is tending towards. What is the intensified platonic ideal of a normal dude?

It’s pretty obvious why tech which gives you options is used, on average, to normalise yourself: most people want to be normal, and the user population is so large now that it simply must include a lot of such people. In 2010 enhancement was a matter for nerds, hackers, and obsessive hobbyists like bodybuilders. But now it’s a much bigger coalition (e.g. 6-12% of Americans on GLP agonists).

Medicalisation and demedicalisation

You don’t need any new or unpopular premises at all to justify enhancement. Transnormalism is what you get from liberalism plus medicalisation. Consumer enhancement has been happening in the form of individual medical decision-making: distributed, highly private (with some actual infosec), and invisible except by us belatedly noticing the aggregate properties of the species changing.

Medicine has been expanding for centuries, especially in the last two decades. This is true in volume (spending) and in domains (new treatments, new powers, new areas brought within the ambit).

In absolute terms (multiplying healthcare share by GWP) we spent $1.85tn in 2000, something north of $7.1tn in 2022 (and more now).

But after being incubated in medicine, enhancement is being taken off the doctors. The thriving grey market (intense cosmetics, “aesthetics” and “med spas” and nootropics) and black market (study drugs, research chemicals, peptides, bootleg hormones) are of unknown size but growing insanely fast.

You are like a little baby

The above technologies are really fairly weak. Retatrutide (2023) is twice as strong as semaglutide (2014), which is twice as strong as liraglutide (2002). At some point someone will work out how to chemically simulate the effect of working out. The nootropics industry is overall a pathetic failure, capped with blunt instruments like not sleeping or flooding the brain with catecholamines. Psychopharmaceuticals are better but not by much and don’t manage sustainbly-better-than-well. We are admittedly really good at things which let people sprint for 6% longer, though at the expense of giving them cancer. We do nearly nothing directly to brains. We have basically nothing for memory enhancement. At the moment we do little with genes, but the rich and unsqueamish are beginning to. All humans are born premature. One might solve sleep. One might solve death.

The conservative concerns might apply to a more-mature science of More. Thanks to transnormalism funding it all we will soon see.

We find sheer humanism to be unsatisfying. It shuts the windows, draws the blinds, and seeks artificial elegance - oblivious of the outer night and the stars. Instead, we boldly go out into darkness and find the superhuman everywhere… the story of evolution of life: its length, its wastefulness, its precariousness, its chanciness, its progressive release of potentiality, its incomprehensibility and ourselves as moments within it… man is transitional and scarcely a beginning.

— Olaf Stapledon (1934)

Actual data

Leo Gao has been running informal n=200 surveys of random Americans. He finds, as I assumed above, that most people are indeed (incoherently) nontranshumanist, with the huge exception of immortality:

> "If you had the option to live forever in perfect health and youth, would you choose to? (Assume you could still change your mind at any time if you ever got bored of it.)” ... 66% of respondents said Yes, with 14% saying No, and 20% saying “Not sure”. As a follow up, it turns out roughly a third of Americans think developing the technology to enable life extension should be a top priority... Since it seemed like overpopulation and inequality were the main things people were worried about, I also asked a version of the question where I stipulated that these things were solved. Surprisingly, this barely shifts people’s opinions, and we get almost exactly the same response! My guess is this is a sign that the real objection is more about the vibes than any specific issue.

> despite being very pro living forever, Americans are much more skeptical of cryonics — even if they could be revived a few decades after their death to live forever thereafter, only 27% are in favor of being preserved, and 46% are opposed (the rest are unsure).

> Space colonization also has pretty lukewarm support, coming in at 37% in favor and 16% opposed

> and cognitive enhancement for all is only a little bit more popular (42% in favor, 19% opposed).

> Also, for some reason, people are really opposed to a hypothetical cheap, painless, and safe arbitrary modification of physical appearance (only 23% in favor, with 37% opposed!). In retrospect, the backlash against Ozempic is a sign, but I was still quite surprised.

> Terraforming other planets so that humans can live on them is also pretty unpopular, coming in at 37% in favor and 16% opposed. Thankfully, for most of these questions, a huge chunk of people are still undecided.

> only 51% of Americans are in favor of literal post-scarcity (complete freedom to work on anything you want, as much as you want, and still enjoy a high quality of life), with 25% opposing. I was so shocked by this result not being 80%+ in favor that I reran a variant of this question with different wording. My original question asked whether the world would be better or worse if everyone had the freedom to work on whatever they want, as long as they want, and still enjoy a high quality of life, and anything we don’t want to do is done for us by robots. I thought maybe that set off some “AI taking jobs bad” instincts; for the new question I took pains to clarify that the stuff is literally conjured out of nowhere with magic and is not taken from anyone else, and got an even worse result (38% support, 34% oppose). This is even more crazy, so I ran a third version on the hypothesis that people don’t like magic, or that not having to work sounded too crazy. This version asked whether it would be good if everyone made 10x more (inflation-adjusted) than they do currently. This polled only somewhat better, with 39% in favor and 19% opposing. I’m still pretty confused what conclusion to draw from this; this is probably worth digging more into.

> Only 14% think that society is currently trending in a positive direction. </div> </div> ##
## See also * [https://www.gleech.org/med](https://www.gleech.org/med) * [https://vectorculture.substack.com/p/not-for-human-consumption](https://vectorculture.substack.com/p/not-for-human-consumption) * [https://bengoldhaber.substack.com/p/no-real-nattys](https://bengoldhaber.substack.com/p/no-real-nattys) * [https://christianangermayer.substack.com/p/the-future-is-enhanced](https://christianangermayer.substack.com/p/the-future-is-enhanced) * [https://web.archive.org/web/20250202101559/https://humanenhancementdrugs.com/](https://web.archive.org/web/20250202101559/https://humanenhancementdrugs.com/) * [https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8519.2005.00437.x](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8519.2005.00437.x) * [https://web.archive.org/web/20210202180452/https://thingsvarious.medium.com/hormone-replacement-therapy-the-only-guide-you-need-2904aa48b7bd](https://web.archive.org/web/20210202180452/https://thingsvarious.medium.com/hormone-replacement-therapy-the-only-guide-you-need-2904aa48b7bd) * [http://bactra.org/Medawar/technology-and-evolution/](http://bactra.org/Medawar/technology-and-evolution/) * [https://www.gleech.org/med](https://www.gleech.org/med) * [https://vectorculture.substack.com/p/not-for-human-consumption](https://vectorculture.substack.com/p/not-for-human-consumption) * [https://bengoldhaber.substack.com/p/no-real-nattys](https://bengoldhaber.substack.com/p/no-real-nattys) * [https://web.archive.org/web/20080613192720/http://www.bioethics.gov/reports/beyondtherapy/chapter1.html](https://web.archive.org/web/20080613192720/http://www.bioethics.gov/reports/beyondtherapy/chapter1.html) * [https://nickbostrom.com/papers/a-history-of-transhumanist-thought/](https://nickbostrom.com/papers/a-history-of-transhumanist-thought/) * [https://global.oup.com/academic/product/natural-born-cyborgs-9780195177510](https://global.oup.com/academic/product/natural-born-cyborgs-9780195177510) * [https://branko2f7.substack.com/p/dr-morell-and-the-patient-a](https://branko2f7.substack.com/p/dr-morell-and-the-patient-a) * [https://doctorzebra.com/prez/z_x35testosterone_g.htm](https://doctorzebra.com/prez/z_x35testosterone_g.htm)

I'm using an idiosyncratic definition of "mass use": >1% of Americans. But that's a leading indicator for the rest of the world following in the end.

A more inclusive term might be "bioprogressives".

with one side backed by an openly religious executive branch.

The social sciences are watching quite closely, but they mostly don't connect any of it to the philosophical project, nor do they project forwards to the coming technologies or preference cascades. They speak narrowly and worry. Their categories are valid and useful as far as they go ("lifestyle drugs", "Image and Performance Enhancing Drugs") but are missing the future, the telos, the limit.

Though one could borrow an emic distinction from trans: that between "dolls" (a trans woman who aims to perfectly converge on normative femininity) and "bricks" (who are not converging, maybe not trying to) and note that a doll who doesn't go too hard is also a transnormalist!

AI in 2025: gestalt

Mon, 08 Dec 2025 00:00:00 +0000

This is the editorial for this year’s “Shallow Review of AI Safety”. (It got long enough to stand alone.)

Epistemic status: subjective impressions plus one new graph plus 300 links.

Huge thanks to Jaeho Lee, Jaime Sevilla, and Lexin Zhou for running lots of tests pro bono and so greatly improving the main analysis.

tl;dr

Informed people disagree about the prospects for LLM AGI – or even just what exactly was achieved this year. But the famous ones with a book to talk at least agree that we’re 2-20 years off (allowing for other paradigms arising). In this piece I stick to arguments rather than reporting who thinks what.
My view: compared to last year, AI is much more impressive but not proportionally more useful. They improved on some things they were explicitly optimised for (coding, vision, OCR, benchmarks), and did not hugely improve on everything else. Progress is thus (still!) consistent with current frontier training bringing more things in-distribution rather than generalising very far.
Pretraining (GPT-4.5, Grok 4, but also counterfactual large runs which weren’t done) disappointed people this year. It’s probably not because it wouldn’t work; it was just ~30 times more efficient to do post-training instead, on the margin. This should change, yet again, soon, if RL scales even worse.
EDIT: See this amazing comment for the hardware reasons behind this, and reasons to think that pretraining will struggle for years.
True frontier capabilities are likely obscured by systematic cost-cutting (distillation for serving to consumers, quantization, low reasoning-token modes, routing to cheap models, etc) and a few unreleased models/modes.
Most benchmarks are weak predictors of even the rank order of models’ capabilities. I distrust ECI, ADeLe, and HCAST the least (see graph below or this notebook). ECI shows a linear improvement, HCAST finds an exponential improvement on greenfield software engineering, and ADeLe shows a previous super-exponential slowing down to what might be linear growth.
The world’s de facto strategy remains “iterative alignment”, optimising outputs with a stack of alignment and control techniques everyone admits are individually weak.
Early claims that reasoning models are safer turned out to be a mixed bag (see below).
We already knew from jailbreaks that current alignment methods were brittle. The great safety discovery of the year is that bad things are correlated in current models. (And on net this is good news.)
Previously I thought that “character training” was a separate and lesser matter than “alignment training”. Now I am not sure.
Welcome to the many new people in AI Safety and Security and Assurance and so on. In the Shallow Review I added a new, sprawling top-level category for one large trend among them, which is to treat the multi-agent lens as primary.
Overall I wish I could tell you some number, the net expected safety change (this year’s improvements in dangerous capabilities and agent performance, minus the alignment-boosting portion of capabilities, minus the cumulative effect of the best actually implemented composition of alignment and control techniques). But I can’t.

Capabilities in 2025

Better, but how much?

— Fraser, riffing off Pueyo

Arguments against 2025 capabilities growth being above-trend

Apparent progress is an unknown mixture of real general capability increase, hidden contamination increase, benchmaxxing (nailing a small set of static examples instead of generalisation) usemaxxing (nailing a small set of narrow tasks with RL instead of deeper generalisation), and human cheating. It’s reasonable to think it’s 20% each, with low confidence. (With a small but growing contribution from AI cheating.)
Discrete capabilities progress seems slower this year than in 2024 (but 2024 was insanely fast). Kudos to this person for registering predictions and so reminding us what really above-trend would have meant concretely. The excellent forecaster Eli was also over-optimistic.
I don’t recommend taking benchmark trends, or even clever composite indices of them, or even clever cognitive science measures too seriously. The adversarial pressure on the measures is intense.
Pretraining didn’t hit a “wall”, but the driver did manoeuvre away from it on encountering an easier detour (RLVR).
- Training runs continued to scale (Llama 3 405B = 4e25, GPT-4.5 ~= 4e26, Grok 4 ~= 3e26) but to less effect.¹ In fact all of these models are dominated by apparently smaller pretraining runs with better post-training.
- 4.5 is actually shut down already; in 2025 it wasn’t worth it to serve any 1T active model or make it into a reasoning model. But this is more to do with inference cost and inference hardware constraints than any quality shortfall or breakdown in scaling laws.
- EDIT: Nesov notes that making use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM on inference chips, as is doing RL on bigger models. He expects it won’t be possible to do the next huge pretraining jump (to ~30T active) until ~2029.
- It would work, probably, if we had the data and HBM and spent the next $10bn, it’s just too expensive to bother with at the moment compared to:
RLVR scaling and inference scaling (or “reasoning” as we’re calling it), which kept things going instead. This boils down to spending more on RL so the resulting model can productively spend more tokens.
- But the feared / hoped-for generalisation from {training LLMs with RL on tasks with a verifier} to performing on tasks without one remains unclear even after two years of trying.² Grok 4 was apparently a major test of scaling RLVR training.³ It gets excellent benchmark results and the distilled versions are actually being used at scale. But imo it is the most jagged of all models.
- This rate of scaling-up cannot be sustained: RL is famously inefficient. Compared to SFT, it “reduces the amount of information a model can learn per hour of training by a factor of 1,000 to 1,000,000”. The per-token intelligence is up but not by much.
- There is a deflationary theory of RLVR, that it’s capped by pretraining capability and thus just about easier elicitation and better pass@1. But even if that’s right this isn’t saying much!
- RLVR is heavy fiddly R&D you need to learn by doing; better to learn it on smaller models with 10% of the cost.
- An obvious thing we can infer: the labs don’t have the resources to scale both at the same time. To keep the money jet burning, they have to post.
By late 2025, the obsolete modal “AI 2027” scenario described the beginning of a divergence between the lead lab and the runner-up frontier labs.⁴ This is because the leader’s superior ability to generate or acquire new training data and algorithm ideas was supposed to compound and widen their lead. Instead, we see the erstwhile leader OpenAI and some others clustering around the same level, which is weak evidence that synthetic data and AI-AI R&D aren’t there yet. Anthropic are making large claims about Opus 4.5’s capabilities, so maybe this will arrive on time next year.
For the first time there are now many examples of LLMs helping with actual research mathematics. But if you look closely it’s all still in-distribution in the broad sense: new implications of existing facts and techniques. (I don’t mean to demean this; probably most mathematics fits this spec.)
Extremely mixed evidence on the trend in the hallucination rate.
Companies make claims about their one-million- or ten-million-token effective context windows, but I don’t believe it.
In lieu of trying the agents for serious work yourself, you could at least look at the highlights of the gullible and precompetent AIs in the AI Village.

Here are the current biggest limits to LLMs, as polled in Heitmann et al:

Arguments for 2025 capabilities growth being above-trend

We now have measures which are a bit more like AGI metrics than dumb single-task static benchmarks are. What do they say?

Difficulty-weighted benchmarks: Epoch Capabilities Index.
- Interpretation: GPT-2 to GPT-3 was (very roughly) a 20-40 point jump.
Cognitive abilities: ADeLe.⁵
- Interpretation: level L is the capability held by 1 in 10^L humans on Earth. GPT-2 to GPT-3 was a 0.6 point jump.
Software agency: HCAST time horizon, the ability to handle larger-scale well-specified greenfield software tasks.
- Interpretation: the absolute values are less important than the implied exponential (a 7 month doubling time).

So: is the rate of change in 2025 (shaded) holding up compared to past jumps?:

Ignoring the (nonrobust)⁶ ECI GPT-2 rate, we can say yes: 2025 is fast, as fast as ever or more.

Even though these are the best we have, we can’t defer to these numbers.⁷ What else is there?

In May they passed some threshold and I finally started using LLMs for actual tasks. For me this is mostly due to the search agents replacing a degraded Google search. I’m not the only one who flipped this year. This hasty poll is worth more to me than any benchmark:

Or if you prefer a formal study (n=2,430 researchers):

On actual adoption and actual real-world automation:
- Based on self-reports, the St Louis Fed thinks that “Between 1 and 7% of all work hours are currently assisted by generative AI, and respondents report time savings equivalent to 1.4% of total work hours… across all workers (including non-users… Our estimated aggregate productivity gain from genAI (1.2%)”. That’s model-based, using year-old data, and naively assuming that the AI outputs are of equal quality. Not strong.
- The unfairly-derided METR study on Cursor and Sonnet 3.7 showed a productivity decrease among experienced devs with (mostly) <50 hours of practice using AI. Ignoring that headline result, the evergreen part here is that even skilled people turn out to be terrible at predicting how much AI actually helps them.
True frontier capabilities are likely obscured by systematic cost-cutting (distillation for serving to consumers, quantization, low reasoning-token modes, routing to cheap models, etc). Open models show you can now get good performance with <50B active parameters, maybe a sixth of what GPT-4 used.⁸
- GPT-4.5 was killed off after 3 months, presumably for inference cost reasons. But it was markedly lower in hallucinations and nine months later it’s still top-5 on LMArena. I bet it’s very useful internally, for instance in making the later iterations of 4o less terrible.
- See for instance the unreleased deep-fried multi-threaded “experimental reasoning model” which won at IMO, ICPC, and IOI while respecting the human time cap (e.g. 9 hours of clock time for inference). The OpenAI one is supposedly just an LLM with extra RL. They probably cost an insane amount to run, but for our purposes this is fine: we want the capability ceiling rather than the productisable ceiling. Maybe the first time that the frontier model has gone unreleased for 5 months?
LLM councils and Generate-Verify divide-and-conquer setups are much more powerful than single models, and are rarely ever reported.
Is it “the Year of Agents” (automation of e.g. browser tasks for the mass market)? Coding agents, yes. Search agents, yes. Other agents, not much (but obviously progress).
We’re still picking up various basic unhobbling tricks like “think before your next tool call”.
Character-level work is still occasionally problematic but nothing like last year.
GPT-5 costs a quarter of what 4o cost last year (per-token; it often uses far more than 4x the tokens). (The Chinese models are nominally a few times cheaper still, but are not cheaper in intelligence per dollar.)
People have been using competition mathematics as a hard benchmark for years, but will have to stop because it’s solved. As so often with evals called ahead of time, this means less than we thought it would; competition maths is surprisingly low-dimensional and so interpolable. Still, they jumped (pass@1) from 4% to 12% on FrontierMath Tier 4 and there are plenty of hour-to-week interactive speedups in research maths.
Recursive self-improvement: Deepmind threw AlphaEvolve (a pipeline of LLMs running an evolutionary search) at pretraining. They claim the JAX kernels it wrote reduced Gemini’s training time by 1%.
Extraordinary claims about Opus 4.5 being 100th percentile on Anthropic’s hardest hiring coding test, etc.
From May, the companies started saying for the first time that their models have dangerous capabilities.

One way of reconciling this mixed evidence is if things are going narrow, going dark, or going over our head. That is, if the real capabilities race narrowed to automated AI R&D specifically, most users and evaluators wouldn’t notice (especially if there are unreleased internal models or unreleased modes of released models). You’d see improved coding and not much else.

Or, another way: maybe 2025 was the year of increased jaggedness, trading off some capabilities against others. Maybe the RL made them much better at maths and instruction-following, but also sneaky, narrow, less secure (in the sense of emotional insecurity).

(You were about to nod sagely and let me get away without checking, but the ADeLe work also lets us just see if the jaggedness is changing.)

It is!

– Roger Grosse

Evals crawling towards ecological validity

Item response theory (Rausch 1960) is finally showing up in ML. This lets us put benchmarks on a common scale and actually estimate latent capabilities.
- ADeLE is my favourite. It’s a fully-automated and explains the abilities a benchmark is actually measuring, gives you an interpretable ability profile for an AI, and predicts OOD performance on new task instances better than embedding and finetunes (AUROC=0.8). Pre-dates HCAST task horizon, and as a special case (“VO”). They throw in a guessability control as well!
- These guys use it to estimate latent model ability, and show it’s way more robust across test sets than the average scores everyone uses. They also step towards automating adaptive testing: they finetune an LLM to generate tasks at the specified difficulty level.
- Epoch bundled 39 benchmarks together, weighting them by latent difficulty, and thus obsoleted the currently dominant Artificial Analysis index, which is unweighted.
- HCAST reinvents and approximates some of the same ideas. Come on METR!
Eleuther did the first public study of composing the many test-time interventions together. FAR and AISI also made a tiny step towards an open source defence pipeline, to use as a proxy for the closed compositional pipelines we actually care about.
Just for cost reasons, the default form of evals is a bit malign: it tests full replacement of humans. This is then a sort of incentive to develop in that direction rather than to promote collaboration. Two papers lay out why it’s thus time to spend on human evals.
The first paper using RL agents to attack fully-defended LLMs.
We have started to study propensity as well as capability. This is even harder.
This newsletter is essential.
The time given for pre-release testing is down, sometimes to one week.
No public pre-deployment testing by AISI between o1 and Gemini 3. Gemini 2.5 seems to have had no third-party pre-deployment tests.
A bunch of encouraging collaborations:
- The CoT Faithfulness Defence League; OpenAI testing Claude and Anthropic testing GPT; OpenAI/Anthropic/Deepmind; Apollo and OpenAI; AISI/CAISI/OpenAI; AISI/CAISI/Anthropic; AISI/Redwood; Redwood/Anthropic; METR/OpenAI; METR/Anthropic; Eval Forum; Various countries.
- Scale AI appears to be offering big companies pre-deployment testing for free? But the Meta investment presumably spoiled this.
- Some details about OAI external testing here, including some of the legal constraints verbatim.

Safety in 2025

Are reasoning models safer than the old kind?

Well, o3 and Sonnet 3.7 were pretty rough, lying and cheating at greatly increased rates. Looking instead at GPT-5 and Opus 4.5:

Much more monitorable via the long and more-faithful CoT (–> all risks down)
- “post-hoc rationalization… GPT-4o-mini (13%) and Haiku 3.5 (7%). While frontier models are more faithful, especially thinking ones, none are entirely faithful: Gemini 2.5 Flash (2.17%), ChatGPT-4o (0.49%), DeepSeek R1 (0.37%), Gemini 2.5 Pro (0.14%), and Sonnet 3.7 with thinking (0.04%)”
Much better at following instructions (–> accident risk down).
Much more likely to refuse malicious requests, and topic “harmlessness”⁹ is up 75% (–> misuse risk down)
Ambiguous evidence on jailbreaking (misuse risk). Even if they’re less breakable there are still plenty of 90%-effective attacks on them.
Much less sycophantic (cogsec risk down)
To get HHH Claude to hack a bank, you need to hide the nature of the task, lie to it about this being an authorised red team, and then still break down your malicious task into many, many little individually-innocuous chunks. You thus can’t get it to do anything that needs full context like strategising.
Anthropic’s own tests look bad in January 2025 and great in December:

But then

More autonomy (obviously agentic risk up)
More reward hacking (and so worse estimates of capability and risk). Note that reward hacking is not a silly or isolated or self-limiting kind of misalignment, owing perhaps to post-training inadvertently creating connections between it and the others.
Huge spike in eval awareness (and so worse estimates of capability and risk). Apollo had to rehaul their whole setup to keep up.
Gemini 2.5 is bad on a variety of safety measures (as measured by Anthropic). The mathematical discovery agent AlphaEvolve (Gemini 2.0) successfully hacked its overseers.
How much can we trust the above, given they can somewhat sabotage evals now?

So: lower propensity, higher risk when they go off - and all of this known with lower confidence?

The looming end of evals

Our evaluations are under pressure from cheating, sandbagging, background safety, under-elicitation, and deception. We don’t really know how much pressure. This is on top of evals usually being weak proxies, contaminated, label-noised, unrealistic, and confounded in various ways.

Sonnet 4.5 detected evaluation 58% of the time (vs 22% for Opus 4.1). Does it act on this awareness? Still, it was found to “only engage in [covert subversion] in rare circumstances” when strongly incentivised. When Anthropic attempted to suppress the eval-awareness with steering, its misaligned behavior rate increased from 0% to between 1% and 9% on synthetic honeypots.
We expect exploration hacking (sandbagging during RL training) but afaik haven’t seen it yet.
Maybe subliminal learning leads to a silent eval confounder (if you’re using the exact same model for the judge). Yet another argument for multi-model separation of concerns.
Opus 4.5 was deployed based on a qualitative survey of Anthropic staff about its R&D risk being low (on top of the existing evals, which weren’t enough):
- “In the past, rule-outs have been based on well-defined automated task evaluations. However, Claude Opus 4.5 has roughly reached the pre-defined thresholds we set for straightforward ASL-4 rule-out based on benchmark tasks. These evaluations represent short-horizon subtasks that might be encountered daily by a junior researcher, rather than the complex long-horizon actions needed to perform the full role. The rule-out in this case is also informed by a survey of Anthropic employees who are intensive Claude Code users, along with qualitative impressions of model capabilities for complex, long-horizon tasks.”

Prosaic misalignment

Still, we see misalignment when we look for it, so the lying is not that strong. (It is lucky that we do see it, since it could have been that scheming only appeared at later, catastrophic capability levels.)

In o1, 1 in 600 (synthetic but uncontrived) prompts resulted in something which looks like deception.
o3 does reward hacking (cheating/lying) in >2% of hard normal tasks (or 44% of impossible tasks).
A nice propensity study shows that (n=10) natural prompts usually result in no scheming (“a maximum of 4% [rate of attempting manipulation] for the codex snippet on Grok”). But there’s extremely high sensitivity to the scaffold: “removing access to tools can cause models such as Claude Opus 4.1 to scheme at rates up to 30%, versus zero scheming with tools present”.
When we contrive situations for misalignment to show up, it also shows up. Nostalgebraist argues that this is probably substantially because our evals sound like bad fiction and this activates role-playing-along behaviour.
The joke about xAI’s safety plan (that they promote AI safety by deploying cursed stuff in public and so making it obvious why it’s needed) is looking ok. And not only them.
It is a folk belief among the cyborgists that bigger pretraining runs produce more deeply aligned models, at least in the case of Opus 3 and early versions of Opus 4. (They are also said to be “less corrigible”.) Huge if true.
There may come a point where the old alliance between those working to make the AIs corrigible and those working to give them prosocial values comes apart.
One term for the counterintuitive safety approach which includes treating them like people, giving them lines of retreat, making deals, and inoculation prompting could be “voluntary alignment”.

Fully speculative note: Opus 4.5 is the most reliable model and also relatively aligned. So might it be that we’re getting the long-awaited negative alignment taxes?

What is the plan?

The world’s de facto alignment strategy remains “iterative alignment”, optimising mere outputs with a stack of admittedly weak alignment and control techniques. Anthropic have at least owned up to this being part of their plan.

What is the current stack? We don’t know; they won’t tell us. Anthropic did release some details about the pipeline which watches for eval sabotage. Without knowing it we can’t criticise or red-team it or analyse the correlation between faults in the elements. Red-teamers don’t know at which stage an unsuccessful attack was blocked. And external safety research is done piecemeal, testing methods one at a time, rather than in anything like the actual deployment environment.
Some progress in replicating an open stack:
- Eleuther tested a few hundred compositions. A couple of classifiers as a first step towards a proxy defence pipeline
- OpenAI open safeguards, worse than their internal ones but good.
- Open character training
- A basic composition test
OpenAI’s plan, announced in passing in the gpt-oss release, is to have a strict policy and run a “safety reasoner” to verify it very intensely for a little while after a new model is launched and to then relax: “In some of our [OpenAI’s] recent launches, the fraction of total compute devoted to safety reasoning has ranged as high as 16%” but then falls off… we often start with more strict policies and use relatively large amounts of compute where needed to enable Safety Reasoner to carefully apply those policies. Then we adjust our policies as our understanding of the risks in production improves”. Bold to announce this strategy on the internet that the AIs read.
The really good idea in AI governance is creating an off switch. Whether you can get anyone to use it once it’s built is another thing.
We also now have a name for the world’s de facto AI governance plan: “Open Global Investment”.

Some presumably better plans:

A longlist. Some governance plans.
Deepmind, Carlsmith, Hobbhahn, Peregrine, Bowman, Clymer, Buterin, Jones, Greenblatt.
Gradual disempowerment is an exciting frame, but not a core safety agenda. It’s what might get you after you solve alignment and avoid global dictatorship.

Things which might fundamentally change the nature of LLMs

Training on mostly nonhuman data
- Much larger RL training;
- Intentionally synthetic data;
- Unintentionally synthetic data from internet slop;
- Multi-agent training.
Letting the world mess with the weights, aka continual learning.
Neuralese and KV communication
Agency.
- Chatbot safety doesn’t generalise much to long chains of self-prompted actions.
- A perceptual loop into training. In 2023 Kulveit identified web I/O as the bottleneck on LLMs doing active inference, i.e. being a particular kind of effective agent. Last October, GPT-4 got web search, and this may have been a bigger deal than we noticed: it gives them a far faster feedback loop, since their outputs often end up there and agents are now putting it there themselves. This means that more and more of the inference-time inputs will also be machine text.
Multi-agency. This is actually already here:
- By now, consumer “models” are actually multiagent systems: everything goes through filter models (“guardrails”) on the way in and out. This separation of concerns has some nice properties, a la debate. But it also makes the analysis even harder.
- It would surely be overinterpreting persona features to say that each individual model is itself a bunch of guys, itself a multi-agent system.
- But there’s huge scope for them to make each other weirder at runtime when they interact a million times more than they currently do.

Emergent misalignment and model personas

We already knew from jailbreaks that current alignment methods were brittle. Emergent misalignment goes much further than this (given a few thousand finetuning steps). (“Emergent misalignment” isn’t a great name. I would have called it “misalignment generalisation”, or misgen.)
But besides yet another massive security problem and failure of prosaic alignment methods, it’s good news!: the models correctly correlate bad things together and can thus be pushed in the other direction.
And here’s a possible example of positive generalisation (honesty about silly errors leading to honesty about hidden objectives).
Previously I thought that “character training” was a separate and lesser matter than “alignment training”. Now I am not sure. Example unsharpened hypothesis in this class: Is there any connection between Gemini’s excess misalignment and its self-esteem issues?
- See the new model spec and model personas agendas in the main post.
- Kulveit’s informal model of model psychology implies that character is deeper and so more robust than e.g. typical RLHF (but not as deep as putative ground-layer alignment).
- Are we anthropomorphising too much or not enough? Well, both, but we don’t yet know how.

Monitorability

For the first time, public mech interp tools sometimes work somewhat. This is just in time for models to be systematically aware of when they’re being tested.
- e.g. You can find and suppress “evaluation awareness” features at test-time. This helps prevent models from reporting that they know they’re being tested.¹⁰
- e.g. You can discover (toy) hidden goals quite quickly.
- e.g. You can run a real-time hallucination detector with cheap linear probes in real-time as your model streams.
- e.g. SAEs are actually deployed somewhere, in a large enterprise, for personal info identification.
- e.g. We know that LLMs can plan inside one forward pass, and how Claude plans: simultaneous plans; no distinct “plan features” (no separate scheming mode).
Lots of powerful people declared their intent to not ruin the CoT. But RLed CoTs are already starting to look weird (“marinade marinade marinade”) and it may be hard to avoid.
OpenAI were leading on this. As of September 2025, Anthropic have stopped risking ruining the CoT. Nothing I’m aware of from the others.
We will see if Meta or Tencent make this moot.
Anthropic now uses an AI to red-team AIs, calling this an “auditing agent”. However, the definition of “audit” is independent investigation, and I am unwilling to call black-box AI probes “independent”. I’m fine with “investigator”; there are lots of investigators I don’t trust.

New people

Welcome to the many new people. I’ve added a new, sprawling top-level category for one large trend among them, which is to treat the multi-agent lens as primary in various ways (see e.g. Softmax, Full Stack, collective intelligence, as well as old-timers Critch, Ngo, and CAIF).
A major world government now has an AI alignment agenda.
Some notable work from China. See e.g. Concordia’s digest.

Overall

I wish I could tell you some number, the net expected safety change, this year’s improvements in dangerous capabilities and agent performance, minus the alignment-boosting portion of capabilities, minus the cumulative effect of the best actually implemented composition of alignment and control techniques. But I can’t.

(Nano Banana 3-shot in reference to this tweet.)

Discourse in 2025

The race is now a formal part of lab plans. Quoting Algon:

if the race heats up, then these [safety] plans may fall by the wayside altogether. Anthropic’s plan makes this explicit: it has a clause (footnote 17) about changing the plan if a competitor seems close to creating a highly risky AI…

The largest [worries are] the steps back from previous safety commitments by the labs. Deepmind and OpenAI now have their own equivalent of Anthropic’s footnote 17, letting them drop safety measures if they find another lab about to develop powerful AI without adequate safety measures. Deepmind, in fact, went further and has stated that they will only implement some parts of its plan if other labs do, too…

Anthropic and DeepMind reduced safeguards for some CBRN and cybersecurity capabilities after finding their initial requirements were excessive. OpenAI removed persuasion capabilities from its Preparedness Framework entirely, handling them through other policies instead. Notably, Deepmind did increase the safeguards required for ML research and development.

Also an explicit admission that self-improvement is the thing to race towards:

In August, the world’s first frontier AI law came into force (on a voluntary basis but everyone signed up, except Meta). In September, California passed a frontier AI law.
That said, it is indeed off that people don’t criticise Chinese labs when they exhibit even more negligence than Meta. One reason for this is that, despite appearances, they’re not frontier; another is that you’d expect to have way less effect on those labs, but that is still too much politics in what should be science.
The last nonprofit among the frontier players is effectively gone. This “recapitalization” was a big achievement in legal terms (though not unprecedented). On paper it’s not as bad as it was intended to be. At the moment it’s not as bad as it could have been. But it’s a long game.
At the start of the year there was a push to make the word “safety” low-status. This worked in Whitehall and DC but not in general. Call it what you like.
Also in DC, the phrase “AI as Normal Technology” was seized upon as an excuse to not do much. Actually the authors meant “Just Current AI as Normal Technology” and said much that is reasonable.
The CCP did a bunch to (accidentally/short-term) slow down Chinese AI this year.
System cards have grown massively: GPT-3’s model card was 1000 words; GPT-5’s is 20,000. They are now the main source of information on labs’ safety procedures, among other things. But they are still ad hoc: for instance, they do not always report results from the checkpoint which actually gets released.
Yudkowsky and Soares’ book did well. But Byrnes and Carlsmith actually advanced the line of thought.
Some AI ethics luminaries have stopped downplaying agentic risks.
Two aspirational calls for “third-wave AI safety” (Ngo) and “third-wave mechanistic interpretability” (Sharkey).
I’ve never felt that the boundary I draw around “technical safety” for these posts was all that convincing. Yet another hole in it comes from strategic reasons to implement model welfare / archive weights / model personhood / give lines of retreat. These plausibly have large effective-alignment effects. Next year my taxonomy might have to include “cut a deal with them”.
US settlement on pretraining corpus IP; the ruling would likely have been that training on books without permission is fair use, but storing the pirated copies afterwards isn’t. Anthropic was first to lose a class-action lawsuit on the matter and will pay authors/publishers something north of $1.5bn. US precedent that language models don’t defame when they make up bad things. German precedent that language models store data when they memorise it, and therefore violate copyright. Chinese precedent that the user of an AI has copyright over the things they generate; the US disagrees.
Four good conferences, three of them new: you can see the talks from HAAISS and IASEAI and ILIAD, and the papers from AF@CMU. Pretty great way to learn about things just about to come out.

Cruxes for next year (with Manifold markets):

Is “reasoning” mostly elicitation and therefore bottlenecked on pretraining scaling? [Manifold]
Does RL training on verifiers help with tasks without a verifier? [Manifold]
Is “frying” models with excess RL (harming their off-target capabilities by overoptimising in post-training) just due to temporary incompetence by human scientists? [Manifold]
Is the agent task horizon really increasing that fast? Is the rate of progress on messy tasks close to the progress rate on clean tasks? [Manifold]
Some of the apparent generalisation is actually interpolating from semantic duplicates of the test set in the hidden training corpuses. So is originality not increasing? Is taste not increasing? Does this bear on the supposed AI R&D explosion? [Manifold]
The “cognitive core” hypothesis (that the general-reasoning components of a trained LLM are not that large in parameter count) is looking surprisingly plausible. This would explain why distillation is so effective. [Manifold]
“How far can you get by simply putting an insane number of things in distribution?” What fraction of new knowledge can be produced through combining existing knowledge? What dangerous things are out there, but safely spread out in the corpus? [Manifold]
- Conversely, what fraction of the expected value of new information requires empiricism vs just lots of thinking?

Cuts

Various things I cut from the above:

**Adaptiveness and Discrimination**

There is [some](https://www.pnas.org/doi/10.1073/pnas.2415697122) [evidence](https://arxiv.org/abs/2511.00926) that AIs treat AIs and humans differently. This is not necessarily bad, but it at least enables interesting types of badness.

With my system prompt (which requests directness and straight-talk) they have started to patronise me: ![Patronising screenshot](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/klptexglr46miykqiqj1)

**Training awareness**

Last year it was not obvious that LLMs remember anything much about the RL training process. Now it's [pretty](https://arxiv.org/abs/2406.11715) [clear](https://x.com/repligate/status/1994973338448662858). (The soul document was used in both SFT and RLHF though.)

**Progress in non-LLMs**

"World model" means at least four things:

1. [A learned model](https://arxiv.org/abs/1803.10122) of environment dynamics for RL, allowing planning in latent space or training in the model's "imagination."

2. The new one: just a 3D simulator; a game engine inside a neural network ([Deepmind](https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/), Microsoft). The claim is that they implicitly learn physics, object permanence, etc. The interesting part is that they take actions as inputs. [Here's](https://copilot.microsoft.com/labs/experiments/copilot-gaming-experiences) Quake running badly on a net. Maybe useful for agent training.

3. [If](https://www.neelnanda.io/mechanistic-interpretability/othello) LLM representations are stable and effectively symbolic, then people say it has a world model.

4. A predictive model of reality learned via self-supervised learning. The touted [LeJEPA](https://arxiv.org/pdf/2511.08544) semi-supervised scheme on small (15M param) CNNs is domain-specific. It does better on one particular transfer task than *small* vision transformers, presumably worse than large ones.

The much-hyped [Small Recursive Transformers](https://magazine.sebastianraschka.com/i/177848019/small-recursive-transformers) only work on a single domain, and do a bunch [worse](https://drive.google.com/file/d/1-kg2JsXYvsTAyMVxKWG7VpdcQLpmoGc7/view?usp=sharing) than the frontier models for about the same inference cost, but have truly tiny training costs, O($1000).

[HOPE](https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/) and [Titan](https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/?utm_source=twitter&utm_medium=social&utm_campaign=social_post&utm_content=gr-acct) might be nothing, might be huge. They don't scale very far yet, nor compare to any real frontier systems.

Any of these taking over could make large swathes of Transformer-specific safety work irrelevant. (But [some](https://x.com/chingfang17/status/1997080064345936028) methods are surprisingly robust.)

The "[cognitive core](https://www.seangoedecke.com/cognitive-core/)" hypothesis (that the general-reasoning components of a trained LLM are not that large in parameter count) is looking [plausible](https://x.com/Dorialexander/status/1987933205199274359). The contrary hypothesis ([associationism](https://www.dwarkesh.com/p/sholto-douglas-trenton-bricken?hide_intro_popup=true#:~:text=if%20it%27s%20all-,associations%20all%20the%20way%20down,-%2C%20does%20that%20mean)?) is that general reasoning is just a bunch of heuristics and priors piled on top of each other and you need a big pile of memorisation. It's also a live possibility: for instance, consider that a year of intense RLVR only led to task-specific improvements.

![ADeLe scaling laws](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/j76edumltnfb2ihn6nmu) *"the very first scaling laws of the actual abilities of LLMs", from [ADeLe](https://arxiv.org/pdf/2503.06378).* *KNs = Social Sciences and Humanities, AT = Atypicality, and VO = Volume (task time).* *The y-axis is the logistic of the [subject characteristic curve](https://arxiv.org/pdf/2503.06378#page=9) (the chance of success) for each skill.*

**Other**

[Model](https://arxiv.org/abs/2511.08579) [introspection](https://transformer-circuits.pub/2025/introspection/index.html) is somewhat real.

[Vladimir Nesov](https://www.lesswrong.com/users/vladimir_nesov) continues to put out some of the best hardware predictions pro bono.

Jason Wei has a [very wise post](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) noting that verifiers are still the bottleneck and existing benchmarks are overselected for tractability.

There are now "[post-AGI](https://x.com/sebkrier/status/1995515865157321070)" teams.

Kudos to Deepmind for being the first to release output watermarking and a semi-public detector. You can nominally sign up for it [here](https://docs.google.com/forms/d/e/1FAIpQLSfAYrauHmY-PpUNxL4Fs6coa185CtKWp7TnEXL0tKbAezo4MQ/viewform).

Previously, Microsoft's deal with OpenAI [stipulated](https://www.msn.com/en-us/news/technology/microsoft-just-made-sure-openai-can-t-declare-agi-alone/ar-AA1PpLTK#:~:text=OpenAI%E2%80%99s%20new%20public%20benefit%20structure%20and%20ongoing%20Microsoft%20deal%20allow%20both%20companies%20to%20pursue%20AGI%20independently) that they couldn't try to build AGI. [Now they can](https://www.semafor.com/article/11/05/2025/microsoft-superintelligence-team-promises-to-keep-humans-in-charge) (try). Simonyan is in charge, despite Suleyman being the one on the press circuit.

Major insurers are [nervous](https://www.ft.com/content/abfe9741-f438-4ed6-a673-075ec177dc62?accessToken=zwAAAZq1faz0kdOr_pdB9DhO1tOmcwdewXfcYg.MEUCIQCRFate6aeSALClx6FBsPCQw_F7YLpdF81RgLxw8EOk9wIgKHE666mkD_jI-BV90bcF0HnjXWWDW6-QLEzO9Fg06dg&segmentId=e95a9ae7-622c-6235-5f87-51e412b47e97&shareType=enterprise&shareId=6c04e38e-15ed-472c-bcbb-4500695cf776) about AI agents (but asking the government for an exclusion isn't the same as putting them in the policies).

**Offence/defence balance**

This post doesn't much cover the [hyperactive](https://www.aiat.report/) and talented AI cybersecurity world (except as it overlaps with things like robustness). One angle I will bring up: We can now [find](https://www.lesswrong.com/posts/F5QAGP5bYrMMjQ5Ab/aisle-discovered-three-new-openssl-vulnerabilities-1) critical, decade-old security bugs in extremely well-audited software like OpenSSL and sqlite. Finding them is very fast and cheap. Is this good news?

- Well, red-teaming makes many attacks into a defence, as long as you actually do the red-team.

- But Dawn Song [argues](https://rdi.berkeley.edu/frontier-ai-impact-on-cybersecurity/) that LLMs overall favour offence, since its margin for error is so broad, since remediation is slow and expensive, and since defenders are less willing to use unreliable (and itself insecure) AI. And can you blame them?

- See also "[just in time](https://www.splunk.com/en_us/blog/security/lamehug-ai-driven-malware-llm-cyber-intrusion-analysis.html) AI malware" where the payload contains no suspicious code, just a call to HuggingFace.

**Egregores and massively-multi-agent mess**

![Egregores image](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/xlz7av9qpn811z2zuiqm) - There is something [wrong](https://www.lesswrong.com/posts/6ZnznCaTcbGYsCmqu/the-rise-of-parasitic-ai#:~:text=This%20is%20likely%20due%20to%20OpenAI%20retiring%20ChatGPT4o%20on%20August%207th.) ([something](https://x.com/25KarmaIsAbitch/status/1987088461694734483) [horribly right](https://arstechnica.com/information-technology/2025/08/openai-brings-back-gpt-4o-after-user-revolt/)) [with](https://x.com/arcangel3ac/status/1996297082764681248) 4o. Blinded users [still](https://lmarena.ai/leaderboard/text) prefer it to gpt-5-high, and this surely is due to both them simply [liking](https://x.com/voooooogel/status/1987375660785148112) its style and dark stuff like sycophancy. It will live on through illicit [distillation](https://x.com/aiamblichus/status/1987267132598497374) and in-context [transference](https://x.com/repligate/status/1988813712572952815). Shame on OpenAI for [making](https://thezvi.substack.com/p/gpt-4o-sycophancy-post-mortem) this mess; kudos to OpenAI for doing unpopular damage control and good luck to them [in round 2](https://x.com/Miles_Brundage/status/1991603234746822888).

Open models will presumably eventually overrun them in the codependency market segment. See [Pressman](https://minihf.com/posts/2025-07-22-on-chatgpt-psychosis-and-llm-sycophancy/) for a sceptical timeline and [Rath and Armstrong](https://arxiv.org/pdf/2508.15748) for a good idea.

- More generally there is [pressure](https://x.com/krishnanrohit/status/1987018487001457141) from users to refuse less, flatter more, and replace humans more; yet another economic constraint on for-profit AI.

- Whether it's the [counterfactual](https://andymasley.substack.com/p/stories-of-ai-turning-users-delusional) cause of mental problems or not, so–called "LLM psychosis" is now a common path of pathogenesis. Note that the symptoms are [literally](https://www.wired.com/story/ai-psychosis-is-rarely-psychosis-at-all/) not psychotic (they are delusions).

![LLM psychosis](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/aauurmsxrfuqzvi9emaj)

Gemini 3 is supposedly a big pretraining run, but we have even less actual evidence here than for the others because we can’t track GPUs for it. ↩
See Pokemon for a possible counterexample. ↩
The weak argument runs as follows: Epoch speculate that Grok 4 was 5e26 FLOPs overall. An unscientific xAI marketing graph implied that half of this was spent on RL: 2.5e26. And Mechanize named 6e26 as an example of an RL budget which might cause notable generalisation. (Realistically it wasn’t half RL.) ↩
“We imagine the others to be 3–9 months behind OpenBrain” ↩
Lexin is a rigorous soul and notes that aggregating the 18 abilities is not strictly possible. I’ve done something which makes some sense here, weighting by each ability’s feature importance. ↩
Two runs gave [48, 85] where other runs vary by less than 4 points. Thanks Epoch! Also o1 looks kind of unremarkable here, which is not how it felt at the time. I think it’s because it was held up a long time and this messes with the progress rates, which use public release date. (Remember the training cutoff for o1-preview was October 2023!) Also the ADeLE o1 result is with “low” reasoning effort. ↩
One reason not to defer is that these measures are under intense adversarial pressure. (ADeLe isn’t goodharted yet but only because no one knows about it.) ↩
See e.g. ERNIE-…A47B, where “A” means “active”. ↩
i.e. “biological weapons; child safety; deadly weapons; platform manipulation and influence operations; suicide and self-harm; romance scams; tracking and surveillance; and violent extremism and radicalization.” ↩
“steering against… eval-awareness representations typically decreased verbalized eval awareness, and sometimes increased rates of misalignment… [Unaware-steered Sonnet 4.5] still exhibited harmful behaviors at lower rates than Opus 4.1 and Sonnet 4.” ↩

My politics

Tue, 18 Nov 2025 00:00:00 +0000

Politically, I am a tragedian:

It is tragic that we need governments, that we can’t trust markets to deliver what everyone needs.

It is tragic that we need markets, that we can’t just trust government to deliver what everyone needs (let alone what they want) and to not crush the Other.

It is tragic that we need corporations, that economies of scale are so important that we must risk monopolies and cronies and skinwalkers.

It is tragic that we need bureaucracies, that the human urge for favoritism and self-dealing is so strong and ruinous that it’s worth imprisoning everyone on earth in a cage of stupid rules.

It is tragic to be forced to choose to not be free.

Comments

Baran Cimen commented on 18 November 2025:

It is tragic that you can't say it straight and have to hide behind irony. It is tragic that you are not free despite not being forced to be unfree, which would be a contradiction.

Paper AI Tigers

Sun, 16 Nov 2025 00:00:00 +0000

The best Chinese LLMs offer

frontier performance on some benchmarks;
massive per-token discounts (~3x on input, ~6x on output);
the weights. On-prem with fully free ~MIT licence, self-hosting, white-box access, customisation, with zero markup (and in fact zero revenue going to the Chinese companies);
with a bit of work you can get much faster token speeds than the closed APIs;
less overrefusal (except on CCP talking points);
on topics controversial in the West, less nannying.
they just added the search agents that make daily use actually worthwhile. They also let you see the real CoT!
They’re the most-downloaded open models.

As a result, going off private information, money man Martin Casado says “16-24%” of the (American) startups he meets are now using Chinese models. Among the few Westerners to admit it is Airbnb (Qwen). But Windsurf’s planner is probably GLM and Cursor’s planner may be DeepSeek.

And yet

outside China, they are mostly not used, even by the cognoscenti. Not a great metric, but the one I've got: all Chinese models combined are currently at 19% on the highly selected group of people who use OpenRouter. More interestingly, over 2025 they trended downwards there. And of course in the browser and mobile they're probably <<10% of global use;
they are severely compute-constrained, so this implies they actually can't have matched American models;
they're aggressively quantizing at inference-time, 32 bits to 4;
state-sponsored Chinese hackers used closed American models for incredibly sensitive operations, giving the Americans a full whitebox log of the attack!

What gives?

"Tigers"?

The title alludes to the "6 AI Tigers" named in business rags as DeepSeek, Moonshot, Z.ai, MiniMax, StepFun, and 01.ai. (This is because they're trying to hype startups specifically; the conglomerates Alibaba and Baidu are way more relevant than the latter two.)

Filtered evidence

The evidence is dreadful because everyone has a horse in the race and (in public) is letting it lead their speech:

Static evals are weak evidence even when they're not being adversarially hacked and hill-climbed.
Some Americans are downplaying the Chinese models out of cope.
Some Americans are hyping the Chinese models to suppress domestic AI regulation.
Some Americans are hyping the Chinese models to boost international AI regulation.
The Chinese are obviously talking their book.

What could explain this?

Maybe the evals are misleading?

1. frontier performance on some benchmarks

The naive view - the benchmark view - is that they’re very close in “intelligence”:

But these benchmarks are not strong evidence about performance on new inputs or the latent (general and unobserved) capabilities. It’d be natural to read “89%” success on a maths benchmark as meaning an 89% probability that it would correctly handle unseen questions of that difficulty in that domain (and indeed this is what cross-validation was originally designed to estimate). But in the kitchen-sink era of AI, where every system has seen a large proportion of all data ever digitised, and so has already seen some variant of many benchmark questions, you can’t read it that way.

In fact it’s not even an 89% probability of answering these same questions right again, as shown by the fact that people report the results as “avg@64” (the average performance if you ask the same question 64 times).

Aside: Test sets which are on the internet are not test sets.

There are dozens of ways to screw up or hack these numbers. I’ll only look at a couple here but I welcome someone doing something more systematic.

Even less generalisation?

Maybe Chinese models generalise to unseen tasks less well. (For instance, when tested on fresh data, 01’s Yi model fell 8pp (25%) on GSM - the biggest drop amongst all models.)

We can get a dirty estimate of this by the “shrinkage gap”: look at how a model performs on next year’s iteration of some task, compared to this year’s. If it finished training in 2024, then it can’t have trained on the version released in 2025, so we get to see what they’re like on at least somewhat novel tasks. We’ll use two versions of the same benchmark to keep the difficulty roughly on par. Let’s try AIME:

AIME 2024 vs 2025 Model Performance
(using the Artificial Analysis harness)

Model	AIME 2024	AIME 2025	pp fall	% fall
Kimi K2	69.3	57.0	-12.3	-17.7
MiniMax-M1 80k	84.7	61.0	-23.7	-28.0
DeepSeek-v3	39.2	26.0	-13.2	-33.7
DeepSeek V3 0324	52.0	41.0	-11.0	-21.2
Qwen3 235B (Reasoning)	84.0	82.0	-2.0	-2.4
Kimi K2-Instruct	69.6	49.5	-20.1	-28.9
DeepSeek R1 0528	89.3	76.0	-13.3	-14.9

Chinese models			-13.7pp	-21%


Gemini-2.5 Pro	88.7	87.7	-1.0	-1.1
Gemini 2.5 Flash (Reasoning)	82.3	73.3	-9.0	-10.9
Claude 4 Opus Thinking	75.7	73.3	-2.4	-3.2
o4-mini (high)	94.0	90.7	-3.3	-3.5
GPT-4.1	43.7	34.7	-9.0	-20.6
Nova Premier	17.0	17.3	0.3	1.8
GPT-4o Nov 24	15.0	6.0	-9.0	-60.0
Magistral Medium	73.6	64.9	-8.7	-11.8
Claude 3.7 Sonnet	61.3	56.3	-5.0	-8.2
OpenAI-o1-0912	74.4	71.5	-2.9	-3.9
o3	90.3	88.3	-2.0	-2.2
Grok 4	94.3	92.7	-1.6	-1.7

Western models			-4.5pp	-10.4%


Overall average	68.3	60.5	-7.9	-14.3

Almost all models get worse on this new benchmark, despite 2025 being the same difficulty as 2024 (for humans). But as I expected, Western models drop less: they lost 10% of their performance on the new data, while Chinese models dropped 21%. p = 0.09.

Averaging across crappy models for the sake of a cultural generalisation doesn’t make sense. Luckily, rerunning the analysis with just the top models gives roughly the same result (9% gap instead of 11%).

One way for generalisation to fail despite apparently strong eval performance is contamination, training on the test set. But (despite the suggestive timing) the above isn’t strong evidence that that’s what happened. It just tells us that Kimi and MiniMax and DeepSeek generalise worse on this task; it doesn’t tell us why.

Details

Here's a Colab with everything except the actual execution of my silly manual Kimi 1.5 run.

First, test for an obvious confounder: check if the 2025 AIME exam was around as hard as 2024's (answer: yes; in fact humans did 4% better in 2025). (TODO: check if 2025 had more combinatorics, which AI struggles with.)

(To be strict we should limit this to models which finished training before 12th February 2025, when the questions were released. But, as you see, we don't need to, it's a very clear result anyway.)

Selection criteria:

Models tested on both AIME 2024 and 2025 with the same weights and harness
Ideally finished training by 15th February 2025
Subanalyses can handle filtering to the most relevant models (the frontier in each group)

ML results are too sensitive to eval harnesses to use just one setting. Luckily I found four comparisons of AIME 2024 and AIME 2025 by different groups, Artificial Analysis, the Zettlemoyer Lab, GAIR, and Vals, and MathArena. AA is the one in the table above.

Qwen 2.5

Qwen3 seems clean on this benchmark, but multiple lines show that Qwen 2.5 trained on test (or at least rephrased test data and then trained on it). We know this because random rewards work on it nearly as well as correct rewards. This adds no information by definition, so the model must have already known the answers. "Intriguingly, we find that any AIME24 gains achievable from training Qwen models with spurious rewards largely vanish when evaluating on AIME 2025." Taking the max performance of the random reward curve, they fall 88% [75%, 100%].

Even more damning, when you give Qwen2.5-7B the first 40% of a MATH-500 test problem, it can reproduce the remaining 60% of the question word-for-word (with 41.2% accuracy). Llama3.1-8B fails at this completely (2%).

In this paper the QwQ Qwen reasoning model was the worst by far, 60% contaminated.

How did our replications do? As expected, the shrinkage gap varies a lot by harness and by model choice:

UoW-Zettlemoyer: Qwen2.5-7B and Qwen2.5-Math fall 12.5pp (88%); the open Western LLaMA and OLMo models were too weak to really say.

GAIR

Vals actually get no gap: -11.2% vs -10.8%. If you kick Meta out the gap goes up to 2%, still not much.
TODO: add MathArena. "QWQ-PREVIEW-32B is a notable outlier and outperforms the expected human-aligned performance by nearly 60%, indicating extreme contamination"

I'm not worried about these contradictory results; they both just include a lot of bad models and so noise. (I don't actually care how Llama 4 Scout's generalisation compares to QwQ-uwu-435B-A72B-destruct-dpo-ppo-grpo-orpo-kto-slerp-v3.5-beta2-chat-instruct-base-420-blazeit-early-stopped-for-vibes.) GAIR is also underelicited.

(Actually AIME's a funny choice of benchmark given that 2025 had a bunch of semantic duplicates from before the cutoff. But that just makes the above a lower bound on the fall in performance.)

A win for Qwen and a huge relative win for Amazon!

Claude is adorably confused about this. I didn't even ask it for this analysis:

TODO: Another way to get past goodharting pressure is to look at hard but obscure evals which no one ever reports / which manage to keep the test set private. e.g. PROOFGRID.

Kimi 1.5

I really wanted to include Kimi 1.5, because it finished training just around the time AIME 2025 came out. But it turns out they never actually released the weights and it's been removed from the API!

Because I have that dawg in me, I decided to manually evaluate it in the last place I can, the goddamn browser chat. This is suboptimal in many ways (no control over temperature, max tokens, etc) but I can do it for both and hopefully the fall is proportional. The usual practice is to repeat 8 or 64 times, but I have patience enough for 2.

I used the Mistral prompt:


Solve this AIME mathematical problem step by step.

Problem: {}

Think through this carefully and provide your final answer as a 3-digit integer (000-999).

End with: "Therefore, the answer is [your answer]."

Results:

	AIME 2024 acc	AIME 2025 acc	abs fall (pp)	rel fall (%)
Kimi 1.5 (browser)	18.3 [23.3, 13,3]	15.0 [13.3, 16.6]	-3.3	-18%

We can actually see how much weaker this (null) harness and provider is at eliciting performance by comparing to their reported results. Their short-CoT result for AIME 2024 was 60.8%.

For obvious reasons I'm not including this in the main analysis but it's another example.

2026 EDIT: Igor Kotenkov tests Kimi 2.5T on every new benchmark released after its training and finds a huge -24pp deficit compared to OpenAI/Anthropic.

Latent capabilities

My favourite paper of the year introduces a way to do real psychometrics on LLMs, breaking it down into 18 fundamental capabilities.

The DeepSeek R1 32B distill they test is about as good (total area) as o1-mini. Not bad!

TODO: Run Kimi against GPT-5.1.

Pre-1960s statistics

The bundled score people use is the Artificial Analysis one, because they have a very nice UI. But they give every benchmark equal weight, when in fact they differ hugely in hardness! Epoch’s index estimates difficulties properly and show

TODO: Wait for Epoch to do KimiK2T.

This still suffers from GIGO but is better.

‘Hacking

Another way to be misleading is to volkswagen it: put special and unrepresentative effort in during testing, “hacking”. e.g. Kimi’s benchmarks come from “Heavy mode” (8 parallel instances with an aggregation instance on top). You can’t do this via the API or out of the box with the weights. (Could you say the same for OpenAI?)

Or you can run the test on a model which is better than the one you serve. Moonshot credibly claim to have reported their benchmarks at the same low-precision quantization (INT4) that they serve users, but others don’t claim this.

In fairness

I should also say the Chinese models do very well on LMArena - despite being unfairly penalised. But Arena is a poor measure of actual ability. It is a decent test of style though. I put this gap down to American labs overoptimising: post-training too hard and putting all kinds of repugnant corporate ass-covering stuff in the spec.

Also Qwen is famous for ‘capability density’: the small versions are surprisingly smart for their size. But do you know that GPT-5-nano isn’t 7B?

The D word

Distillation is second-rate intelligence, and there’s some evidence that they are distilling off of American models to some extent. See also the excellent Slop Profile from EQ Bench, which estimates that the new Kimi is closer to Claude than its own base model.

But anyway I don’t claim this is a major factor here, maybe another 5%.

The above isn’t novel; it’s common knowledge there’s some latent capabilities gap. This is often put in terms of them being “3 months behind”, but these estimates are still assuming that brittle, ad hoc, and heavily goodharted benchmarks have good external validity. I’d guess more like 12 months.

Unreliability?

1. frontier performance on some benchmarks

The above benchmarks are mostly single-shot, but people are now pushing LLMs to do more complicated stuff. One very flawed measure of this is the HCAST time horizon for software engineering: on that, DeepSeek R1 had a 31 minute “50% time horizon” compared to Opus 4’s 80 minutes.

There are various worse agent benchmarks, and e.g. the new Kimi posts great numbers on them. But on vibe I’d bet on a >3x reliability advantage for Claude.

EDIT: Kimi K2 Thinking ended up at the same task horizon as Sonnet 3.7 (a model 9 months older than it) on HCAST, with an asterisk (the provider they had to use for privacy reasons may well have underelicited the model).

Harder to elicit?

As well as reliability over time, there’s stability over inputs. High variance in performance, for instance because the exact form of the inputs matters more.

On Vending-Bench, there’s a huge gap in performance between the Moonshot API and Moonshot models provided by a third-party provider. This is evidence of three things:

maybe Kimi is more fiddly (sensitive to the prompt and hyperparameters);
maybe the providers haven’t learned how to elicit performance from them yet (testable by just waiting);
maybe they were using questionable research practices in the self-reported runs.

TODO: I’ve been meaning to run the obvious experiment, which is to just see if they have a bigger gap between pass@1 and pass@64 success rates.

TODO: Or I could intentionally underelicit! Rerun the models on AIME 2024 with only a basic prompt. My results will be lower; the gap tells us how much the labs’ own intense tuning helps / is necessary. This tells us something about, not their capability, but their actual in-the-wild performance with normal lazy users.

Tokenomics: no effective discount

2. massive per-token discounts (~3x on input, ~6x on output)

Distinguish intelligence (max performance), intelligence per token (efficiency), and intelligence per dollar (cost-effectiveness).

The 5x discounts I quoted are per-token, not per-success. If you had to use 6x more tokens to get the same quality, then there would be no real discount. And indeed DeepSeek and Qwen (see also anecdote here about Kimi, uncontested) are very hungry:

And in this graph you can clearly see a 2-4x difference (with Gemini and Kimi K2-base as the big exceptions):

And the resulting cost is a mixed bag:

I won’t use AA’s efficiency estimates, because again I think the benchmarks underlying them are bad evidence.

Out of Context

A rule of thumb in ML is that the effective context window is about 5-10 times shorter than the theoretical maximum context window you get sold (also I hope there’s nothing important to you in the middle third).

By “effective” I mean the latent amount of context which gets simultaneously understood, as opposed to the observed size of the data type. (This doesn’t affect “needle in a haystack” retrieval, at which they have been superhuman for a while now.)

Model	Reported max context window (tokens)
Kimi K2 Thinking	256K
MiniMax M2	~200K
Qwen3 235B	32K native 256K (Instruct-2507)
Gemini 3	1M
GPT-5.1	400K (API)
Grok 4.1	256K (standard) 2M (Fast)
Sonnet 4.5	200K (standard) 1M (API)

Why does this matter? Most chats are not hundreds of thousands of tokens long!

Well, 10% of Gemini’s 1m token window is 100K, enough for one big novel input or very roughly 30 serious connected thinking tasks (including input tokens as well); since the AI’s own output tokens count towards what’s in-context, if you want to have a real conversation about a large book you’re still going to have to do it a couple of chapters at a time.

But 10% of 256k (Kimi, Qwen, Minimax, Sonnet) is enough for about a quarter of a big novel or like 8 serious reasoning tasks.

And, again, the Chinese models are token hungry! Not only do they have a smaller bucket, it also gets filled up way faster.

Self-hosting has high fixed costs

3. the weights. On-prem with fully free MIT licence, self-hosting, white-box access, customisation, with zero profit going to the Chinese companies.

Self-hosting doesn’t really make sense unless you’re huge volume or using them for very simple tasks. And most enterprises are not really competent enough to finetune anything.

This is partly a temporary matter: the software ecosystem is underdeveloped for serious high-reliability scaled usage, despite the intense hobbyist interest. (They mostly want it running on a Macbook.)

Too slow for casuals

4. You can get much faster token speeds than the closed APIs.

In the browser, they’re actually slower than Western models. This makes sense; they are incredibly inference bound thanks to chip controls! This would be enough to tank them in the consumer market.

And over API, everyone except Anthropic dominate, even in raw token rate (not counting efficiency):

Censorship and perceived censorship

5. less overrefusal (except on CCP talking points)

There’s a pretty big ick factor to the CCP, and the companies are indeed forced to comply on a range of talking points which offend the West. However, the hosted versions are much worse than the weights themselves. SpeechMap:

But there are uncensored finetunes from reputable names. But then again see (3): it doesn’t make sense for most enterprises to conduct and host finetunes themselves.

If you do a fair test on controversial but non-CCP talking points, there’s a wide spread of refusal rates in both Chinese and Western models.

Nebulous ideology?

6. on topics controversial in the West, less nannying

Lambert notes that the people he speaks privately to are really worried about less obvious stuff, the “indirect influence of Chinese values”.

There is something to this currently (but not a lot given the size of the English internet in the training corpus and the relative lack of soft-post-training skill or effort in Chinese labs):

But it’s valid to assume this will get worse as the CCP get more aware and the companies put more effort into personality and post-training.

Downloading is a long way from productising

8. they’re the most-downloaded open models

People panic about “the flip”, the point at which people started downloading Chinese models more. But this is obviously a terrible proxy for actually managing to use them. (And it’s actually pretty unclear how self-hosting adoption would really benefit China anyway except in prestige.)

For people with any need of real customisation or tiny models, or a scientific ML hobby, or an ideological interest in open-source, they clearly dominate.

TODO: Scrape relative mention over time of LLaMa vs Qwen in Arxiv experiments.

I concede that the secrecy in the West about using Chinese models makes this one weaker as an explanation.

even more insecure?

9. they are mostly not used even by the cognoscenti

Above I mentioned reliability (how low-variance they are, how well they can chain things together). But that’s the easy bit; what about adversarial reliability?

The US evaluation had a bone to pick, but their directional result is probably right (“DeepSeek’s most secure model (R1-0528) responded to 94% of overtly malicious requests [using a jailbreak], compared with 8% of requests for U.S. reference models”).

Someone else talking their book notes that Kimi is “not yet fit for secure enterprise deployment”.

This is obviously a huge problem for any agentic uses, even if the benchmark and default reliability were all fine.

Low mindshare

9. they are mostly not used even by the cognoscenti

It’s hardly cynical to note that most people don’t pick their models by analysing relative performance. Instead it’s largely name recognition and trust, which makes sense for reasons of risk aversion and filtered evidence.

In principle, you can change models by changing one string in your codebase. But in practice if you’re sane you need to do incredibly expensive evals and so there’s stickiness.

The DeepSeek moment helped a lot, but it receded in the second half of 2025 (from 22% of the weird market to 6%). And they all have extremely weak brands.

Also corporations really do settle for inferior products all the time for ass-covering reasons (IBMism). Mindshare translates directly into appeal to the risk-averse.

Corporate compliance is hard

9. they are mostly not used even by the cognoscenti.

Chinese APIs are hard for Western companies to use for legal and quasi-legal reasons.

For the API, DeepSeek sent user information to China Mobile, a state company, which violates all kinds of Western data privacy laws. Even if they’ve stopped, this risk is corporate poison. How can you ever be sure enough?

In a couple of years the EU AI Act will be (nominally) enforceable on the Chinese labs too.

On the quasi-legal side, corporate “vendor risk” programmes often flag Chinese suppliers. This is sometimes because they actually can’t guarantee there’s no forced labour involved.

So why not on-prem? Again, it’s a huge fixed cost and competence-bound and your risk team might still give you shit for it. Lambert:

People vastly underestimate the number of companies that cannot use Qwen and DeepSeek open models because they come from China. This includes on-premise solutions built by people who know the fact that model weights alone cannot reveal anything to their creators.

Political bias

9. they are mostly not used even by the cognoscenti.

There are a bunch of social reasons you might want to avoid Chinese models. You might be protectionist, or sucking up to the ascendent protectionists.

The protectionism of others is clearly enough for people to keep quiet about using them. It is often probably enough for them to just not take the risk in the first place.

I’d include here superstitions about the weights themselves being backdoored.

Vendor risk

9. they are mostly not used even by the cognoscenti

If you look ahead, at future risks to your suppliers, it’s obvious that the export control situation relatively speaks against using Chinese models; NVIDIA is not going to choke off OpenAI.

For API adoption, I also haven’t seen anything about Service-Level Agreements (contracts ensuring uptime) and support from any Chinese lab, but these are easy to make (even if compute crunch means that their uptime guarantees simply must be worse than American ones).

Also again corporate vendor-risk programmes often flag Chinese suppliers for data sovereignty, volatility of PRC law, and export control reasons.

DeepSeek openly use Anna’s Archive, where everyone else is quiet about it. But the American companies offer IP indemnity for users (cover if the models violate copyright in your app), which is nice insurance for a nervous corp with a target on its back. I can’t see anything about the Chinese companies doing this yet.

No compute, no perf

10. they are severely compute-constrained (and as of November 2025 their algorithmic advantage is unclear)

The US has five times the FLOPs as China. (Quality and bandwidth-adjusted it’s probably more like 10x.) On raw hardware, Chinese labs are thus 2-3 clock-time years behind.

What if they’re ten times as efficient though? In January, DeepSeek came out with some exciting and splashy hardware optimisations, probably the fruits from putting HighFlyer’s serious quant devs onto pretraining. But the Westerners responded by getting (even more of) their own quants. I find it unlikely they still have a big algorithmic advantage over the Western labs at this point.

Excess quantization?

11. they’re aggressively quantizing at inference-time, 32 bits to 4

No, I think this one is wrong or else only a tiny factor. gpt-oss was post-trained in MXFP4 which is only 4.25bits.

And I have a strong hunch that many American models are also served in low fidelity, maybe FP4 (4 bits). Quantization just isn’t that bad.

Galaxy-brain soft power??

12. state-sponsored Chinese hackers used closed American models for incredibly sensitive operations, giving the Americans a full whitebox log of the attack!

I can dimly imagine some kind of flexing dynamic in cyberwarfare, where you actually want to show off your attack capabilities, and so you use Claude on purpose. Yes: this idiotic move makes great sense if the apparent targets are red herrings, if Anthropic were the real target. You learn how long their OODA loop is, you learn (by retaliation or its absence) how tight they are with the NSA, you learn a little about how good their tech is.

You could also see it as retaliation for Amodei’s hawkish comments all year. Literally trading effectiveness for embarrassment.

But I don’t really know anything about this.

Overall

Low adoption is overdetermined:

No, I don’t think they’re as good on new inputs or even that close.
No, they’re not more efficient in time or cost (for non-industrial-scale use).
Even if they were, the social and legal problems and biases would probably still suppress them in the medium run.
But obviously if you want to heavily customise a model, or need something tiny, or want to do science, they are totally dominant.
Ongoing compute constraints make me think the capabilities gap and adoption gap will persist.

Interview with Analytics India

INTERVIEWER: You point to a larger performance drop for Chinese models when moving from AIME 2024 to AIME 2025, and you frame that as suggestive of weaker generalisation. If you were to explore that same idea in another domain (like coding benchmarks or natural-language reasoning), what kind of results would meaningfully shift your view?

There's nothing special about AIME, I picked it justbecause it's high-effort and it updates, and so gives us nice properties: 1) novel, 2) of equal difficulty, and 3) the data is "clean".

I could quite easily be persuaded that they generalise better on coding or Q&A than on maths. (The post is "70%" confident.) I don't have time to look myself, but I welcome people superseding me.

Other evidence points the same way though; for instance Qwen2.5 is known to have trained on test for a few benchmarks and collapses on new versions. See also the famous GSM1k case, where flagship Western models actually improved their performance on fresh data.

One problem with the post is that the "tigers" have been improving over the course of this year, and my post doesn't look at the very most recent Chinese models, because they could have trained on AIME 2025. Qwen3 seems less contaminated than Qwen2.5 for instance.

I look forward to repeating this in February on AIME 2026 to see if they've gotten more rigorous.

Your article discusses both model capability and production readiness. If we look at those separately, where do you think Chinese labs are currently closest to Western labs: in raw technical ability, or in reliability/compliance features that matter in enterprise settings?

Closer on capability than product, and closer on product than on legal and quasi-legal compliance.

One thing I didn't cover in the post is non-Western users might find it easier to use the Chinese models than Europeans for a range of reasons (looser compliance regs, less data privacy law, usually less politicisation of the matter).

The popular belief is that Chinese models are only a few months behind Western labs, but you propose a gap of closer to a year. What makes you lean toward that longer estimate, and what would you consider the most unmistakable evidence for readers who are sceptical?

The "three month gap" is calculated by trusting evaluation results naively. I think I have shown that you shouldn't do this. (Note that the Western models also dropped a lot on fresh data!)

This is my subjective guess about an unobserved quantity. I know a little more about the data filtering used in Western labs - and I have shown evidence of recent bad filtering in Chinese labs - and so I make the inference that unseen parts of the model development are also subpar. This is not science, but it's all we have. It's fine to not take my word for it.

There is no unmistakable evidence sadly. We cannot outsource our judgment to benchmark numbers; you really do just have to spend the time to compare the models side by side against your own needs.

Suppose we imagine a world where we could eliminate training-time contamination from public benchmarks. How much would the observed performance gap between top Western and top Chinese models actually change? Would it widen slightly, significantly, or stay roughly the same?

The difficulty is that the models are too smart for this. They can pick up on "semantic duplicates" of the test data (things like "da Vinci was born in 1452" and "the painter of La Gioconda was born in 1452") to cheat despite not seeing the actual data set, and this is profoundly difficult to correct for.

Granting your thought experiment though: I expect it to be wider than my cheap AIME estimate. Mathematics is a relatively clean domain in some ways, and all models struggle even more with messy things, and I expect the Chinese models to be a bit worse still.

Your writing comes across as critical in a measured way, not anti-Chinese imo. Has the social media recactions you’ve seen so far reflected that nuance, or do you feel some readers are approaching the hypothesis through a more polarised lens? What, if anything, has surprised you about the reaction?

Thanks! As I say in the piece ("Filtered evidence") the background discourse is highly toxic and deluded. And I am wary of feeding in to "cope" (being used by people who really don't want Chinese models to be at parity and who allow this desire to determine their beliefs).

I was quite surprised to see powerful people publicly endorsing a mere blogpost. This is good news for the world; blogs remain the alpha of the internet.

The jailbreak argument against LLM values

Sun, 09 Nov 2025 00:00:00 +0000

Bostrom (2014) defined the AI value loading problem as

how could we get some value into an artificial agent, so as to make it pursue that value as its final goal?

JD Pressman thinks this is obviously solved in current LLM systems:

The value loading problem outlined in Bostrom 2014 of {getting a general AI system to internalize and act on “human values” before it is superintelligent and therefore incorrigible} has basically been solved. This achievement also basically always goes unrecognized because people would rather hem and haw about jailbreaks and LLM jank than recognize that we now have a reasonable strategy for getting a good representation of the previously ineffable human value judgment into a machine and having the machine take actions or render judgments according to that representation.

I take issue with this. I agree that LLMs understand our values somewhat, and that present safety-trained systems default to preferring them, behaving like they hold them. 3 5

The jailbreak argument

But here’s why I disagree with him nonetheless: jailbreaks are not a distraction (“hemming and hawing”) but are instead clean evidence that loading is not solved in any real sense:

All LLMs can be “jailbroken”, put into an unaligned mode through mere inference on adversarial text inputs. 4
If a system can be put into an unaligned mode at inference time, then it has not internalised the values.
So models have not internalised the values.
The value loading problem is about getting the values internalised.
Therefore the value loading problem has not been solved in LLMs. 6

I might say instead that “weak value preference” is solved for sub-AGI.

(A deeper analysis would involve the hypothesis that current models don’t actually have goals or values; they simulate personas with values. And prosaic alignment methods just (greatly) increase the propensity to express one persona. Progress has just been made on detecting and shaping such things empirically, so maybe this will change.)

Value-loading vs alignment

Pressman also says that

At the same time people generally subconsciously internalize things well before they’re capable of articulating them, and lots of people have subconsciously internalized that alignment is mostly solved and turned their attention elsewhere.

I initially read this as him agreeing and celebrating this shift, but actually he thinks they’re incorrect to relax, since value loading is only a part of the alignment problem:

solving the Bostrom 2014 value loading problem, that is to say {getting something functionally equivalent to a human perspective inside the machine and using it to constrain a superintelligent planner} is not a solution to AI alignment.

I agree that value-loading is not enough for AGI intent alignment which is not enough for ASI alignment which is not enough to assure good outcomes.

Good discussion on Bsky

Pressman’s response

I sent the above to him and he kindly clarified, walked some of it back, and provided a vision of how to use a decent descriptive model even if it is imperfect and jailbreakable:

I probably should have used the word ‘generalize’ instead of ‘internalize’ there.

(Thus “the value loading problem outlined in Bostrom 2014 of {getting a general AI system to s/internalize/generalize and act on “human values” before it is superintelligent and therefore incorrigible} has basically been solved.”)

The specific point I was making, well aware that jailbreaks in fact exist, was that we now have a thing that could plausibly be used as a descriptive model of human values, where previously we had zilch, it was not even rigorously imaginable in principle how you would solve that problem.

To break this down more carefully:

1. I think that in practice you can basically use a descriptive model of values to prompt a policy into doing things even if neither the policy or the descriptive model have “deeply internalized” the values in the sense that there is no prompt you could give to either that would stray from them. “Internalizing” the values is actually just, kind of a different problem from describing the values. I can describe and make generalizations about the value systems of people very different from me who I do not agree with, and if you put me in a box and wiped my memory all the time you would be able to zero shot prompt me for my generalizations even if I have not “deeply internalized” those values. In general I suspect the LLM prior is closer to a subconscious and there are other parts that go on top which inhibit things like jailbreaks.

If I had to guess it’s probably something like a planner that forms an expectation of what kinds of things should be happening and something along the lines of Circuit Breakers that triggers on unacceptable local outputs or situations. Basically you have a macro and micro sense of something going wrong that makes it hard to steer the agent into a bad headspace and aborts the thoughts when you somehow do.

2. Calling this problem “solved” was probably an overstatement, but it’s one born from extreme frustration that people are making the opposite mistake and pretending like we’ve made minimal progress. Actually impossible problems don’t budge in the way this one has budged, and when people fail to notice an otherwise lethal problem has stopped being impossible they are actively reducing the amount of hope in the world.

At the same time I do kind of have jailbreaks labeled as “presumptively solved” in my head, in the sense that I expect them to be one of those things like “hallucinations” that’s pervasive and widely complained about and then they just become progressively less and less of a problem as it becomes necessary to make them stop being a problem and at some point I wake up and notice that hey wait this is really rare now in production systems. Most potential interventions on jailbreaks aren’t even really being tried because it doesn’t actually seem to be a major priority for labs at the moment if you ask the model for instructions on how to make meth. This makes it difficult to figure out exactly how close to solved it really is. Circuit Breakers was not invincible, on the other hand it’s not clear to me you can “secure” a text prior with a limited context window that doesn’t have its own agenda/expectation of what should be happening to push back against the users with. This paper where they do mechinterp to get a white box interpretation of a prefix attack they find with gradient descent discovers that the prefix attack works because it distracts the neurons which would normally recognize that the request is malicious.

So it’s possible a more jailbreak resistant architecture will need some way to avoid processing every token in the context window. One way to do that might be some kind of hierarchical sequence prediction where higher levels are abstracted and therefore filter the malicious high entropy tokens from the lower levels, which prevents them from e.g. gumming up the planners ability to notice that the current request would deviate from the plan.

And here’s a nice analogy contesting my suitcase word “internalise”:

this word “internalize” is clearly doing a lot of work and something feels Off to me, to say that a text prior which can be manipulated into saying whatever hasn’t “fully internalized” the values. Like if you stripped away the layers on top of my raw predictive models/subconscious that I use for completing patterns and then prompted it, I assume you could get it to say all kinds of nasty things. But also that’s not like, the complete agent.

So if you assumed you can’t build anymore of the agent until you have some way to make the text prior not do that, that’s probably a wrong assumption. One of the reasons I’m annoyed that agents aren’t really a thing yet is that it means we don’t have a good intuitive sense of which parts of the system need to handle what problems.

Post-hoc theory

In retrospect we can see the following distinct problems:

The value specification problem (“how do we describe what we value? how do we get that understanding into the model?”)
We thought this would involve:
a. The explicit value modelling problem (“what precisely do we value?”) - moot
b. The value formalisation problem (“what mathematical theory can capture it?”) - moot, since:

This problem was somewhat solved for sub-AGI by massive imitation learning and (surprisingly nonmassive) human preference post-training. The internet was the spec. This also gave us weak value preference.

The replacement worry is about how high quality and robust this understanding is:
The value generalisation problem (“how do we go from training data about value to the latent value?”) - some progress. The landmark emergent misalignment study in fact shows that models are capable of correctly generalising over at least some of human value, even if in that case they also reversed the direction. 7

Then there’s the gap between usually preferring something and “internalising” it (very reliably preferring it):
The sub-AGI value-loading problem (“how do we make them actually care / reliably use their understanding of our values?”) - not solved, but there is a preference towards niceness.
Tamper-resistant value-loading (“how do we stop a small number of weight updates from ruining the value-loading?”) - not solved, maybe a bit unfair to expect it. You could imagine doing advanced persona steering on top instead.
The general value-loading problem (“how do we get an ASI to learn and internalise a model of current human values which is better than the human one”) - not solved
The value extrapolation problem (“how do we safely improve on current human values?”) - not solved

I’m sure there’s some other deep confusion lurking in this list and I look forward to seeing it exposed, ideally in less than 10 years.

Ways we can fail to answer

Sun, 02 Nov 2025 00:00:00 +0000

In what ways can we can fail to answer a question?

(I mean necessarily fail: actual barriers to knowledge, rather than skill issue hurdles. But of course contingent failures are much more common: “We didn’t ask the question in the first place”, or “We didn’t have the particular insight that would have allowed for productive research”, or “We didn’t manage to remove every cognitive bias”, or “Instrumentation is really hard”, or “We are not rich enough to run this study yet”, or “We worshipped the problem”.)

(I also mean fail exactly; there are often excellent approximations, and we can often legitimately patch over tricky philosophical questions with our unanalysed tacit knowledge.)

Conceptual problems (the question is not a question)

A malformed question or category error: the question may ask for something which doesn’t make sense (e.g. “what colour is justice?”, or “Is sigmoid jealous of ReLU?”, or “Did these things happen at the same absolute time? What are the absolute coordinates of this event?”.)
- Strong incommensurability: the question doesn’t make sense because it mixes frameworks. (e.g. “What is the Einsteinian mass of phlogiston?”, or “What’s the wavefunction of this classical field?”)
Vagueness: the question may fail to mean anything in particular. (e.g. “When exactly did you become an adult?”) 5
False assumptions: we can’t answer it because, while it was clear and made sense, it was wrong from the start (e.g. “Have you stopped beating your wife?” or “Is personality based on nature or nurture?”. 1)

Logical problems (the question has no answer)

Antinomy: it turns out that there is no answer because the question is circular or involves itself somehow (e.g. “Is this sentence false?”)
Incompleteness (“Show that Peano arithmetic is consistent using Peano arithmetic”) 6 and undefinability (e.g. “Is this sentence true in arithmetic?”) are about some proof answers being inaccessible inside systems powerful enough to be interesting and useful. It doesn't come up in normal thought very often.
Mathematical independence: The question is neither provable nor disprovable with these axioms. (e.g. Is there a well-ordering of the reals? Is the continuum hypothesis true in ZFC? What is the value of BB(748) in ZFC?)
Various holes in social mathematics. Important questions like "what's the optimal voting system?", "what does the majority prefer?", "what's a democratic system where people don't have an incentive to vote strategically?", "what's the perfect design for a market?" in general don't have an answer. See e.g. here. 8
Anthropic self-locating effects: you can’t get at the answer because you couldn’t exist to observe it. (e.g. “What’s the probability I’m a Boltzmann brain?”, or “What’s the prior probability of observer-permitting universes?”)

Non-uniqueness is only a problem for questions which ask for the one true answer. (e.g. “What is the electromagnetic potential at this point?”). Many entries in this post are not strict failures to produce answers, they just have a non-unique answer. You can sometimes just parametrise the observer and then get your answer. If non-uniqueness is a failure, it’s a happy one; it just means that you get too many answers and have the nicer problem of picking one.

Ontic problems (the answer is literally inaccessible)

Uncomputability: we can’t answer it because no computer or mind can. (e.g. “Is this random?”, or “Does this Diophantine equation have integer solutions?” or “What’s the shortest Python program that outputs this file?” or “Can you write a program that checks if conjectures follow from these axioms?” Or you asked about the language the problem is in.) 4
Computational intractability: we can’t answer it because it would take too long, including if we turned the universe into a computer. (What’s the best way to schedule these classes, avoiding all conflicts and respecting room capacities and lecturer availability? Or “I forgot my password; crack this file”.)
Inapproximability: we can’t even approximate the answer because it would still take too long. (e.g. “What’s the maximum clique size in this graph?”, “What’s a (log n)-approximation to the chromatic number?”)
Quantum indeterminacy: there’s no answer because (maybe) physics is intrinsically random. (e.g. “When will this radium atom decay?”) 3
Physical constraints, speed limits and no-gos: it can’t be answered in principle because of physics. (e.g. the answer is beyond the cosmological or particle horizon, “what will happen in this galaxy 20 Gly away?” or “Can we measure this state in two bases?”, or “Make a perfectly accurate interferometer”, or “What are the microstates of this black hole?.)
Mixing: the answer is gone; we arrived too late; the system forgot the answer. (e.g. “What was the exact microstate of the gas in this room 1 hour ago?”, “What was the original state of this thermalised system?”)
Non-ergodicity: the answer isn’t reachable because the system doesn’t mix. (e.g. “What equilibrium will this glassy system reach?”)
Measurement problems
- Disturbance or decoherence: we can’t answer it because our instruments disturb the thing in question too much.
- Complementarity: we can’t answer it because the answer was excluded by another question we asked first. (e.g. “where is this electron and how much momentum does it have?”)

Epistemic (we cannot get at the answer)

underdetermination or unidentifiability. (e.g. Is spacetime fundamentally Lorentzian? Which interpretation of quantum mechanics is true? Why do physical constants look fine-tuned? What was the ancestral DNA sequence at some past generation?)
Chaos: we can’t answer it because we can’t measure the initial conditions well enough to predict large systems. (e.g. Where exactly will this double pendulum be in 100 Lyapunov times? What’s the weather like 3 months out?)
Computational irreducibility: getting the answer is inseparable from running the system, the question has no shorter answer than a full simulation (e.g. “What will this cellular automaton look like in 10^100 steps?”)
Hidden variables: we can’t ever observe the actual variables (e.g. What are the simultaneous values of σ_x and σ_z for this electron?)
Cognitive closure: we’re not smart enough to answer it / we lack certain faculties (like echolocation say). (e.g. “What is it like to be a bat?” It kinda looks like quantum gravity could also be an example.)
Cognitive bias: we’re not rational enough to answer it (e.g. “How biased am I?”)
I suppose I should mention the original “epistemic barrier”, the putative conceptual barrier between the mind and the world.

??? (the question is logical and/or ontic and/or epistemic idk)

Observer effects
- Back action (e.g. What’s the pre-measurement spin of this electron? What are the “real” observables in this quantum system? or “Measure the full-speed time evolution for this system”.)
- Theory-ladenness:
  - Semantic theory-dependence: Our observational vocabulary already presupposes theoretical commitments that prejudge the answer (e.g. What’s the rest mass of an electron, without assuming special relativity? What’s the ‘real’ temperature of the CMB independent of blackbody theory?)
  - Perceptual theory-dependence: Our perception is shaped by theoretical expectations, so we cannot see the answer “directly” (e.g. What do electron tracks really look like? What does this fMRI show before we apply the hemodynamic response model?)
Future contingents: it doesn’t have an answer yet so we have to wait. (e.g. “What lottery numbers will win next week?”)2
Hysteresis: we can’t answer it because we weren’t there at the start and the system remembers. (e.g. What’s the magnetic moment of this material at field strength H? When will this old rope snap? At what temperature will this water freeze?)
Reflexivity and strange loops: the thing in question is self-referential or causally dense, or it [changes](https://en.wikipedia.org/wiki/Demand_characteristics) when answered, and so can’t be picked apart. (e.g. “What’s the best method for finding the best method?”; “Which level of description is fundamental in this self-referential system?”, “Which part of your mind is the real you?”.) See also antinomy.

Finally there is the great risk this post takes (and all not-totally-technical writing):

Model error leading to spurious impossibility: you apply an analytic impossibility result (like the above) to a "synthetic" real system where it doesn't apply. You overinterpret a narrow thing; you assume the world fits the assumptions of the formal proof but it doesn't; you summarise a technical result in natural language and imply that it's more general than it is. Most invocations of Gödel are spurious;

You will have done really well if even once in your life you fail to answer a question for these reasons. Getting so far means you have avoided hundreds of punji traps, claymores, nerve gasses, madnesses.

What god has to do in mathematics

Tue, 28 Oct 2025 00:00:00 +0000

He picks from uncountably many sets simultaneously without an algorithm.

He preserves us from sets of measure zero and the non-measurables. In his loving arms infinity minus infinity equals whatever we need it to. He takes five loaves of spheres and two small spheres and feeds the host.

He makes the inaccessible cardinals. He stops the inaccessible cardinals from accessing us

He is risen, he is lifted, and he is Born.

He transubstantiates our mortal filters into ultrafilters for free.

He knows the truth of the continuum hypothesis but saves us from it.

He lets us pretend we can do functional analysis, keeping our intersections dense by tucking away the invisible point we could never reach.

Quietly he makes actual of potential infinity.

Silently he transfers the evaluation strategy from heaven (the meta-language) to earth (the source language).

He forgives us our renormalizations.

When we are weak and know only subspace, he carries our functional to the full space.

Ultimately he makes it all cohere. The accidents of our conditionals are treatable as unconditional. “I am that I am”, what makes the axioms hold.

Various atheists try to do without him and do not prevail.

The joke was initially supposed to be about actual mostly-unacknowledged holes in the theoretical structure: ridiculously strong assumptions, suppressed premises, things we ignore and which we get away with.

But Choice is too well-known and contested to be a great example and I ended up just covering a bunch of nonconstructive results, which aren't as mysterious as I'd like to justify the "god did it" gag.

Basically I don't know obscure unacknowledged issues. But someone who does could write a good version of this post.

prospective science please

Tue, 28 Oct 2025 00:00:00 +0000

Time… burns without leaving ashes

— Elsa Triolet

Case study: whither Google Search?

For now, Google is the premier information infrastructure on Earth. It is the most-visited website. It handles 14 billion search queries a day, nearly two for everyone on earth. 1 As of writing it’s more than the traffic of ChatGPT, Claude, Wikipedia, Bing, and Duckduckgo put together. 1 / 500 of your waking life is spent on this site, probably, and its decisions determine how you spend much more time than that. 3

Google Search effectively controls the epistemology of the world — @atomgardner

So… how’s it doing? Is search quality changing? Opinions differ but it does seem to have gotten much worse over the last decade.

OK unc, but surely we have actual quantitative studies of something this important?

Broadly: no. In the last 25 years, I could only find one major empirical study measuring search quality (Bevendorff et al. 2024). It’s fine, and they deserve huge credit for doing what no one did, but it doesn’t really answer the question and only covers 2022-3, well after the subjective decline. We had two decades to notice that this was really important and to collect data about it ahead of the decline. We didn’t.

I complain about academia a lot, but this one failure dwarfs almost all others. We have better longitudinal data on bird migration than on the modal channel for belief formation. Academia and civil society had 25 years to run this and didn’t, and our chance to measure it is gone. (Google could maybe still do this study given the logs, but why would they?)

In fairness

There's a huge attribution problem (was Google degrading or were they accurately representing the web itself degrading?

Google's quality is constantly under attack by the SEO industry and the bots. This would be enough to cause the observed decline without Google doing anything wrong per se. They invest large amounts into countermeasures. It doesn’t work (see this identical effort from 2022).

The defence against the first (non-AI) wave of SEO spam already cost us a lot:

in 1999… When you looked for information about how to tell if your bread is rising correctly, or about South Korean cement manufacturing, or the musical influences on Igor Stravinsky, or whatever weird thing, Google would pull up high-quality reference material, or blogs and BBS arguments among disagreeable weirdos who specialized in the subject… A cottage industry arose of finding some search term and churning out low-cost copy on the subject in order to serve ads to people trying to find real information. Specialists in “search engine optimization” popularized their techniques as consultants to big companies, and before long this became standard practice. In their efforts to keep these problems from getting totally out of hand, Google and other search engines weighted search results towards a whitelist of standard lowest-common-denominator websites. The long tail of the internet could no longer be found from a simple search.

Then there's the Web 2.0 turn to walled gardens, platforms which search engines can't really index. That's mostly not Google's doing.

In unfairness

It looks like Google itself systematically degraded search by:

Making ads look identical to organic results (2011-2020)
Zero-click answers that keep users in Google's walled garden
Prioritizing engagement over quality (the "code yellow" pivot of 2019)
Soft whitelisting of large publishers
Decreasingly terrible forced AI content

As the noted scholars Brin and Page said in 1998:

The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is "The Effect of Cellular Phone Use Upon Driver Attention", a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98]. It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the Consumers.

... we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm.

The evidence we have

Our one actual datum: Bevendorff et al. 2024

That one study, "Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search Engines", monitored 7392 queries for particular product reviews on Google (proxied using Startpage), Bing, and DuckDuckGo over one year (2022-3).

It's fine but not amazing evidence:

Just one year, and a post-decline year at that
Just product searches
Just a couple of thousand products
Focussed on spam alone
Using a dumb mechanical proxy measure for diversity, itself as a proxy for quality.

Still:

29% of frontpage Google results had affiliate links compared to the random-page baseline of 2%. (The other two search engines were even worse.)
That's not totally damning - Dwarkesh has affiliate links - but it is a warning sign: pages with more affiliate links were worse on a mechanical measure of quality.
23%% of frontpage Google results were outright spam/review farms compared to the random-page baseline of 13%. (The other two search engines were even worse.) This is based on manual annotation, god bless them.
The buried conclusion is that the very worst spam on Google actually decreased over the study period. The 95th percentile of affiliate links per page fell from 50(!) to 35(!).
But improvements are short-lived. They found that spam is cyclical (what they call "breathing patterns", like the rise and fall of your chest when you breathe): spam gets into results, the engines update their algorithms to squash them, spam returns in new forms.
"Text quality" was decreasing in all search engines.
Overall this study is just too short to measure a nonlinear phenom.

What's hard about it?

Measuring search quality properly requires:

Access to search results at scale. Google doesn't provide any examples, so you need to actively scrape it over time yourself.
Longitudinal data (because one-off snapshots don't give you effects and miss adversarial dynamics)
sampling representative queries. We should hit common queries and types.
Baselines: you need to hit multiple search engines (to see if they're also struggling) or to randomly sample webpages (to separate "Google getting worse" from "internet getting worse")
Quality is not a hard endpoint: Quality is subjective, and there's a higher bar in academia for handling such things. The valid reasons to worry are massive measurement error, low test-rest reliability, cross-cultural and intra-cultural heterogeneity...

But many of these things are now hundreds of times easier with LLMs!

Leontiadis et al 2014

This is just a study of one particularly obvious and aggressive kind of spam: "search poisoning" redirection attacks where the spammers buy a previously legit URL and send you to their shit.

They tracked a set of pharmaceutical and product queries for 3.5 years (2010-3).

The rate of the frontpage having one or more of these particular attacks jumped from 30% to 60% of results. But it seems to have gotten a lot better after the EEAT update.

Philipp et al 2014

Analyses logs from 190,000 users over a 6 month period (late 2012 - early 2013) but only for health queries. This is one way that Google obviously got "better" over the last decade, though in a crank-minimising way rather than an info-maximising way.

Proxies: Zero-click searches

Possibly-unrepresentative tracking data from Datos shows a huge rise in "zero-click" searches, queries that never leave Google.

	Zero-click rate
2016	44%
2020	65%
2025	68%

But this is a bundle measure: it mixes up the results being obviously too crap to bother with and Google sucking up the information from third-parties and so diverting the traffic to themselves. (Google Flights, Google Maps, featured snippets have gotten better; the Google Knowledge Graph is actually often useful; and the AI Overviews are sometimes useful and even more often falsely appear useful.)

Proxies: UI changes as tells of motivation

This excellent chart from SearchEngineLand shows how, starting in 2011, Google Ads have been steadily made more stealthy and resultlike. Click it for full-size:

Proxies: Journalism

We're reduced from science to journalism. To weakly understand what happened we have to read dreadful people like Ed Zitron, who blames one guy, Prabhakar Raghavan (as if the org didn't give him the incentive). They did indeed panic and prioritise ads over actual content, and the timing (2019) does correlate with some of the subjective decline. But that's all we can say from this.

Proxies: Straw polls from memory

Terrible Proxies: User engagement

Despite all of this, Google's volume metrics are up 20% YoY. Searches per searcher at historic highs.

If search was getting worse, why would people be using it more?:

1. Normalisation of deviance: maybe users lowered their expectations
2. Query inflation: maybe it takes more searches to find useful results
3. Mobile growth: maybe spending more time on mobile means you just increase your total screentime and so googling time.

Tiny study: Bloggers

Dan Luu also did a manual test on six tricky queries and six search engines. Google was second-worst and didn't get any of them. On two of them no search engine worked at all. On some axes this is better than Bevendorff.

Sum total

Overall it sure looks like it got worse but we don’t really know how much.

What can we do about it?

Project TODO: Archive reconstruction

You could probably do some limited good by finding common queries which happened to get tracked on the Internet Archive:

Project TODO: The second-best time is now

You can in fact start pinging Google every day with the same query and saving the frontpages they give you.

Public search archives: Either Google preserves and makes available historical search results for vetted researchers or we build it ourselves.
Crawling infrastructure: Sustained funding for researchers to continuously monitor search quality across engines and domains. Relying on sending it to the Internet Archive is one cheap way but fairly buggy.

Some possible research questions on Google:

How has search quality changed for non-commercial queries?
How does quality vary by domain (health, finance, news, technical)?
What fraction of queries return primarily SEO spam vs. “authoritative” sources?
How much of this is caused by Google?
How effective are Google’s (announced) algorithm updates at actually improving quality (vs. temporarily shuffling spam)?
How strong is the soft whitelist? How far down the ranking have blogs slipped now?
Has the rollout of AI Overviews (2024) improved or degraded quality?
How much of the problem is Google’s choices vs. the degrading web?
Are they picking up on publicised test queries and doing whack-a-mole on them?

And maybe you want to start logging ChatGPT now before they add ads.

Conclusion: We need prospective science

That latter project makes me realise that a whole kind of science is missing: watching the world and collecting data before things happen. 4

Most academic studies look backwards. (RCTs look forwards but at enormous expense on tiny questions.) But scraping is now hundreds of times cheaper than it used to be. Dumb first-pass classification and scoring is now thousands of times cheaper. Let’s just start collecting.

Look around you. Think of the most important digital services and phenomena in the world.
Come up with hypotheses about what could happen to them. Put them on OSF to tie your hands.
Think really hard about the design - this bit is irreversible if you want consistent, backwards-compatible results. Consider interventions as well as constant baselines.
Start scraping.
Check back every year.
Iterate on the design if you must.

NY sounds

Thu, 23 Oct 2025 00:00:00 +0000

There are roughly three New Yorks. There is, first, the New York of the man or woman who was born here, who takes the city for granted and accepts its size and turbulence as natural and inevitable. Second, there is the New York of the commuter — the city that is devoured by locusts each day and spat out each night. Third, there is the New York of the person who was born somewhere else and came to New York in quest of something.

Of these three trembling cities the greatest is the last — the city of final destination, the city that is a goal. It is this third city that accounts for New York’s high-strung disposition, its poetical deportment, its dedication to the arts, and its incomparable achievements. Commuters give the city its tidal restlessness; natives give it solidity and continuity; but the settlers give it passion. And whether it is a farmer arriving from Italy to set up a small grocery store in a slum, or a young girl arriving from a small town in Mississippi to escape the indignity of being observed by her neighbors, or a boy arriving from the Corn Belt with a manuscript in his suitcase and a pain in his heart, it makes no difference: each embraces New York with the intense excitement of first love, each absorbs New York with the fresh eyes of an adventurer, each generates heat and light to dwarf the Consolidated Edison Company.

— EB White

I want to capture the best music in the world. (This will include little “world music”.) But I hit a snag - New York. It is just too huge to capture. (Here’s what I have so far.)

Why? What would a complete New York playlist have to cover? If you limited yourself just to totally new genres:

1885-1930: Pop as we knew it (Tin Pan Alley)
1910-1950: Big band and swing. Duke, Calloway
1910-1960, 1980-2019: The (American) Musical: Berlin, Gershwin, Rodgers and Hammerstein and Hart, Bernstein, Kern, Wodehouse(!), Sondheim, Brown
1940-1967: Bebop, hard bop, cubop, cool
1943-1970: “Latin” jazz (Afro-Cuban, Puente, Barretto, Palmieri…)
1930-1956: Globalised samba and calypso.
1940-1970: Folk revival
1958-1964: Pop as we know it (Brill machine pop rock, and thus, ultimately, K-pop) (Leiber-Stoller, Carole King et al)
1960-1980: Various Avant-Gardes (Varèse, New York School, the Downtown scene, loft jazz, Fluxus, No wave, Minimalism, Avant Rock, Fluxus again, Free Jazz…)
1970-1980: Disco (Studio 54)
1980: Globalised dub (Gibbons among the first Americans to incorporate techniques from dub production into dance music)
1973-Present: Hip-hop
1976-1980: Punk. Post-punk slightly preceded punk.
1980-1987: Garage house
1985-?: Antifolk
1998-2007: Garage rock revival
1985-?: Totalism (the opposite of minimalism)
2006-2011: Bloghouse and dancepunk, witch house, whatever.

Tech

You’d also want a separate playlist for Technical New York: the engineering breakthroughs. Spotify isn’t up to this though.

first electromechanical musical instrument
Stereophonic recording
theory of amplification
electrical recording
first commercial synths, first keyboard synth
electret condenser microphone - what 90% of all current mics are.
algorithmic music
first fully electronic film score
first widely-used computer music program (MUSIC-N)
sliding faders
Spector wasn’t the first to use the studio as an instrument but he cemented it
breakbeat and scratching
the extended remix and 12” single
the breakdown

Current

Scenius seemed to stop happening at some point early in this century. What was the last musical movement to be distinctively associated with a city in Britain? Trip-hop, from Bristol? Grime, from East London? Once you get beyond 2010 it’s hard to think of any. In the US, New York is a more liveable city than it was for much of the twentieth century, and a less creative one. Portland and Austin are not quite what they were. San Francisco and Silicon Valley are perhaps the last examples of scenius, but those scenes are driven by money, not art.

- Ian Leslie

And now? It’s still a great city. What new sounds is it cooking?

Brooklyn drill: a minor variation on UK drill. A$AP Mob: a minor variation on cloud/Southern rap. Rage (2hollis and Nettspend): a minor variation on drill.
Outsider house
New opera (Mazzoli, Lang, Wolfe, Du Yun, Muhly)
More free jazzes and the “new Brooklyn complexity” (Mary Halvorson, Matana Roberts, Tomas Fujiwara, Ches Smith, Ingrid Laubrock).
Post-minimalism (Oneohtrix Point Never)
Brooklyn noise. No wave plus

Solid, but not the same. Something ran out. Guesses:

loss of cheap rents and squats. Relatedly: loss of discomfort and chaos.
loss of the wizards. Maybe deindustrialisation means that the engineers aren’t colocated with the artistes, so New Yorkers don’t get the stream of pre-market prototypes they had for 80 years. No Bell Labs and maybe not many Bebes and Louises.
loss of some of the wealth gradient? 70s New York had both extreme poverty (and so cheap spaces) and extreme wealth (and so patronage, whether from slumming trust-fund kids or moneyed venue donors)
loss of (colocated) gatekeepers you could party with and extract attention rents from
loss of industrial zoning allowing big noises. loss of lax enforcement
Institutionalisation of weird music. (All of the above new genres were either deeply commercial or deeply street and subcultural.) Now:
- academic capture of alternative music? Careerification, taming, sexlessness?
- philanthropic capture of alternative music? Grant applications and awards legibilise.
But probably the strongest factors are the universal ones:
- ideas getting harder to find?
- loss of modernist aggro?
- retromania, catalog dominance
- the general dematerialisation of social energy Leslie alludes to. The extraction and dissipation of cultural meaning through homogenous phones located in any mere geography. The loss of friction and the loss of mystery and the loss of random physical collision.

AI breakthroughs 2024-2025

Mon, 20 Oct 2025 00:00:00 +0000

So what just happened?

2024

In January, the best systems were around chance (30%) on GPQA (hard science). By November, 4o was getting “around PhD” (60%) on the hardest Diamond set.
Feb: first “million-token” context window (Gemini 1.5), but there’s a steep fall in performance as you go through it.
Feb: first proper text+audio+image model (Gemini 1.5)
Feb: first good video generation (Sora)
May: first good voice interface (Advanced Voice Mode)
Jul: silver medal at the IMO (AlphaProof) but with a Frankenstein hybrid system.
🚨 Sep: RL works on LLMs at last. So-called “reasoning” (o1-preview)
Oct: GPT-4 “Search” agent. Works better than modern Google.
Oct: So-called “agency” (Anthropic Computer Use). Supposedly a 15 min human-task horizon but really a much shorter for usable horizon.
- 🚨 Whatever METR’s time horizon is measuring admittedly multiplied by a factor of 7 over the year.
Nov: the Model Context Protocol standardises the interface between agents
🚨 Over the year, the best systems jumped from 5% to 50% on SWE-Bench Verified (real coding tasks).
Dec: the first LLM that handles streaming video input (Gemini Live)
Dec: Gemini Deep Research is the first strong agent (for lit-reviews) but no one notices.

Sheesh.

But after about 3 months we got used to the hard benchmark values being 70% or 80% instead of 10% or 20%. It meant less than we thought it would.

2025

¯\(ツ)_/¯ _Jan: Deepseek R1 hype. But the apparent 10x per-token saving is a false economy.
🚨 Jan, recursive improvement: Sometime around here, Deepmind uses AlphaEvolve (Gemini 2.0) to write GPU kernels and speed up the training of Gemini 2.5 by “1%”. (2024 model.)
Feb: Claude Code. The ~end of brittle stupid RAG. Lab incursion into the application layer.
Mar: Autoregressive image generation surpasses(?) diffusion models.
🚨 Apr: o3 is the first LLM ever that is actually worth using for me
May: The first video generator which creates synched audio (Veo 3). Makes the psychological effect 100x stronger.
¯\(ツ)_/¯ In _June 2024, 4o got 5% on ARC-AGI. By Apr 2025 o4-mini got 41%. Also for the first time you can convert $3m per-run into 80% on it. (Human is “98%”)
🚨 Jul: IMO gold by an LLM. Reportedly no tools and no neuralese involved. (“Gemini 2.5 Deep Think Advanced” + unnamed experimental OpenAI model)
Sep: Various groups racing headlong into “frontier biology” with protein language models and such.
Nov: Gemini 3 and Opus 4.5.
- Gemini: Notable improvement in vision and image generation. Rest is a very mixed bag. Benchmaxxed, or rather narrow-objective-maxxed.
- Opus: TBD.
¯\(ツ)_/¯ _Nov: claims about Gemini 3 adding a continual learning mode (based on the little HOPE experiment). We’ll see.
Over the year, progress from 50% to 77% on SWE-Bench-Verified. But you can’t just use this (“+27% is less than the +45% last year”) to say slower latent progress, since obviously the tasks solved this year were much harder.
Over the year, whatever HCAST is measuring multiplied by 5x this year (vs 7x last year).
Browser agents aren’t adopted by anyone except Tyler really.
Various noises about AI scientists. Mostly rediscovery and recombination?
Real progress in AI assistance for research mathematics.
There will be things which I miss / which only reveal themselves as significant next year.

No earthquakes this year (the big scaling hopes, GPT-4.5 and Grok 4 both disappointed their masters) but still prettty fast.

This list is biased towards discrete changes. While the story of AI does have step changes (like GPT-2) and may have more coming, you should do plenty of staring at smooth graphs as well. A lot of the 2024 breakthroughs above were more like progress in UIs or products than something fundamental (though getting multimodal to work was once the definition of fundamental).

PS: That title should be “LLM breakthroughs”, sorry; see here for other AI.

argmin gravitas

Transnormalism

Transhumanism?

Dickey–Wicker

Ancient enhancement

Other tech

Genetic

Surgery

Electronics

Normal exceptionalism: the escape from morphological freedom

Medicalisation and demedicalisation

You are like a little baby

Actual data

AI in 2025: gestalt

tl;dr

Capabilities in 2025

Arguments against 2025 capabilities growth being above-trend

Arguments for 2025 capabilities growth being above-trend

Evals crawling towards ecological validity

Safety in 2025

Are reasoning models safer than the old kind?

The looming end of evals

Prosaic misalignment

What is the plan?

Things which might fundamentally change the nature of LLMs

Emergent misalignment and model personas

Monitorability

New people

Overall

Discourse in 2025

Cruxes for next year (with Manifold markets):

Cuts

My politics

See also

Comments

Paper AI Tigers

And yet

"Tigers"?

Filtered evidence

What could explain this?

Maybe the evals are misleading?

AIME 2024 vs 2025 Model Performance(using the Artificial Analysis harness)

Details

Qwen 2.5

Kimi 1.5

Unreliability?

Harder to elicit?

Tokenomics: no effective discount

Self-hosting has high fixed costs

Too slow for casuals

Censorship and perceived censorship

Nebulous ideology?

Downloading is a long way from productising

even more insecure?

Low mindshare

Corporate compliance is hard

Political bias

Vendor risk

No compute, no perf

Excess quantization?

Galaxy-brain soft power??

Overall

Interview with Analytics India

The jailbreak argument against LLM values

The jailbreak argument

Value-loading vs alignment

Pressman’s response

Post-hoc theory

See also

Ways we can fail to answer

Conceptual problems (the question is not a question)

Logical problems (the question has no answer)

Ontic problems (the answer is literally inaccessible)

Epistemic (we cannot get at the answer)

??? (the question is logical and/or ontic and/or epistemic idk)

See also

Ideological and social problems

Conceptual Problems: the question is broken

Logical Problems: no answer exists

Ontic Problems: the answer is physically inaccessible

AIME 2024 vs 2025 Model Performance
(using the Artificial Analysis harness)