argmin gravitas

A Life Really Worth Living

Fri, 03 Jul 2026 00:00:00 +0000

This class is about a fatal trap that lies in wait for the brilliant but unwary young person: success; other people’s definitions of success. I have seen your CVs. They are too long. You are in danger.

What do we mean by success

“Excellence”: Ordinary success. Pejorative. Default. extrinsic rewards. money, status, fame, Ivy League degree, fancy title, proximity to power
“Greatness”. What do you think of the operational definition “Doing something no one has ever done before?” Indeed: we need “instantiating of a great tradition” as well
Self-fulfilment. whatever you intrinsically want including all of the above iff they are intrinsically valuable.

These three things are connected! Becoming the world’s greatest violinist is great, and is often rewarded with the laurels of excellence, and hopefully is also quite fulfilling (though top performers are often marked by constant dissatisfaction). Excellence gives you resources. Greatness is often quite fulfilling (because it’s rare to keep up the necessary motivation without intrinsic fulfilment as an intermediate reward).

A simple hack is to see if anything is in the triple intersection for you.

Interactive: 1. What do you want in life? (5 mins)

5 minutes silent writing please.

While you’re thinking I will list some examples.

Scientific discovery, art, parenting, happiness, freedom, chill, Enlightenment, fun, safety, doing something genuinely new, retiring early, saving the world, varietymaxxing. Abundance, achievement, adventure, affiliation, altruism, apatheia, art, asceticism, austerity, autarky, authority, autonomy, beauty, benevolence, bodily integrity, challenge, collective property, commemoration, communism, community, compassion, competence, competition, competitiveness, complexity, comradery, conscientiousness, consciousness, contentment, cooperation, courage, creativity, crime, critical thinking, curiosity, democracy, determination, dignity, diligence, discipline, diversity, duties, education, emotion, envy, equality, equanimity, excellence, excitement, experience, fairness, faithfulness, family, fortitude, frankness, free will, freedom, friendship, frugality, fulfillment, fun, good intentions, greed, happiness, harmony, health, honesty, honor, humility, idealism, idolatry, imagination, improvement, incorruptibility, individuality, industriousness, intelligence, justice, knowledge, law abidance, life, love, loyalty, modesty, monogamy, mutual affection, nature, novelty, obedience, openness, optimism, order, organization, pain, parsimony, peace, peace of mind, pity, play, population size, preference fulfillment, privacy, progress, promises, property, prosperity, punctuality, punishment, purity, rationality, reliability, religion, respect, restraint, rights, sadness, safety, sanctity, security, self-control, self-denial, self-determination, self-expression, self-pity, simplicity, sincerity, social parasitism, society, spirituality, stability, straightforwardness, strength, striving, subordination, suffering, surprise, technology, temperance, thought, tolerance, toughness, truth, tradition, transparency, valor, variety, veracity, wealth, welfare, wisdom.

Class share their examples

Interactive: 2. What’s missing in the world? (3 mins)

Same question rotated outwards. 3 minutes silent writing please.

Class share their examples

Quiet desperation

Interactive: Of the adults you know personally, how many work on stuff they find interesting? Great? How many are overall contented?

Right, I’m glad we agree.

On average the world has a way of leading us away from greatness and self-fulfilment, even among the doubtless capable and unusually well-resourced adults in your unusual lives. There is a deep gravity well which leads towards dissatisfaction and mediocrity. It is actually deeper for talented people like you. But you also have more thrust to escape with.

Why?

Fear

For most people, uncertainty is painful → ladders. Another way of saying this: for most people, perfect freedom is quite horrible. 25-50% of PhD students drop out, and that’s among the subsets of humanity most hardened to uncertainty.

You are born. You gain awareness and skill. You are faced with a huge practically-infinite-dimensional space of possible lives. How do you search such a thing?

Interactive: Well, how do you solve a problem with too many dimensions?

Many good suggestions like sampling, gradient descent, searching within a promising subspace, assuming linearity, using evolved or learned heuristics.
Sampling actually fails in theory but works in real life.

Yes. However, I claim that the most common algorithm is instead pick a single socially-reinforced axis and live within it.

I call artificially flattened, fixed, socially-reinforced paths through life ladders.

From the perspective of reducing fear, they have many nice properties:

Precomputed
Static
Low-dimension. Often close to 1 dimensional (promotion through the ranks)
Piecewise
Unambigous

People get scared or fail to search for themselves so they climb ladders.

Corporate ladder - L7 ($1m a year), C-level, first class flights, PA, suits
Government ladder
Academia ladder - how many of you know how tenure works?
Nonprofit ladder - grand speeches and unbelievable inefficiency
Startup ladder - TKS
EA ladder
Radical dropout off-grid ladder
The ESPR ladder

Interactive: Which ladder are you most at danger of climbing?

I am sorry to start talking about jobs, but the fact is that your choice of job is probably the second-most influential decision on your ability to achieve your intrinsic and especially extrinsic values. (After #1: who you choose to spend time with.)

How do people spend their lives?

Let’s look at the distribution of jobs in the global population; among “elite” undergraduates (R1s); among people like you (ESPR alumni); and among this year’s ESPR instructors.

	Global	Elite undergrads	ESPR alumni	ESPR 2026 instructors
Academic	0.40%	30%	22%	8%
Unemployed	5%	2%	10%	15%
Self-employed	25%	10%	12%	22%
Large corporation	15%	25%	18%	8%
Small business	39%	8%	10%	8%
Startup (equity)	5%	10%	20%	16%
Charity	1%	10%	6%	15%
Government	10%	5%	2%	8%

Despite being 0% full-time academics, crucially, we are also 23% PhD holders plus 23% PhD dropouts. We know academia well. Crucially we are also 50% full-time researchers. We love research. What does this gap between 0% and 46-50% tell you about us? About academia?

Now consider the instructors distribution. i.e. 70% of us rejected ladders. This should make you think both “ladders bad” and “ESPR are weirdos who I maybe shouldn’t listen to”. Staff differ from the global population at p=0.002. But you already knew that.

That said, by KL-divergence, Jensen-Shannon and total variation, ESPR alumni and staff are the closest fit. So you just need to work out if you’re not an atypical alum to listen.

Path dependency

Golden handcuffs. “Accustomed quality of life” as one-way ratchet.
“Too old” -> can’t retrain
“I should get a job in my major otherwise it’s a waste”
Tenure bandwagoning → death
Dependents → need money

Status

You and I are apes. That is to say, we are intensely social and intensely status-seeking beings. Most of what we do has some component of looking good and competition and conformity. This is not intrinsically bad. But we go a little crazy when we are starved of it, as young people are.

Status = doing things because other people value them

e.g. You are all overrating the extrinsic rewards of college. It probably washes out to some degree - “employer learning”.

The terrible thing - the diabolical thing - about status is that talking about it creates pressure towards it. No way around it. I’m sorry. The best we can do is make intrinsic goods high-status.

Settling

To stay sane, the minds of adults look like this over the course of their lives:

Comfort is both the goal and an enemy.

Ignorance

Ignorance of the option set

The first time I heard of Olympiads I was 25yo.
The first time I heard of ESPR I was 30yo.
I put off starting a PhD for years because I assumed that PhDs needed to be smart. Actually the main predictor is pain threshold.

Permission

Miscalibration about skill requirements
A missing self-image: “I can do things without asking permission”

I’m the first person in my family to go to college. Grandparents all farmers. Mother a cleaner. I was also shy and depressed for a lot of the first 20 years. It took me a while to get going.

Several times in my life, people have tapped me on the shoulder and told me that I wasn’t living up to my potential.

I was working in London finance, doing “very well”, not working very hard. But the work was meaningless and the corporation I inhabited was extremely dysfunctional. I told a researcher I met that I didn’t think I was good enough to do a PhD. He scoffed and said that I was obviously far above median.
A bit later, I was a researcher, I was in a PhD. But no one was really paying any attention. I sent a famous public intellectual my work. He sent me $30,000 and promoted my work to millions of people. This led to me starting my company, working on incredibly fun things all the time, and never worrying about money ever again.
At the same time, I didn’t have a circle. I had friends but no collaborators, no one who cared about the things I cared about, no one pushing me to do concrete better work. I didn’t think of myself as a leader. I went to Czechia and met this crazy bunch of researchers and meditators and programmers. It was one of the most fun weeks of my life. Some time later, they asked me to lead this camp for teenagers they run.

I was waiting for permission and was deluded about how much it takes to be great, to do actually new things.

I hope this class is the last permission you need. I hereby give you permission to live a good life in any way you choose.

the Egregores Lagrange (AKA checks and balances)

One way of viewing yourself in society is that you are a tiny mote of dust in a system of vast grinding gears. Bureaucracies want your time and obedience. Corporations want your money. Governments want your money and your obedience.

It is tragic that we need governments, that we can’t trust markets to deliver what everyone needs.

It is tragic that we need markets, that we can’t just trust government to deliver what everyone needs (let alone what they want) and to not crush the Other.

It is tragic that we need corporations, that economies of scale are so important that we must risk monopolies and cronies and skinwalkers.

It is tragic that we need bureaucracies, that the human urge for favoritism and self-dealing is so strong and ruinous that it’s worth imprisoning everyone on earth in a cage of stupid rules.

It is tragic to be forced to choose to not be free.

And yet - Who knows what a Lagrange point is? Societies are free to the extent that they create Lagrange points in between these hungry forces of gravity.

Sometimes remember this viewpoint, that society is trying to get you to do things you don’t and shouldn’t want to do.

tame the algorithm lest it tame you
tame the incentives around you
tame your parents
tame your compulsions
tame espr

Ways to identify fake ambition

Riff on this.

Ways money makes things worse

Riff on this

You have options

Student: “but we have no choice, we need to pick a ladder”.

No. Options exist always.

You could not have a job. Bohemians trade income and security for freedom. My brother lives on about $5000 a year.
You do not need credentials to do things. The European CDC runs me and my friend’s code. We are not epidemiologists.

Most old or large orgs are horribly ladderified. But conversely new things are not!

So start something!

New companies are sometimes great
New government agencies are often great
New artistic scenes are alive and filled with real ones

What is the good life?

ability to fulfill your intrinsic values
ability to resist centres of wealth and prestige if you choose
ability to minimise bullshit
ability to endure discomfort in service of a goal if you actively and reflectively choose to

What can ESPR do for you? What can you do for the above this week? (5 mins)

help finding who to contact (profs, founders, anons)
help writing cold emails
help starting a blog and plugging it
help calibrating ambition level correctly (usually up)
help finding funding
help developing mastery (maths, forecasting, public speaking, whatev)
scholarships
detailed research feedback
finding collaborators

You fail this class if you do not ask for something

Damage control

Most useful work in the world occurs inside ladders. You cannot scale without a ladder.

The world would not function if everyone just did what they wanted. Most unpleasant jobs nonetheless help the world, sometimes enormously – consider the example of Prime Minister of the United Kingdom. My point is that you need the ability to steer away from centers of wealth and prestige if you want to.

Interactive: who is planning to climb a ladder?

One thing you might take away from this class is that you need to do bigger things, put even more pressure on yourself. No! I am telling you that you have that option, you are more powerful than you think. You should view this as a PARTIAL argument for trying an independent life. The forces that suggest you climb ladders are legion and as soon as you leave ESPR; they will come crashing back in full force.

Ladders are useful. Most useful work takes place inside them, despite them. You can’t scale without a ladder Do not let me tell you what to do.

Most work makes the world better in a small and quiet way. This is beautiful. Mihaly once worked on Google’s central TCP/IP code. He commented out one line and sped up every single web request that Google performs. A quadrillion of milliseconds saved every day = 11 years of human life every year. He says this is likely the most valuable work he ever did for them.

Getting to L6 at Google, having 2 children, saying true things, retiring at 50, being kind. This is a lot. It is more than most humans who ever lived achieved

Bayesianity

Sat, 20 Jun 2026 00:00:00 +0000

Lesson plan

Setup

Start with philosophy of science to motivate the basic prob theory.
Interactive: what do we want from a theory of science?
coherence; answers the actual question $P(H \mid E)$; generality; quantified uncertainty and graded confirmation; falsification = $LR \to 0$; reproducibility; mechanisms for the intuitions (total evidence, order & stopping-rule invariance, surprise, variety, conservation of expected evidence, Hume’s maxim, Cromwell’s rule); experiment design; parsimony for free.

Bug

Cancer calculation. Interactive
Correct answer 8%; doctors’ modal answer 95%; only ~20% right; unchanged for 50 yrs. Downstream costs.
smart people are not automatically rational.

Equation

Venn: conditioning = assuming true = zooming in. Ivy
Bayes’ theorem; three-line proof from the product rule; law of total probability (discrete case).
Posterior ∝ likelihood × prior: how well you predicted the data × what you believed before.
the quantitative theory of aggregating evidence; H lives in the map, E is the territory’s only channel
educated guess; shift when surprised; model everything; keep all info; never be certain; no ratchets.
there is a way to be rational. Guess, let the world in, and compare likelihood.

Debugging

Biases as broken Bayes:
base-rate neglect = deleting the prior (cf. p-value misreading); interactive: conjunction (Linda);
interactive: conservatism (bookbags)
confirmation bias as distorted likelihood, repaired by the likelihood ratio;
interactive: overconfidence
Four heuristics: base rate first; likelihood ratio, not likelihood; move a little; keep score.
you can do better even without calculating.

As optimal

convergence; washout; unification with decision theory; optimality; uniqueness.
Dutch book, accuracy dominance, complete class, bounded regret, epistemic utility, Cox.
The principles: credences; probabilism; conditionalisation; use priors.
otherwise sure loss, inaccuracy, inadmissibility, regret.

As incomplete

1. Silent on inputs: priors are personal and unconstrained and unexplained. hidden vs visible subjectivity; washout honest but weaker than advertised.
2. Intractable, violated in practice: NP-hard, incomputable; incoherent approximations; improper priors and marginalisation paradoxes.
3. Justification gap: finite vs countable additivity; nonconglomerability; elicitation indeterminacy; the precision regress; coherent rivals
4. Closed hypothesis space: grain of truth; confidently wrong under misspecification; no native model criticism; blind to new hypotheses.
5. Lonely: a single-agent theory of a social enterprise.
every theory has fatal problems; this theory is not finished.

Nonetheless (3 min)

Supermammogram: re-vote, calculate, 48%.
Exhortation: same intelligence; one equation installed. don’t let it go.

Notation note:

“A” and “B” denote abstract sets with areas in [0,1] part of a space with area 1.
“H” and “E” denote the epistemic interpretation of probabilities

Assuming that we want a theory of rationality…

1. Desiderata

Interactive: What do we want from a theory of science? Epistemology more generally? Rationality more generally?

(Let them speak for 10 mins. They know but they won’t know the terminology)

Coherence (the world is coherent, so our theory should be)
Answer the right goddamn question: $P(H \mid E)$, “How likely is the hypothesis given the data?”. Confirm hypotheses; degrees of actual confidence about the real question.
Generality. Be strictly larger than science: Answer any question. Cover all reasoning.
- Handles unique events
  - What is the probability that Lincoln will be assassinated?
  - What is the probability that Trump will be assassinated?
- Handles inadmissible evidence
- We’d like one theory that covers science, good intuition, and maths and logic.
Uncertainty
Quantification
- Degrees of confirmation
- Exactness: specific values
- Exactness: specific uncertainty
Falsification. Take out the trash
- Just $LR \to 0$. The “asymmetry of science” to be explained by the asymmetry of available likelihood ratios.
- Better! disconfirmation is not binary
Reproducibility: unlike others, Bayes puts all its judgments into the model.
Mechanisms for our intuitions
- “Always use all information. It’s always rational to look (at free evidence).”
- “The order of evidence shouldn’t matter.”
- “Experimenter intentions shouldn’t matter”: Bayesian inference is independent of when you stop collecting evidence!
- “surprising evidence confirms a hypothesis more” Lone anomalies can destroy theories.
- Varied evidence confirms more. Different researchers, different fields, different instruments, different models, different statistical frameworks!
- “you can’t reason your way to a predetermined conclusion.” Conservation of expected evidence. capacity for confirmation is purchased with exposure to refutation.
- “Extraordinary claims need extraordinary evidence”. Hume’s maxim. how extraordinary the evidence must be.
- “There’s something wrong with putting 0 on a hypothesis”. Cromwell’s rule
Help us design experiments. Lindley (1956): expected information gain makes it a theory of inquiry (which question to ask), not just belief revision.
Parsimony. Automatic (marginal likelihood: a model that can fit anything assigns little to any particular dataset - Jefferys & Berger 1992, MacKay) and explicit (complexity-weighted priors; Solomonoff as the limit). No razor axiom (but simplicity enters from our priors, a backdoor).

Finally consider some ultra-ambitious goals for such a theory:

Convergence guarantees. Intersubjectivity: priors should “wash out”
Unification of thought and act (not just epistemology, also decision theory). Beliefs are for acting: Savage integrates credence with utility, and Wald’s complete-class theorems say every admissible decision rule is a (limit of) a Bayes rule
Optimality
Uniqueness

Full list

Coherence (the world is coherent, so our theory should be)
- synchronic coherence (probabilism)
- diachronic coherence (conditionalisation)
- Sequential coherence (Show: updating on (x₁,x₂) jointly = updating on x₁ then x₂. This coherence under decomposition is special - most ad hoc belief-revision rules fail it.)
Answer the right goddamn question: $P(H \mid E)$, “How likely is the hypothesis given the data?”. Confirm hypotheses; degrees of actual confidence about the real question.
- Bayesianism is a quantitative confirmation theory: $E$ confirms $H$ iff $P(H \mid E) > P(H)$, i.e. iff $P(E \mid H) > P(E \mid \neg H)$.
Generality. Be strictly larger than science: Answer any question. Cover all reasoning.
- Handles unique events
  - What is the probability that Lincoln will be assassinated?
  - What is the probability that Trump will be assassinated?
  - Not scientific questions, clearly. Bayes works fine.
- We’d like one theory that covers science, intuition, and logic. Classical logic handles certainty; Bayes claims to be its unique extension to uncertainty (Cox: real-valued plausibilities are consistent with propositional logic).
- frequentism is a theory of procedures, not beliefs; falsificationism is qualitative and flees from the commonsensical idea of conformation; formal learning theory is about eventual convergence.
- Big problem: logical uncertainty
Uncertainty
Quantification
- Degrees of confirmation
- Exactness: specific values
- Exactness: specific uncertainty
Falsification. Take out the trash
- Just $LR \to 0$. The “asymmetry of science” to be explained by the asymmetry of available likelihood ratios.
- Better! disconfirmation is not binary
Reproducibility: unlike others, Bayes puts all its judgments into the model.
Mechanisms for our intuitions
- Use all information. Combine information from multiple sources and related phenomena. Use your hunches. It’s always rational to look at free evidence. “Consider all the evidence” and “looking is rational”
- The order of evidence shouldn’t matter.
- Experimenter intentions shouldn’t matter: Bayesian inference is independent of when you stop collecting evidence!
  - Likelihood Principle: Inference shouldn’t depend on intentions, past analyses, stopping. In Bayes (and LLism) inference depends on the data only through the likelihood function. Birnbaum (1962) derived it from sufficiency plus conditionality.
- Lone anomalies can destroy theories. “surprising evidence confirms a hypothesis more”
- varied evidence confirms more. Different researchers, different fields, different instruments, different models, different statistical frameworks!
  - Replications of the same kind share instruments, auxiliaries, and systematic errors, so they’re highly correlated under the rivals too, so $LR \approx 1$, a negligible boost to the posterior. Diverse evidence is evidence nearly independent conditional on each rival, so each new kind discriminates against a different chunk of ¬H.
- “you can’t reason your way to a predetermined conclusion.” Conservation of expected evidence. $\mathbb{E}[\text{posterior}] = \text{prior}$: no experiment can be expected to confirm, and if E would confirm then ¬E must disconfirm. No experiment can confirm under every outcome; capacity for confirmation is purchased with exposure to refutation.
- Qu antitative Hume’s maxim: “Extraordinary claims need extraordinary evidence.” Testimony establishes a miracle only if the testimony’s falsehood would be more improbable than the miracle - i.e. the likelihood ratio must exceed the prior odds against. how extraordinary the evidence must be.
- “There’s something wrong with putting 0 on a hypothesis”. Cromwell’s rule
Help us design experiments. Lindley (1956): expected information gain makes it a theory of inquiry (which question to ask), not just belief revision.
Parsimony. Automatic (marginal likelihood: a model that can fit anything assigns little to any particular dataset - Jefferys & Berger 1992, MacKay) and explicit (complexity-weighted priors; Solomonoff as the limit). No razor axiom (but simplicity enters from our priors, a backdoor).
~~De Finetti exchangeability.~~ Explains what iid models and “objective chance” talk mean in credence terms, bridge rationality theory and statistical practice (plus Lewis’s Principal Principle for chance–credence coordination).
Handle classic puzzles:
- Quantitative answer to the ravens paradox (a non-black non-raven confirms “all ravens are black” - a negligible amount; Hosiasson-Lindenbaum 1940),
- Tames irrelevant conjunction
- Handles Duhem–Quine: blame for a failed prediction is apportioned across theory and auxiliaries by their priors and likelihood contributions (Dorling 1979).

Finally, some ultra-ambitious goals for a theory:

Optimality (guaranteed convergence to truth, adversarial robustness, epistemic efficiency)
Intersubjectivity: priors should wash out
Unification of thought and act (not just epistemology, also decision theory)
- Beliefs are for acting: Savage integrates credence with utility, and Wald’s complete-class theorems say every admissible decision rule is a (limit of) a Bayes rule.
Uniqueness. The insane claim that no one else can compete with Bayes.

Is there anything that gets us all of these at once?

2. Bug

Imagine you’re a doctor. A patient comes in for a routine screening without symptoms.

The prevalence of breast cancer in her cohort is 1%.

Mammogram sensitivity $P(\text{positive test} \mid \text{has cancer})$: 90%.

False-positive rate $P(\text{positive test} \mid \text{no cancer})$: 9%.

She tests positive.

What’s the probability she has cancer? $P(\text{cancer} \mid \text{positive mammogram})$:

Interactive: give them a minute to write. Take a vote.

	Tally of student answers
Not enough info
1%
9%
48%
91%

Most will say ~91%. Calculate the correct answer. I will justify this calculation in a moment.

\[P(\text{has cancer} \mid \text{+mamm})\]

Who has seen this thing | before? It means “given”, “after”, “assuming”, “having observed”. So P(having cancer assuming that I have tested positive on this weak test).

\[\begin{aligned} P(\text{cancer} \mid \text{+mamm}) &= \frac{P(\text{+mamm} \mid \text{cancer}) \times P(\text{cancer})}{P(\text{+mamm})} \\ &= \frac{0.9 \times 0.01}{0.9 \times 0.01 \,+\, 0.09 \times 0.99} \\ &= \frac{0.009}{0.0981} \\ &\approx 0.09 \end{aligned}\]

Yes, 9%: more than nine out of ten positive tests are false alarms. These are real parameter values by the way, for the US right now.

What do doctors answer? The most common answer among them is “90%”. Only around 20% of them get it right. We first tested them on this 50 years ago. It’s pretty famous and they still haven’t improved. This is a horrible bug in human thought!

This test is performed millions of times a year, so consider the downstream costs: we scare millions of women a year and send them for unnecessary biopsies - a giant needle taking a little chunk of your chest. Overdiagnosis, overtreatment, terrified patients. And the people failing here are highly intelligent, caring, and trained for a decade.

Something makes us fail here, and it’s not lack of talent. Humans fail here because we don’t use the equation.

Big fact 1: Intelligence != rationality. smart humans are not automatically rational

2. Equation

Proof

Draw a bunch of Venn diagrams.

Let $$\Omega$$ be the set of all possible events. Let $$A$$ be some event. Let $$P(A)$$ be its area.

Let $$P(A \mid B)$$ be the area of A that is inside B: the conditional probability of A given B.

We can also read | as "zoom". A zoom B.

Interactive: what's $$P(A \mid A)$$? It equals 1 — iff $$P(A) > 0$$.

Conditioning means to assume true. It is zooming in such that nothing exists except in B.

I have reduced my uncertainty. If I'm looking in Omega, the whole space, I have maximum uncertainty. If I condition on some event A, I get to ignore everything that isn't in A. Ignore the worlds where ~A.

Big fact 2: To reduce uncertainty is to destroy possibility. To zoom in.

Among olympiad kids, what fraction are really cool?
Among really cool people, what fraction are olympiad kids?
Among ESPR participants, what fraction are olympiad kids?
Among olympiad kids, what fraction went to ESPR?

We proceed to derive Bayes' theorem from axioms. We grant ourselves set theory. Consider an "event" A with a probability (area) P(A).

The essence of a probability is being about the size of sets, thus nonnegative and summing to 1:

A1: $$P(A) \geq 0$$
A2: $$P(\Omega) = 1$$

and conjure probability theory from it:

$$P(A \mid B) := P(A \cap B) \,/\, P(B)$$
$$P(A \mid B)\,P(B) = P(A \cap B)$$
$$P(B \mid A)\,P(A) = P(B \cap A)$$
$$A \cap B = B \cap A$$
$$P(A \cap B) = P(B \cap A)$$
$$P(A \mid B)\,P(B) = P(B \mid A)\,P(A)$$
$$P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}$$

Interactive: What's missing from line 1? $$P(B) \neq 0$$

A3: $$P\Big(\bigcup_i A_i\Big) = \sum_i P(A_i)$$

with $$P(B) = \sum_i P(B \mid A_i)\,P(A_i)$$ (law of total probability)

($$\neg$$ = "not", negation)

$$P(B) = P(B \mid A)\,P(A) + P(B \mid \neg A)\,P(\neg A)$$

This is famous so every part of it has a name

posterior = likelihood × prior over “weight of evidence”

Interpretation

We now depart mathematics for the wastelands of philosophy and apply this basic theorem to the question of “updating our mental model after seeing evidence”.

H = hypothesis
E = evidence

Write:

\[P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)} = \frac{P(E \mid H)\,P(H)}{P(E \mid H)\,P(H) + P(E \mid \neg H)\,P(\neg H)}\]

new mind ∝ how much each of my existing beliefs expected this × old mind
how well the hypothesis predicts the data you actually saw × how plausible the hypothesis was before you saw anything
the map after the world has pushed back ∝ the channel through which the territory transmits × the map before contact

H is in the mind. Hypotheses are made of map: models, theories, suspicions.
E arrives from the world. Evidence is the only thing the territory sends you.

Evidence is the only channel by which the world should rewrite a mind*, Bayes is the unique coherent protocol to communicate over that channel.

Plug in cancer

What's H in the cancer example? What's E?
What are $$P(E \mid H)$$ and $$P(H)$$?
So what is $$P(H \mid E)$$?

Eddington's confirmation of gravitational lensing

Who knows how general relativity was confirmed as a replacement for Newtonian mechanics? Right, Eddington went to Antarctica and took like 8 crude photographs during an eclipse, to see how much the sun bends light.

Identify H and E, write down the likelihoods, turn the crank:

Einsteinian $$H_E$$: $$\delta = 1.75''$$
Newtonian $$H_N$$: $$\delta = 0.87''$$
Null $$H_0$$: $$\delta = 0''$$

Observed: $$E = 1.98''$$ (arcseconds)

Bayes factor $$P(\text{GR} \mid E) \,/\, P(\text{Newton} \mid E) \approx 10^8$$

THEREFORE the prior doesn't really matter. Take an illustrative 1919 prior, $$P(H_E) = 0.2$$, $$P(H_N) = 0.7$$, $$P(H_0) = 0.1$$

$$P(H_E \mid E) = \frac{0.2 \times 0.44}{0.2 \times 0.44 \,+\, 0.7 \times 5 \times 10^{-9}} \approx 0.99$$

Unbelievably strong evidence from one instrument.

The Bayesian backpack

My friend's backpack got lost three days ago, on a night out in a Budapest park.
H1: Stolen
H2: Misplaced under the influence

Same drill: H? E? $$P(E \mid H)$$? $$P(H)$$? And only then $$P(H \mid E)$$.

Interactive: How do you proceed? What evidence should we seek?

CCTV? Well, a clever thief would wait until he was out of view of the cameras - so let's condition on there being no thief and grid-search the park.

Sally Clark

Mrs S, loses two infant sons a year apart, each apparently a cot death (SIDS). At trial, an eminent paediatrician testifies: "The probability of one cot death in a family like this is 1 in 8,500. Two is therefore (1/8,500)² ≈ 1 in 73 million. The chance that Mrs S is innocent is 1 in 73 million." Mrs S goes to prison.

Assume:

P(a first infant dies of SIDS) = 1/8,500 in such families.
SIDS clusters in families (shared genetics, environment): given one SIDS death, the sibling's risk is 1/400.
Covert infanticide disguised as cot death: P(a mother murders her first infant) = 1/320,000; given that, P(she murders the second) = 1/100.
Both double-SIDS (S) and double-murder (M) predict the observed evidence D — two dead infants, no signs of violence — with probability ≈ 1. Ignore mixed hypotheses.

3. Creed

Some think this equation is the essence of the greatest philosophical achievement of the 20th Century. The single rule for all good thinking.

But the P(using “Bayesian” correctly | you’re the kind of person who uses the word “Bayesian”) is really pretty low. In particular, please distinguish the following three REALLY DIFFERENT things.

Bayesian ~= that which uses probability theory for beliefs and belief changes, evidence, etc
Bayesian epistemology: the claim that if you’re doing Bayes, then you are rational
Bayesianity = the claim that if you’re rational then you are doing Bayes or doing Bayes in disguise. This is an insanely strong claim.

you are rational to the degree to which your update dynamics allow the world to rewrite your beliefs.

Bayes cannot work without a prior - where good data doesn’t exist, you must use your (expert) intuition. This is good!

Bayes cannot work without at least 2 hypotheses. Never testing your pet theory alone. This is good!

Big fact 3: there is a way to be rational in theory. Let the world in.

Intuition

Some verbal ways of doing this

1) make an educated guess, 2) shift it if the data surprise you

3) model everything, 4) don’t throw away info

5) never be certain, 6) no ratchets.

7) You only buy predictive power and updates by sticking your neck out

Bayes as debiasing

We cannot spend all day calculating. It’s too slow. So the way to “use” it in >99% of cases is qualitative.

Who has heard of “cognitive biases”? We can explain them as deviations from the standard set by probability theory.

Base-rate neglect

Bayes can’t work without a prior.

Q: What happens if you want to analyse something this way but you have no prior?

Keeping this in mind fixes a gigantic problem with default human cognition. In fact it basically fixes the cancer example above!

Using $P(\text{positive} \mid \text{disease})$ as if it were $P(\text{disease} \mid \text{positive})$: deleting the prior, which is to say deleting everything we knew before this single breast cancer test.

If you don’t have a prior, you’re often just silently assuming a uniform prior, $P(A) = P(\neg A)$.

Biases as broken Bayes

Interactive: Conjunction fallacy (Linda): $P(A \wedge B) > P(A)$, violating the axioms before updating even starts.
Interactive: Conservatism (Edwards’ bookbags-and-pokerchips): humans update in the right direction but too little - under-weighting likelihoods. (Note the tension with base-rate neglect: we under- and over-weight depending on framing)
Confirmation bias: distorting the likelihood - treating $P(D \mid H)$ as high and $P(D \mid \neg H)$ as low for congenial D. Bayesian repair: evaluate the likelihood ratio*; evidence only discriminates if it was more probable under one hypothesis. “What would I expect to see if I were wrong?”
Interactive: Overconfidence: proper scoring rules as the measurement instrument: forecasting tournaments score people with the Brier score; superforecasters are distinguished mainly by granular, frequently-updated, roughly-Bayesian probability revision (Tetlock).

Big fact 4: you can do better even without calculating

(1) ask "what's the base rate?" before "what's the evidence?";
(2) ask for the likelihood ratio, not the likelihood;
(3) update incrementally \- strong priors \+ weak evidence \= small moves. Be a little stubborn;
(4) occasionally state credences as numbers and score yourself

5. Gesturing at Optimality

Surprising evidence for Bayesianism and Bayesianity: many results point the same way despite making different starting points.

Some theorems which claim you must end up here. Full list

No time to go into these but ask me later: Six arguments from four unrelated premises:

Dutch book theorem (Ramsey 1926):

if you’re not Bayesian, you are mathematically guaranteed to not gain money against a clever bookie even if you get to set prices.

Accuracy dominance (Joyce 1998):

If you’re not Bayesian, then you’re throwing away accuracy: there exists an alternative assignment that obeys the probability laws and is closer to the truth.

Complete class (Wald 1950):

a decision rule is “admissible” if it’s not dominated. all admissible rules are Bayesian (or a limit of a Bayesian rule).

The regret minimiser (Solomonoff 1978):

asymptotically a perfect Bayesian always predicts as well as the best expert, even if an adversary chose the sequence specifically to hurt you.

Maximum epistemic utility (Greaves 2006):

Conditionalisation is the unique updating rule that maximises expected epistemic utility. Briggs & Pettigrew (2020) upgrade to dominance!

Cox-Paris theorem (1946 / 1994):

probabilities are (nearly) a unique solution for sensible numerical beliefs

6. Fatal Objections to Bayesian Epistemology and Bayesianity

Full list

a. Says little about the inputs to reason

This is not a complete theory of belief! It’s a theory of belief change. What to do to update existing beliefs optimally.
“ew - priors are personal, but science should be observer-independent.”
“where does the first prior come from?”
“If the priors are bad the posterior will be bad”

(i) A prior is a declared assumption; “objective” procedures are a prior chosen without admitting it. Usually an idiotic improper uniform prior
(ii) The likelihood (the model) is the larger subjective commitment, and every school shares it. De Finetti’s representation theorem: the “objective” iid likelihood is equivalent to an exchangeability judgement.
(iii) objectivism fails on its own terms: uniform priors aren’t reparametrisation-invariant (Bertrand’s paradox); “uninformative” is predicate-relative (grue); Jeffreys and maxent and frequentism each smuggle a choice.
We have no choice - the alternative to visible subjectivity is hidden subjectivity.
Also washout. But: the subjectivist answer “convergence will forgive our priors” is honest but weaker than advertised when models are misspecified or data finite.

b. Intractable, and so violated in practice

we can’t run this as-is. Exact inference is NP-hard (Cooper 1990), #P-complete in general (Roth 1996); even approximation is NP-hard (Dagum–Luby 1993); the Solomonoff ideal is outright incomputable.
Practice deviates:
- approximate posteriors are not coherent;
- improper priors are not probabilities and generate marginalisation paradoxes (Dawid–Stone–Zidek 1973).
Datum: Very little of the actually good AI in the world is Bayesian
- Deep learning beat probabilistic programming, Bayesian nonparametrics, and Bayesian neural nets at nearly everything.
  - nothing in deep learning really touches MCMC/PPL for actually serious scientific inference on hard data, yet. But that’s approximate Bayes, and LLMs can just call PPLs themselves, but not the converse, so their abilities are a superset.
  - Deep learning is very roughly the automated construction of new continuous “theories”, which is what Bayes is bad at in theory.
- cold posteriors: Bayes needs to be actively broken to work for NNs. Optimal uncertainty is not optimal.
- trained transformers deviate from the in-context least-squares/Bayes solution off-distribution.
- Consider Imperial Bayesianism, viz. “successful inference succeeds iff it approximates the Bayesian ideal, Bayes is the unique right frame for intelligence and rationality”.
  - But an idealisation is explanatory iff successful practice approximates it, or if it guided practice. Frontier AI overwhelmingly wasn’t guided by Bayesian design, and the things that actually determine LLM success (architecture, data curation, representation learning, RLHF) usually have no natural Bayesian description.
- And yet…
  - the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others RMS normalization, Nesterov acceleration and AdamW
  - SGD as approximate inference
  - SGD as Bayes samp
  - deep ensembles as crude marginalisation / approximate inference → but ensembles are no longer competitive
  - random networks as priors over function space → but GPs are not very relevant
  - in-context learning as implicitly Bayesian → ICL isn’t martingale so can’t be Bayes,
  - Dropout as approximate Bayes → who cares
  - batch norm as approximate inference
  - (often post-hoc, often procrustean)
- Still, the likelihood of our world is noticeably higher under “Bayes is one lens among several” than under “Bayes is the theory of inference.”

c. It’s actually not general

Can’t use this as stated for mathematical propositions. (“logical uncertainty”)
Lonely theory. Can’t use this as stated for distributed social reasoning. But 99% of science is social, multi-agent
All of the above is single-agent. Science’s objectivity is intersubjective
Impossibility of opinion pooling. No pooling operator preserves both unanimity and independence; external Bayesianity conflicts with eventwise independence (Genest–Zidek 1986; Dietrich–List).
Treating peers’ credences as normal evidence doesn’t work: you’d need a likelihood for other minds.
A shifting reality
- Radical conceptual change
Memory loss
Self-locating belief. Sleeping Beauty, Doomsday, Everettian branching. Indexical information (“it is now Monday”, “I am this observer”) doesn’t fit into propositions-as-world-sets, and there’s no consensus update rule. SSA vs SIA, halfers vs thirders.
Ambiguity
Measure-zero events
Infinity pathologies

d. Justification gap: these theorems don’t fix a perfect unique rationality theory

Infinite case isn’t covered, so washout isn’t proven. The coherence argument delivers only finite additivity, and de Finetti explicitly rejected countable additivity. But Doob, martingale convergence and Blackwell–Dubins all require it.
Also finitely additive credences are nonconglomerable (“reasoning to a foregone conclusion”, Schervish–Seidenfeld–Kadane).
Elicitation indeterminacy: with state-dependent utility, P is not identifiable from preferences at all (SSK 1990).
The regress on precise credences: the hyperprior regress collapses formally (a mixture of priors is a prior), but the collapse presupposes precise mixing weights - the precision objection! Imprecise-probability theorists read Ellsberg-style ambiguity aversion as evidence the precision premise is false, and not as people being irrational.
Bayes is not unique: coherent rivals exist - imprecise probability, Dempster–Shafer, ranking functions.

e. Bayes has a closed hypothesis space, but the world is M-open

We need a grain of truth or nothing works. Every optimality theorem quantifies over a fixed Θ and assumes the truth (or a good-enough KL-neighbour) has positive prior mass (Kalai–Lehrer’s grain of truth; the “M-closed” assumption). Under misspecification the posterior concentrates confidently on the closest wrong model, with miscalibrated credible sets (Grünwald–van Ommen 2017).
no native-Bayesian model criticism. Bayes compares hypotheses within a class. The catch-all “other” has no likelihood, so it cannot be conditionalised on.
- Checking runs on sampling-theory tools (Box 1980; Gelman–Shalizi 2013 as confession).
The problem of new hypotheses: conditionalisation redistributes mass over a fixed Θ; it has nothing to say about conceiving a hypothesis you hadn’t formulated. Scientific revolutions live exactly there.

(“Your philosophy suffers an open objection” is vacuously true and so useless.)

Big fact 5: every theory has fatal problems. This theory is not finished. But it is the most promising we have.

My P(Bayes is usually useful) = 0.98
P(Bayes is the best current philosophy of science) = 0.85 P(Bayesianity, Bayes = rationality, all open problems can be fixed) = 0.2

7. Conclusion

Back to the clinic:

You’re a doctor. A new mammogram has been invented with only a 1% false positive rate. New patient from the same cohort comes in and gets the new test. It’s positive. What’s the posterior probability of her having breast cancer now?

\[\begin{aligned} P(\text{cancer} \mid \text{+supermamm}) &= \frac{P(\text{+supermamm} \mid \text{cancer}) \times P(\text{cancer})}{P(\text{+supermamm})} \\ &= \frac{0.9 \times 0.01}{0.9 \times 0.01 \,+\, 0.01 \times 0.99} \\ &= \frac{0.009}{0.0189} \\ &\approx 0.48 \end{aligned}\]

You didn’t become more intelligent in the last hour. The difference between your old wrong answer and the new correct one is this one equation. Homework: don’t let it go.

Appendices

Followup classes

Distributional Bayes
Cox’s theorem given Halpern
Complete class theorem
Accuracy dominance
Optimality of the Bayes classifier
Solomonoff induction and AIXI: Bayes as entire basis for a perfect AI

Resources and further reading

https://bayesbitsbrains.github.io/
Jaynes, Probability Theory: The Logic of Science,
ch. 1–2; McElreath, Statistical Rethinking
Optimality: Joyce (1998); Greaves & Wallace (2006); Robert, The Bayesian Choice, ch. 2, 8.
The physician data: Casscells, Schoenberger & Graboys, NEJM 1978; Manrai et al., “Medicine’s Uncomfortable Relationship With Math,” JAMA Intern. Med. 2014; Gigerenzer et al., “Helping Doctors and Patients Make Sense of Health Statistics,” Psych. Sci. Public Interest 2007 (the gynaecologist mammography data).
Debiasing: Tetlock & Gardner, Superforecasting; Gigerenzer on natural frequencies.
Epistemology: Ramsey, “Truth and Probability” (1926); Talbott, “Bayesian Epistemology” (SEP); Yudkowsky, “An Intuitive Explanation of Bayes’ Theorem” and Map and Territory*
Critiques

6 things sharing a name

The theorem: a straightforward fact, used by everyone.
- P(BT) = 1 − ε
The statistical toolbox: using this theorem as the engine of inference from data: parameters are random variables, priors over models, inference as conditioning, output as posteriors rather than point estimates. Credible intervals assert what everyone wrongly believes about confidence intervals. Priors = regularisers, MAP = penalised MLE, marginal likelihood = model comparison, posterior predictive = calibrated forecasting.
- P(ST) = 0.97 for “really good tool”, P(ST) = 0.1 for “never use freq”.
- Wald’s complete class theorem; Bernstein–von Mises agreement with the MLE in regular models; the Stein phenomenon read as empirical Bayes (MLE inadmissible in ≥3 dimensions); the engineering record - Bletchley, Kalman filters, hierarchical models, LIGO parameter estimation, Bayesian optimisation
Descriptive theory of perception: perceptual and sensorimotor systems approximately implement Bayesian integration. Cue combination is near-optimal, motor priors integrate as the posterior predicts.
- P(DB) = 0.7
The worldview, Bayesian epistemology: gradation, probabilism, conditionalisation applied to all belief: stats, juries, diagnosis, history, God. Rational belief comes in degrees; the degrees are probabilities; learning is conditionalisation.
- P(probabilism) = 0.6,
- P(conditionalisation) = 0.5,
- P(unique) = 0.2
Descriptive theory of minds: cognition as such approximates Bayes. Bayes-ish at 100ms, catastrophically non-Bayesian at the level of exams?
- P(DM) = 0.15
The imperial worldview: Strong Bayesianism: the claim of the unique global theory of all induction, the claim to subsume science, the claim to subsume (propositional) logic, the claim that all good methods are secretly Bayesian. all good inference is secretly Bayes.
- P(SB) = 0.04
- Warrant: −log w regret bound; the reinterpretation programme (regularisation = MAP, MDL ↔ Bayes codes, ensembles ≈ model averaging). AIXI, MDL, PAC-Bayes

Optimality: the theorems in full

Seven arguments from unrelated premises:

Dutch book

P1: Your degree of belief in A is your fair price for the $1-on-A ticket.
P2: At your fair price you’re indifferent between the two sides, so the bookie picks which side you take.
P3: If each bet in a finite bundle is individually acceptable, the bundle is acceptable.
P4: Don’t hold beliefs that guarantee that you lose money.
Theorem (Ramsey 1926): If your prices violate the probability axioms, there exists a finite bundle of bets, each fair, that must lose you money.
Converse: If your prices obey the axioms, no such bundle exists.
Conclusion: Your credences should obey the probability axioms.

e.g. Take “rain tomorrow” and “no rain tomorrow”. Whatever happens, exactly one ticket pays out. So holding both is holding a pound. P(rain)=0.6, P(no rain)=0.6.

Now suppose you price them at 60p each. You’ve just announced you’d buy a £1 note for £1.20. The bookie sells you the pair, takes £1.20, hands back £1. You lose 20p, guaranteed, regardless of the weather. Price them at 40p each and you’ve announced you’d sell a £1 note for 80p; the bookie buys the pair off you and collects £1 from you tomorrow. Same 20p, other direction. That’s why probabilities must sum to one.

Price a certainty below £1 and you’ll sell a pound for less than a pound.
Price something at a negative number and you’ll pay someone to take a ticket that can’t ever cost you.

(Pragmatic, not epistemic: shows incoherent betting is exploitable, not that incoherent belief is irrational.)

Accuracy dominance

Why obey probabilism if you never bet? Because incoherence is epistemically defective regardless of stakes:

Accuracy-domination (Joyce). Under a reasonable measure of accuracy (a Brier-style score satisfying certain axioms), any incoherent credence function is dominated: there exists a coherent credence function that is strictly more accurate in every possible world.

So a Bayesian need not care about Dutch books at all - incoherence is irrational because it is guaranteed to be further from the truth than some coherent alternative, in all worlds at once. This is the cleanest justification.

A1 (Alethic monism / Vindication). Accuracy is the only epistemic virtue of a credence, and the perfectly accurate credence at $w$ is $\mathbf{v}_w$. So inaccuracy is distance-from-the-truth-values and nothing else.
A2 (Additivity). Total inaccuracy is the sum of the inaccuracies of the individual credences.
A3 (Continuity). Small credence changes → small inaccuracy changes.
A4 (Strict propriety). Every probability function expects itself to be uniquely the most accurate.
A5 (Dominance). If option x does worse than option y in every possible world, on the only dimension that matters, x is irrational.

Theorem (Joyce 1998; cleanest version Predd et al. 2009, IEEE Trans. Inf. Theory 55: 4786–92, which ties coherence and proper scoring rules to Bregman divergence). Given A2–A4: every incoherent $\mathbf{c}$ is strictly accuracy-dominated - there is a coherent $\mathbf{p}$ that is more accurate at every single world. And no coherent credence is dominated by anything.

Purely epistemic argument: no bets, no money.

Complete class

A1 (Finiteness/regularity). Θ finite, with ℛ closed and bounded from below. (Or: compactness plus continuity of the risk function.)
A2 (Randomisation permitted). You may flip coins between rules - which makes ℛ convex.
A3 (Loss and expectation given). A real-valued loss exists, and risk is expected loss under the sampling distribution.
A4 (Admissibility norm). Using a dominated rule is irrational.
Theorem (Wald’s Complete Class Theorem, 1950). With Θ and X finite, a rule is admissible iff it is Bayes with respect to some strictly positive prior. Hence the Bayes rules form a complete class: anything outside is Pareto-beaten by something inside.

Conclusion: Any non-dominated procedure is a Bayes procedure for some prior. Bayes, or you’re either dominated or a Bayesian in disguise. Choosing an “objective” procedure = choosing a prior without admitting it.

Minimising regret

Symbols x₁, x₂, … arrive one at a time. Before each, you name a probability distribution over what comes next. Your penalty for round n is the log score, $-\log q(x_n \mid x^{<n})$. A comparator class $\{P_1, P_2, \ldots\}$ of rival prediction strategies is fixed in advance, with weights $w_j > 0$, $\sum_j w_j \leq 1$. Your strategy is the mixture $P_{\text{mix}} = \sum_j w_j P_j$ - equivalently, Bayesian prediction with prior $w$.

Regret against $P_j$ after $n$ rounds = your cumulative log loss minus $P_j$’s.

A1 (Class declared in advance). Countable, fixed before the data, not chosen post hoc.
A2 (Strictly positive weights). $w_j > 0$ for every $j$ you want a guarantee against.
A3 (Log loss). The score is logarithmic. Not incidental - load-bearing, see below.
A4 (Probabilistic predictions). Each strategy assigns normalised conditional probabilities, so the chain rule applies.
A5 (Comparator, not truth). The benchmark is the best strategy in the class.

Theorem. For every $j$, every $n$, every sequence: $\mathrm{Loss}(P_{\text{mix}}, x^n) \leq \mathrm{Loss}(P_j, x^n) + \log(1/w_j)$. Cumulative regret bounded by a constant, so per-round regret is $O(1/n)$.

https://www.sciencedirect.com/science/article/abs/pii/S037837581300116X

Cox’s theorem: probabilities are (nearly) a unique solution for numerical beliefs

Your gut’s plausibility scale may be as nonlinear as an unmarked mercury column. The theorem doesn’t say your beliefs are numbers in $[0,1]$; it supplies the calibration curve $g$ under which they are, and under which the combination laws become product and sum. Same content as decibels vs watts: the regrading w is the log/exp that converts your combination rule into the canonical one.

One real number. Your degree of belief in claim A given evidence X is a single real, $\mathrm{Bel}(A \mid X)$;
1. all beliefs are comparable on one scale.
2. excludes interval-valued/imprecise credence and Dempster–Shafer by fiat
Negation is local. Belief in ¬A given X depends only on belief in A given X: $\mathrm{Bel}(\neg A \mid X) = S(\mathrm{Bel}(A \mid X))$,
1. one fixed decreasing function $S$ for all subject matter.
Conjunction is local and chains. Belief in A∧B given X is determined by belief in B given X and belief in A given B∧X: $\mathrm{Bel}(A \wedge B \mid X) = F(\mathrm{Bel}(A \mid B \wedge X),\, \mathrm{Bel}(B \mid X))$, one fixed $F$.
Regularity and consistency. $F$ is continuous and strictly increasing in each argument (on the region above impossibility);
1. logically equivalent claims and evidence receive equal numbers; certainty and impossibility sit at the extremes; any two valid ways of computing the same plausibility must agree.
Paris: density. For every triple $(\alpha, \beta, \gamma)$ in $[0,1]^3$ and every $\varepsilon > 0$, there exist nested events $U_1 \supseteq U_2 \supseteq U_3 \supseteq U_4$ whose chained conditional beliefs $\mathrm{Bel}(U_4 \mid U_3), \mathrm{Bel}(U_3 \mid U_2), \mathrm{Bel}(U_2 \mid U_1)$ land within $\varepsilon$ of $\alpha, \beta, \gamma$.
1. your world is rich enough that chained plausibilities approximate every combination of values.

Theorem: There exists a continuous, strictly increasing regrading $g$ such that $P := g \circ \mathrm{Bel}$ satisfies

$$P(\text{certain}) = 1, \qquad P(\text{impossible}) = 0,$$ $$P(\neg A \mid X) = 1 - P(A \mid X), \qquad P(A \wedge B \mid X) = P(A \mid B \wedge X) \cdot P(B \mid X).$$

That is finitely additive conditional probability; Bayes’ theorem drops out of the two factorisations of $P(A \wedge B \mid X)$.

Greaves conditionalisation maximises utility

A1 (Cognitive decision theory). Belief states are treated as acts and evaluated by expected epistemic utility. The animating idea: conditionalising is rational iff it can reasonably be expected to produce epistemically good outcomes.
A2 (Strict propriety). Every coherent credence function uniquely maximises its own expected epistemic utility. G&W motivate this as a stability requirement - an agent shouldn’t be forbidden from continuing to hold a credence distribution on the grounds that holding it fails to maximise expected epistemic utility calculated with respect to that very distribution.
A3 (Partitional evidence, known ex ante). You know the partition now, and you’ll learn exactly one cell.
A4 (Transparency and certainty). You learn which cell with certainty, and you know that you’ve learnt it.
A5 (Prior-relative expectation). The whole calculation is done at t₁, with P.

Theorem. Conditionalisation is the unique updating rule that maximises expected epistemic utility.

Briggs & Pettigrew upgrade to dominance!

AIXI: Bayes as a theory of perfect intelligence

The optimal mind acts by Bayes (Legg and Hutter 2008)

A1 (Reward hypothesis). The goal is the expectation of a discounted sum of a scalar reward the environment supplies.
A2 (Grain of truth). The true environment lies in M. Weak, but not vacuous.
A3 (Universal prior). Weights from Kolmogorov complexity relative to a chosen UTM.
A4 (Expected-utility maximisation). Act to maximise expected value under ξ.
A5 (Dualism, tacit and load-bearing). The agent is not in M. No environment in the class contains AIXI’s own computation, hardware, or the fact of its being an agent. The interface is a fixed pipe.
(1) AIXI is Bayes-optimal. It maximises $\sum_\nu w_\nu V^\pi_\nu$. This is true by construction - it’s the definition of argmax.
(2) AIXI is Pareto optimal. No policy does at least as well in every $\nu \in M$ and strictly better in some.
(3) AIXI is self-optimising. If M admits some policy that is asymptotically optimal in every $\nu \in M$, AIXI is one. Fails for “lower semicomputable” environments - that class admits no self-optimising policies, so the theorem does not apply. And Orseau (2010) showed AIXI is not asymptotically optimal; weak asymptotic optimality turns out to be the only nontrivial notion available, and is achieved by BayesExp but not by AIXI.

Also pointing the same way: Doob’s consistency theorem and Blackwell–Dubins merging. Seven independent arguments lead to the same arithmetic. Surprising convergence is the real theoretical case for Bayes.

Objections: the full list

1. Says little about the inputs to reason

“ew - priors are personal, but science should be observer-independent.”
“where does the first prior come from?”
(i) A prior is a declared assumption; “objective” procedures are a prior chosen without admitting it. Usually an idiotic improper uniform prior
(ii) The likelihood (the model) is the larger subjective commitment, and every school shares it. De Finetti’s representation theorem: the “objective” iid likelihood is equivalent to an exchangeability judgement.
(iii) objectivism fails on its own terms: uniform priors aren’t reparametrisation-invariant (Bertrand’s paradox); “uninformative” is predicate-relative (grue); Jeffreys and maxent each smuggle a choice; frequentist procedures likewise.
We have no choice - the alternative to visible subjectivity is hidden subjectivity.

Reply:

The subjectivist answer (“convergence will forgive our priors”) is honest but weaker than advertised when models are misspecified or data finite. Washout (Doob; Blackwell–Dubins). But: (a) it requires countable additivity, (b) requires a grain of truth, (c) holds almost-everywhere, and the null set actually does bite in nonparametrics (Diaconis–Freedman 1986); (d) it holds under conditions that also make non-Bayesian estimators converge, so it doesn’t discriminate.
Subjectivity is not unique to subjective Bayes: objective-Bayes; Jeffreys; maximum entropy all smuggle a choice.

2. Bayes has a closed hypothesis space, but the world is M-open

We need a grain of truth. Every optimality theorem quantifies over a fixed hypothesis space and assumes the truth (or a good-enough KL-neighbour) has positive prior mass (Kalai–Lehrer’s grain of truth; the “M-closed” assumption). Under misspecification the posterior concentrates confidently on the closest wrong model, with miscalibrated credible sets (Grünwald–van Ommen 2017).
no native-Bayesian model criticism. Bayes compares hypotheses within a class. The catch-all “other” has no likelihood, so it cannot be conditionalised on.
- Checking runs on sampling-theory tools (Box 1980; Gelman–Shalizi 2013 as confession).
The problem of new hypotheses: conditionalisation redistributes mass over a fixed Θ; it has nothing to say about conceiving a hypothesis you hadn’t formulated. Scientific revolutions live exactly there.
Before the derivation we already needed to assume logical omniscience (“The domain assumption”). “You know everything you know and know all of the entailments between them.” P is a function on a sigma-algebra over sets of outcomes. To apply it to propositions A you must identify each proposition A_i with the set of possibilities where it’s true.

3. Conditionalisation is a narrow kind of learning

Conditionalisation presupposes evidence is propositional, certain, cumulative, and not self-locating. All four fail often.
- see e.g. uncertain perceptual input (thus Jeffrey conditioning, but which is order-dependent), forgetting, Sleeping Beauty (Elga 2000).
The rule covers exactly one case: becoming certain of a proposition already in your algebra, permanently, with fixed self-location.
Old evidence (Glymour 1980): Mercury’s perihelion was known 60 years before GR; if $P(E)=1$, conditionalisation confirms nothing - yet it was the confirmation. Mirror image: Bayes assigns no credit for novel prediction, because confirmation is a timeless function of (H, E, K) and genesis is screened off. Same axis, opposite ends. (Live reply: novelty matters only as evidence about overfitting risk - Hitchcock–Sober 2004.)
Bayes assigns no credit for novel prediction
Repairs: counterfactual ur-priors; Garber (1983), update on discovering H ⊢ E rather than on E. But Garber’s fix requires learning entailment facts, and logical omniscience - forced by the axioms, $P(\text{theorem})=1$ - forbids exactly that. The cluster’s two objections block each other’s repairs; real agents are uncertain about mathematics itself.

4. Justification gap (the theorems don't nail down a perfect and unique theory)

Infinite case isn’t covered, so washout isn’t proven. The coherence argument delivers only finite additivity, and de Finetti explicitly rejected countable additivity. But Doob, martingale convergence and Blackwell–Dubins all require it.
Also finitely additive credences are nonconglomerable (“reasoning to a foregone conclusion”, Schervish–Seidenfeld–Kadane).
Elicitation indeterminacy: with state-dependent utility, P is not identifiable from preferences at all (SSK 1990).
The regress on precise credences: the hyperprior regress collapses formally (a mixture of priors is a prior), but the collapse presupposes precise mixing weights - the precision objection! Imprecise-probability theorists read Ellsberg-style ambiguity aversion as evidence the precision premise is false, and not as people being irrational.
Bayes isn't unique, coherent rivals exist - imprecise probability, Dempster–Shafer, ranking functions.

5. Intractable, and so violated in practice

we can’t run this as-is
Exact posterior inference is NP-hard (Cooper 1990), #P-complete in general (Roth 1996); even approximation is NP-hard (Dagum–Luby 1993); the Solomonoff ideal is outright incomputable.
- “Bounded Bayesian rationality” is yet to be completed
Practice deviates by the theory’s own lights:
- approximate posteriors are not coherent;
- improper priors are not probabilities and generate marginalisation paradoxes (Dawid–Stone–Zidek 1973).

6. Lonely. But science is social, multi-agent

All of the above is single-agent. Science’s objectivity is intersubjective
No pooling operator preserves both unanimity and independence; external Bayesianity conflicts with eventwise independence (Genest–Zidek 1986; Dietrich–List). Science’s objectivity is intersubjective; Bayesianism is irreducibly monadic, and the peer-disagreement literature shows no Bayesian resolution.
The supra-Bayesian move (treat peers’ credences as evidence) doesn’t work: you’d need a likelihood for other minds.

Answers

Howson (2000): Bayesianism is a logic of consistency only
Generalised Bayes:
Imprecise Bayes: Seidenfeld–Wasserman 1993. conditioning can strictly widen every member’s interval whatever the outcome; belief inertia. vacuous credal sets never learn; and no agreed IP decision theory
tempered posteriors
Bounded/logical-uncertainty Bayes
Reverse Bayesianism (Karni–Vierø 2013)
Open-minded Bayesianism (Wenmackers–Romeijn 2016)
Awareness growth (Steele–Stefánsson).

And there are various binds in fixing these:

Defending washout consumes premises attacked in 2 (grain of truth) and 4 (countable additivity).
Defending Garber consumes logical learning, forbidden by the axioms defended in 4.
Jeffrey conditioning loses order invariance
Defending 2 and 5 via generalised posteriors (SafeBayes) abandons the likelihood. Which loses you all the coherence arguments.

Big fact 5: every theory has problems. “Your philosophy suffers an open objection” is vacuously true and so useless. This theory is not finished.

Mihaly’s version: the worlds method

(Notes on Mihaly’s presentation of the same material, via possible-worlds bookkeeping.)

Introduction
- Famous example.
- 1% of 40 year old women have breast cancer. Test for breast cancer which detects 90% of cancer cases; has 10% false positive rate. Woman has test result positive; what’s the probability that she has breast cancer?
  - Many doctors will forget that 1% of women have breast cancer - if test result is positive, people will ignore Bayes’ rule and think it’s likely that they have cancer.
  - How to think about it?
    - Let’s imagine 10,000 women. What portion of them have cancer? 100.
    - 9900 people don’t have cancer, 100 do
    - of the 9900:
      - 990 will have positive result
    - of the 100:
      - 90 will have positive result
    - What’s the chance that she’s in the positive result group?
  - Bayes’ Rule.
    - This is hard to work with.
    - What you should work with is something easier.
  - One woman, two possible worlds. In one world, she has cancer. In the other she is healthy. We are thinking in these two parallel worlds. Both are possible, we don’t know which world we are in (this is our uncertainty).
    - Both worlds have different weight. To each world we attach a weight in proportion to how probable the world is. Cancer world has small weight (1); healthy world has larger weight (99)
    - Proportion of weights matters. Higher proportion –> higher probability.
    - We do test; result comes positive. What is the chance of the thing happening in the cancer world? 0.9. What is the chance in the healthy world? 0.1. We multiply the weights by the chances.
    - Cancer world:
    - Normal world:
    - We want to know whether she has cancer. What’s the proportion of cancer world to total weight?
    - - same calculation.
Worlds method
- Three stages.
- CancerHealthyPriorLikelihood ratioPosterior
  - The ‘prior’ is the probability that
  - is the case anyway
  - The ‘likelihood ratio’ is the probability that
  - happens given
- Multiple worlds
  - Same principle applies.
  - You have a coin.
  - probability that it’s fair,
  - probability that it’s biased towards heads (0.75 probability heads, 0.25 probability tails),
  - probability that it’s biased towards tails (0.25 probability heads, 0.75 probability tails).
  - Hidden assumption: coin flips are independent.
  - Trial - gives results {Tails, Heads, Tails}. What’s probability coin is fair?
  - Fair bias H bias T3/62/61/61/23/41/48/643/649/6424/3846/3849/384
    - Answer:
- Infinite worlds
  - We have a coin. We know nothing about it. What can we say? (don’t know what its bias is)
  - Continuum of possible worlds. For every probability of coin coming up as heads we have a world. No more tables - instead we have a function.
  - Function maps from biasedness of coin to heads (probability that coin will give heads) to the probability that the coin is that biased.
  - Have functions for probability of heads / probability of tails given world.
  - Multiply functions etc. Integrals.
  - (to be completed - not covered in talk)
Probability Theory
- Calibration
  - If you predict 60% and the thing happens 33% of time, what do you do?
  - Compute p values?
  - Different scoring rules. e.g. if the thing happens and you predict it with probability p, you get (1-p)^2
  - nunosempere.github.io/rat/Self-experimentation-calibration.html
  - If you record this many times, you can produce graphs to help you calibrate your probabilities.
- Bayesian Hypothesis Testing
  - Say you have a box in your head - interaction between what you think and the world.
  - Sometimes outputs 0 and sometimes outputs 1. You don’t actually know the probability p for which it outputs 1 and for which it outputs 0. How do we work this out?
    - Explanation TBC / not transcribed
  - Probability Theory - E. T. Jaynes
    - Cox’s theorems. Derive the rules of probability theory from innocent-looking assumptions. If you don’t do Bayes’ theorem, you can be exploited. But that ends up not mattering a lot –> you’re boundedly rational anyways.
  - Superforecasting. Tetlock.
- Betting
  - Good way to ‘put some skin on it’ –> help you calibrate.

Bayes principles

Bayes tells you how beliefs must hang together; not what to believe. An agent with a coherent but idiotic prior is coherently idiotic. The theory is silent about where priors come from and whether priors are good. Generality and emptiness go together. Still, humans are bad enough that an empty skeleton can still help us.

The principles

Credences. The objects of epistemic evaluation are real-valued degrees of belief in [0,1].
Probabilism. Rational credences obey the probability axioms.
Conditionalisation. On learning $E$ with certainty, $P_{\text{new}}(\cdot) = P(\cdot \mid E)$.
Use priors. Don’t pretend you’re a baby: summarise all your past evidence into one big lump. Lets you turn the crank on any question.

What this gets you: a ~unique, quantitative, normative theory of rationality.

Induction and the prior

Bayesianism doesn’t solve Hume - it pushes the tacit solution to the problem inside your mind. The inductive leap lives entirely in the prior; coherence + conditioning then propagate it without further assumptions. given exchangeability, de Finetti’s representation theorem says you must act as if learning a parameter from i.i.d. data. The leap takes place from one symmetry premise about your beliefs, which is at least a precise location.

The reply to “your conclusion is just your prior”: merging of opinions (Blackwell–Dubins 1962; Doob). Agents with different priors but the same likelihood, provided their priors are mutually absolutely continuous and put positive mass near the truth, have posteriors that converge (in total variation), almost surely, as data accumulate. With enough data the prior washes out.

BUT convergence can be arbitrarily slow*; it can fail under model misspecification; and it fails entirely if the prior assigns probability 0 to the truth

The constitutive theory of beliefs

what belief is and how it moves

Gradation. Belief is quantitative; deductive logic is the limiting case ($E \vDash H \implies P(H \mid E) = 1$). Cox: real-valued graded belief under consistency desiderata is probability.
1. probability handles defeasible inference (new evidence can lower P(H)), which monotonic logic can’t.
Coherence. three independent vindications: pragmatic (Dutch book), preference-theoretic (Savage 1954), and, most philosophically important, epistemic: Joyce (1998) showed incoherent credences are accuracy-dominated.
A determinate update rule. Conditionalisation is vindicated diachronically (Lewis/Teller) and by expected-accuracy maximisation (Greaves & Wallace 2006). Corollary: order-invariance - total evidence fixes the posterior regardless of arrival sequence.
1. (Caveat: Jeffrey conditionalisation on uncertain evidence doesn’t commute unless evidence is logged as Bayes factors - Wagner 2002.)
Honesty is optimal. Proper scoring rules make you sincerely reporting your credence the unique expectation-maximiser. Lying to yourself is also strictly dominated.
Belief–action unity. Posteriors plug directly into expected utility; Savage’s single representation theorem delivers belief and desire jointly. Epistemology put to work.

Practical advantages over frequentism

disclaimer first: frequentist methods are often fine and sometimes superior in practice (speed, robustness-by-convention, no prior to argue over). The claim is about structural advantages.

The stopping-rule problem

Scenario: two labs collect identical data - 9 successes in 12 trials. Lab A fixed n=12 in advance; Lab B sampled until 3 failures. Same data, same likelihood ($\propto \theta^9(1-\theta)^3$) - but different sample spaces, so different p-values (one crosses 0.05, one doesn’t, for $H_0$: $\theta = 0.5$). The frequentist answer depends on the experimenter’s intentions - unobservable mental states. The Bayesian posterior is identical for both labs (Likelihood Principle).

Optional stopping is the engine of p-hacking and the replication crisis. Peeking at data invalidates p-values silently; it leaves posteriors untouched.

Structural advantages

$P(\theta \in \text{interval} \mid \text{data})$ vs. the confidence-interval contortion (“95% of intervals constructed this way cover…”). Students already misread CIs as credible intervals; Bayes makes their intuition correct instead of wrong.
Small n. No asymptotic approximations needed; the posterior is exact at n=5. Priors regularise exactly where data are weakest.
No nuisance parameters: marginalise them out - one integral. Frequentist treatment (profiling, conditioning) is a per-problem art form.
Hierarchical models / partial pooling: school-effects example (Rubin’s 8 schools in one sentence). Multilevel structure is native; the frequentist analogue (random effects + corrections) is bolted on. Also quietly dissolves the multiple-comparisons “family” problem - shrinkage replaces correction.
Decision integration: posterior × utility → expected loss. Inference and action in one framework; no p-value-to-decision folklore.
Uncertainty propagates: predictive distributions integrate over parameter uncertainty instead of plugging in point estimates

Costs

Computation (MCMC/variational machinery vs. closed-form tests);
priors require justification in adversarial settings (regulators, courts);
model checking is less native to pure Bayes than to error-statistics (Mayo’s severity point: error probabilities attach to procedures and are auditable by calibration).

Eulogy

Thu, 12 Mar 2026 00:00:00 +0000

Transnormalism

Fri, 20 Feb 2026 00:00:00 +0000

Society is unlikely to fall suddenly under the spell of the transhumanist worldview. But it is very possible that we will nibble at biotechnology’s tempting offerings without realizing that they come at a frightful moral cost… Modifying any one of our key characteristics inevitably entails modifying a complex, interlinked package of traits, and we will never be able to anticipate the ultimate outcome… we may unwittingly invite the transhumanists to deface humanity with their genetic bulldozers and psychotropic shopping malls.

— Fukuyama (2004)

the great artisan… made man a creature of indeterminate nature, and… said to him ‘Adam, we give you no fixed place to live, no form forever peculiar to you, no function that is yours alone. According to your desires and judgment, you will have and possess whatever place to live, whatever form, and whatever functions you yourself choose… To you is granted the power of degrading yourself into the lower forms of life, the beasts, and to you is granted the power, contained in your intellect and judgment, to be reborn into the higher forms, the divine.’… To us it was given to be whatever we choose to be, and so that is what we want.

— Pico della Mirandola (1486)

Around 2003 there was a big, rancorous human enhancement debate 3.

The reigning “bioconservatives” predicted that new biotech would lead to moral or political catastrophe, and the loss of human dignity, and maybe wouldn’t even boost welfare; the “transhumanists” 2 argued that nuh uh.

Transhumanism?

The 1998 declaration:

1. ...broadening human potential by overcoming aging, cognitive shortcomings, involuntary suffering, and our confinement to planet Earth.
2. humanity’s potential is still mostly unrealized.
3. humanity faces serious risks, especially from the misuse of new technologies. There are possible realistic scenarios that lead to the loss of most, or even all, of what we hold valuable... not all change is progress.
4. Research effort needs to be invested into understanding these prospects.
5. Reduction of existential risks, and development of means for the preservation of life and health, the alleviation of grave suffering, and the improvement of human foresight and wisdom should be pursued as urgent priorities
6. Policy making ought to be guided by... respecting autonomy and individual rights, and showing solidarity with and concern for the interests and dignity of all people
7. We advocate the well-being of... humans, non-human animals, and any future artificial intellects, modified life forms, or other intelligences
8. We favour allowing individuals wide personal choice over how they enable their lives ["morphological freedom"]

Clearly this is a tame and cuddly ideology compared to stuff like posthumanism and accelerationism, but for some reason the critics settled on attacking all biotech-curious ideologies under the name "transhumanism".

Dickey–Wicker

The big policy move from the debate, banning federal funding for embryonic stem cell research, was actually passed in 1996 under Clinton. Bush's 2001 executive order further restricted NIH funding to pre-existing cell lines, but was rescinded by Obama in 2009.

We routed around some of the damage: in 2006, the invention of induced pluripotency allowed for the creation of (second-rate) stem cells without touching embryos.

Some of the technologies they fought over (cloning, germline editing) haven’t happened at scale yet. But others (GLPs, hormones, hair tech, embryo selection) did, and were rapidly adopted by millions of people who had no interest in either ideology.

And so: a new era of mass chemical enhancement and healthy polypharmacy 1:

(link, link)

(link)

(link - a huge decline but still 1% of US pop! And trans is up to another 1%)

As so often, Fukuyama (quoted above) looks wrong but is not wrong: society indeed did not become transhumanist - that is, not in belief. Few of the biotech users endorse the philosophy of technological transcendence. But society is heading there in deed. Conservative forces (religion, disgust, precaution) were in this case grossly outgunned by the force of sheer desire.

We got, not transhumanism (as deliberate, informed, rational decision to self-consciously go beyond natural human capacity), but surreptitious transhuman behaviour, without the weird philosophy or the new aesthetics. Technology by default, without conscious ideology. Playing god - but using these new, unfathomable powers to… become more normal. So call it transnormalism. 5.

What of the bioconservative prediction of a reckoning for civilisation? So far none arrived. Either

the current techs are not powerful enough yet; or
the corrosive effects (on say fairness, authenticity, self-concept) are lagged or hard to measure; or
mass enhancement is just fine.

Ancient enhancement

A weak reason to think it’s fine is that we’ve been doing it for all of history and prehistory. Enhancement is older than humanity. What’s new is just the size of the enhancements and the biological, internal, invisible nature of the modifications.

Classic:

New:

Old School	New School
Alcohol (prehuman)	Antidepressants
Caffeine (ancient)	Adderall (1996)
Dexedrine (1937)	Adderall (1996)
Glasses (1300)	Intraocular lens (1999)
Makeup (ancient)	Tretinoin (1971), botox (1989), etc
Fluoride (1945)	Hydroxyapatite (1980)
Aspirin (1899)	COX-2 inhibitors (1998)
Antacids (1852)	Proton pump inhibitors (1989)
Statins (1989)	PCSK9 inhibitors (2015)
Dental fillings (ancient)	Osseointegrated (c. 1970s)
Anabolics (CDMT / Turinabol, 1968)	HGH (1985), rhEPO (1993), cardarine (2001), ACP-105 (2009), bimagrumab (2013)
The Pill (1960)	IUDs (2000), subdermals (2006)
Sunscreen (1940)	nanoparticle (2000s)
Alarm clocks (c. 1904)	light alarms (2010s)
Hearing aids (c. 1800)	Cochlear implants (1984)
Inactivated vaccines (1796)	mRNA (2020)

Other tech

I've been pretty focussed on chemical and biochemical enhancement in the above. There's a lot more:

Genetic

Not prevalent yet.

Surgery

Only 38 million cosmetic surgeries a year? Surprising!

It's hard to say what fraction of the 310 million major surgeries are "enhancers" but not many.

Electronics

The above omits a very early and widespread kind: electronic aids and implants.

1936: first wearable hearing aid. Now 300 or 400 million
1958: internal pacemaker. Now around 30? million people.
1977: cochlear implants. Maybe 2 million.

Nonmedical use isn't mainstream yet. The grinders (people who do DIY surgery to implant electronics for nonmedical use) are roughly as strange as they were 15 years ago.

Normal exceptionalism: the escape from morphological freedom

With the exception of the bodybuilding and trans communities 6, we don’t see much weirding at mass scale. We aren’t expressing our morphological freedom to look more different. It seems to me that the result of power over our appearance is not deviance and weirdness but heightened normative normalcy. Bigger biceps, fewer wrinkles, and nerds suddenly getting normative gender presentation. Not many cyborgs.

I’m not fit to do any comparative analysis of the history of cosmesis and beauty standards. This essay is great. But it seems to me that the beauty queens and movie stars of the 1960s no longer look as exceptional as they once did, because cosmetic technology (and image editing ig) has shifted the (top decile of the) distribution upwards so much. I can’t say what all of this is tending towards. What is the intensified platonic ideal of a normal dude?

It’s pretty obvious why tech which gives you options is used, on average, to normalise yourself: most people want to be normal, and the user population is so large now that it simply must include a lot of such people. In 2010 enhancement was a matter for nerds, hackers, and obsessive hobbyists like bodybuilders. But now it’s a much bigger coalition (e.g. 6-12% of Americans on GLP agonists).

Actual data

Leo Gao has been running informal n=200 surveys of random Americans. He finds, as I assumed above, that most people are indeed (incoherently) nontranshumanist, with the huge exception of immortality:

> "If you had the option to live forever in perfect health and youth, would you choose to? (Assume you could still change your mind at any time if you ever got bored of it.)” ... 66% of respondents said Yes, with 14% saying No, and 20% saying “Not sure”. As a follow up, it turns out roughly a third of Americans think developing the technology to enable life extension should be a top priority... Since it seemed like overpopulation and inequality were the main things people were worried about, I also asked a version of the question where I stipulated that these things were solved. Surprisingly, this barely shifts people’s opinions, and we get almost exactly the same response! My guess is this is a sign that the real objection is more about the vibes than any specific issue.

> despite being very pro living forever, Americans are much more skeptical of cryonics — even if they could be revived a few decades after their death to live forever thereafter, only 27% are in favor of being preserved, and 46% are opposed (the rest are unsure).

> Space colonization also has pretty lukewarm support, coming in at 37% in favor and 16% opposed

> and cognitive enhancement for all is only a little bit more popular (42% in favor, 19% opposed).

> Also, for some reason, people are really opposed to a hypothetical cheap, painless, and safe arbitrary modification of physical appearance (only 23% in favor, with 37% opposed!). In retrospect, the backlash against Ozempic is a sign, but I was still quite surprised.

> Terraforming other planets so that humans can live on them is also pretty unpopular, coming in at 37% in favor and 16% opposed. Thankfully, for most of these questions, a huge chunk of people are still undecided.

> only 51% of Americans are in favor of literal post-scarcity (complete freedom to work on anything you want, as much as you want, and still enjoy a high quality of life), with 25% opposing. I was so shocked by this result not being 80%+ in favor that I reran a variant of this question with different wording. My original question asked whether the world would be better or worse if everyone had the freedom to work on whatever they want, as long as they want, and still enjoy a high quality of life, and anything we don’t want to do is done for us by robots.

I thought maybe that set off some “AI taking jobs bad” instincts; for the new question I took pains to clarify that the stuff is literally conjured out of nowhere with magic and is not taken from anyone else, and got an even worse result (38% support, 34% oppose).

This is even more crazy, so I ran a third version on the hypothesis that people don’t like magic, or that not having to work sounded too crazy. This version asked whether it would be good if everyone made 10x more (inflation-adjusted) than they do currently. This polled only somewhat better, with 39% in favor and 19% opposing. I’m still pretty confused what conclusion to draw from this; this is probably worth digging more into.

> Only 14% think that society is currently trending in a positive direction.

Medicalisation and demedicalisation

You don’t need any new or unpopular premises at all to justify enhancement. Transnormalism is what you get from liberalism (consenting adults) plus medicalisation (doctors being given power and responsibility over more and more of the body and mind, and smaller and smaller ailments). Consumer enhancement is mostly happening in the form of individual medical decision-making: distributed, highly private (with some actual infosec), and invisible except by us belatedly noticing the aggregate properties of the species changing.

Medicine has been expanding for centuries, especially in the last two decades. This is true in volume (spending) and in domains (new treatments, new powers, new areas brought within the ambit).

In absolute terms (multiplying healthcare share by GWP) we spent $1.85tn in 2000, something north of $7.1tn in 2022 (and more now).

But after being incubated in medicine, enhancement is being taken off the doctors. The thriving grey market (intense cosmetics, “aesthetics” and “med spas” and nootropics) and black market (study drugs, research chemicals, peptides, bootleg hormones) are of unknown size but growing insanely fast.

You are like a little baby

The above technologies are really fairly weak. Retatrutide (2023) is twice as strong as semaglutide (2014), which is twice as strong as liraglutide (2002). At some point someone will work out how to chemically simulate the effect of working out. The nootropics industry is overall a pathetic failure, capped with blunt instruments like not sleeping or flooding the brain with catecholamines. Psychopharmaceuticals are better but not by much and don’t manage sustainably-better-than-well. We are admittedly really good at things which let people sprint for 6% longer, though at the expense of giving them cancer. We do nearly nothing directly to brains. We have basically nothing for memory enhancement. At the moment we do little with genes, but the rich and unsqueamish are beginning to. All humans are born premature. One might solve sleep. One might solve death.

The conservative concerns might apply to a more-mature science of More. Thanks to transnormalism funding it all we will soon see.

We find sheer humanism to be unsatisfying. It shuts the windows, draws the blinds, and seeks artificial elegance - oblivious of the outer night and the stars. Instead, we boldly go out into darkness and find the superhuman everywhere… the story of evolution of life: its length, its wastefulness, its precariousness, its chanciness, its progressive release of potentiality, its incomprehensibility and ourselves as moments within it… man is transitional and scarcely a beginning.

— Olaf Stapledon (1934)

AI in 2025: gestalt

Mon, 08 Dec 2025 00:00:00 +0000

This is the editorial for this year’s “Shallow Review of AI Safety”. (It got long enough to stand alone.)

Epistemic status: subjective impressions plus one new graph plus 300 links.

Huge thanks to Jaeho Lee, Jaime Sevilla, and Lexin Zhou for running lots of tests pro bono and so greatly improving the main analysis.

tl;dr

Informed people disagree about the prospects for LLM AGI – or even just what exactly was achieved this year. But the famous ones with a book to talk at least agree that we’re 2-20 years off (allowing for other paradigms arising). In this piece I stick to arguments rather than reporting who thinks what.
My view: compared to last year, AI is much more impressive but not proportionally more useful. They improved on some things they were explicitly optimised for (coding, vision, OCR, benchmarks), and did not hugely improve on everything else. Progress is thus (still!) consistent with current frontier training bringing more things in-distribution rather than generalising very far.
Pretraining (GPT-4.5, Grok 4, but also counterfactual large runs which weren’t done) disappointed people this year. It’s probably not because it wouldn’t work; it was just ~30 times more efficient to do post-training instead, on the margin. This should change, yet again, soon, if RL scales even worse.
EDIT: See this amazing comment for the hardware reasons behind this, and reasons to think that pretraining will struggle for years.
True frontier capabilities are likely obscured by systematic cost-cutting (distillation for serving to consumers, quantization, low reasoning-token modes, routing to cheap models, etc) and a few unreleased models/modes.
Most benchmarks are weak predictors of even the rank order of models’ capabilities. I distrust ECI, ADeLe, and HCAST the least (see graph below or this notebook). ECI shows a linear improvement, HCAST finds an exponential improvement on greenfield software engineering, and ADeLe shows a previous super-exponential slowing down to what might be linear growth.
The world’s de facto strategy remains “iterative alignment”, optimising outputs with a stack of alignment and control techniques everyone admits are individually weak.
Early claims that reasoning models are safer turned out to be a mixed bag (see below).
We already knew from jailbreaks that current alignment methods were brittle. The great safety discovery of the year is that bad things are correlated in current models. (And on net this is good news.)
Previously I thought that “character training” was a separate and lesser matter than “alignment training”. Now I am not sure.
Welcome to the many new people in AI Safety and Security and Assurance and so on. In the Shallow Review I added a new, sprawling top-level category for one large trend among them, which is to treat the multi-agent lens as primary.
Overall I wish I could tell you some number, the net expected safety change (this year’s improvements in dangerous capabilities and agent performance, minus the alignment-boosting portion of capabilities, minus the cumulative effect of the best actually implemented composition of alignment and control techniques). But I can’t.

Capabilities in 2025

Better, but how much?

— Fraser, riffing off Pueyo

Arguments against 2025 capabilities growth being above-trend

Apparent progress is an unknown mixture of real general capability increase, hidden contamination increase, benchmaxxing (nailing a small set of static examples instead of generalisation) usemaxxing (nailing a small set of narrow tasks with RL instead of deeper generalisation), and human cheating. It’s reasonable to think it’s 20% each, with low confidence. (With a small but growing contribution from AI cheating.)
Discrete capabilities progress seems slower this year than in 2024 (but 2024 was insanely fast). Kudos to this person for registering predictions and so reminding us what really above-trend would have meant concretely. The excellent forecaster Eli was also over-optimistic.
I don’t recommend taking benchmark trends, or even clever composite indices of them, or even clever cognitive science measures too seriously. The adversarial pressure on the measures is intense.
Pretraining didn’t hit a “wall”, but the driver did manoeuvre away from it on encountering an easier detour (RLVR).
- Training runs continued to scale (Llama 3 405B = 4e25, GPT-4.5 ~= 4e26, Grok 4 ~= 3e26) but to less effect.¹ In fact all of these models are dominated by apparently smaller pretraining runs with better post-training.
- 4.5 is actually shut down already; in 2025 it wasn’t worth it to serve any 1T active model or make it into a reasoning model. But this is more to do with inference cost and inference hardware constraints than any quality shortfall or breakdown in scaling laws.
- EDIT: Nesov notes that making use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM on inference chips, as is doing RL on bigger models. He expects it won’t be possible to do the next huge pretraining jump (to ~30T active) until ~2029.
- It would work, probably, if we had the data and HBM and spent the next $10bn, it’s just too expensive to bother with at the moment compared to:
RLVR scaling and inference scaling (or “reasoning” as we’re calling it), which kept things going instead. This boils down to spending more on RL so the resulting model can productively spend more tokens.
- But the feared / hoped-for generalisation from {training LLMs with RL on tasks with a verifier} to performing on tasks without one remains unclear even after two years of trying.² Grok 4 was apparently a major test of scaling RLVR training.³ It gets excellent benchmark results and the distilled versions are actually being used at scale. But imo it is the most jagged of all models.
- This rate of scaling-up cannot be sustained: RL is famously inefficient. Compared to SFT, it “reduces the amount of information a model can learn per hour of training by a factor of 1,000 to 1,000,000”. The per-token intelligence is up but not by much.
- There is a deflationary theory of RLVR, that it’s capped by pretraining capability and thus just about easier elicitation and better pass@1. But even if that’s right this isn’t saying much!
- RLVR is heavy fiddly R&D you need to learn by doing; better to learn it on smaller models with 10% of the cost.
- An obvious thing we can infer: the labs don’t have the resources to scale both at the same time. To keep the money jet burning, they have to post.
By late 2025, the obsolete modal “AI 2027” scenario described the beginning of a divergence between the lead lab and the runner-up frontier labs.⁴ This is because the leader’s superior ability to generate or acquire new training data and algorithm ideas was supposed to compound and widen their lead. Instead, we see the erstwhile leader OpenAI and some others clustering around the same level, which is weak evidence that synthetic data and AI-AI R&D aren’t there yet. Anthropic are making large claims about Opus 4.5’s capabilities, so maybe this will arrive on time next year.
For the first time there are now many examples of LLMs helping with actual research mathematics. But if you look closely it’s all still in-distribution in the broad sense: new implications of existing facts and techniques. (I don’t mean to demean this; probably most mathematics fits this spec.)
Extremely mixed evidence on the trend in the hallucination rate.
Companies make claims about their one-million- or ten-million-token effective context windows, but I don’t believe it.
In lieu of trying the agents for serious work yourself, you could at least look at the highlights of the gullible and precompetent AIs in the AI Village.

Here are the current biggest limits to LLMs, as polled in Heitmann et al:

Arguments for 2025 capabilities growth being above-trend

We now have measures which are a bit more like AGI metrics than dumb single-task static benchmarks are. What do they say?

Difficulty-weighted benchmarks: Epoch Capabilities Index.
- Interpretation: GPT-2 to GPT-3 was (very roughly) a 20-40 point jump.
Cognitive abilities: ADeLe.⁵
- Interpretation: level L is the capability held by 1 in 10^L humans on Earth. GPT-2 to GPT-3 was a 0.6 point jump.
Software agency: HCAST time horizon, the ability to handle larger-scale well-specified greenfield software tasks.
- Interpretation: the absolute values are less important than the implied exponential (a 7 month doubling time).

So: is the rate of change in 2025 (shaded) holding up compared to past jumps?:

Ignoring the (nonrobust)⁶ ECI GPT-2 rate, we can say yes: 2025 is fast, as fast as ever or more.

Even though these are the best we have, we can’t defer to these numbers.⁷ What else is there?

In May they passed some threshold and I finally started using LLMs for actual tasks. For me this is mostly due to the search agents replacing a degraded Google search. I’m not the only one who flipped this year. This hasty poll is worth more to me than any benchmark:

Or if you prefer a formal study (n=2,430 researchers):

On actual adoption and actual real-world automation:
- Based on self-reports, the St Louis Fed thinks that “Between 1 and 7% of all work hours are currently assisted by generative AI, and respondents report time savings equivalent to 1.4% of total work hours… across all workers (including non-users… Our estimated aggregate productivity gain from genAI (1.2%)”. That’s model-based, using year-old data, and naively assuming that the AI outputs are of equal quality. Not strong.
- The unfairly-derided METR study on Cursor and Sonnet 3.7 showed a productivity decrease among experienced devs with (mostly) <50 hours of practice using AI. Ignoring that headline result, the evergreen part here is that even skilled people turn out to be terrible at predicting how much AI actually helps them.
True frontier capabilities are likely obscured by systematic cost-cutting (distillation for serving to consumers, quantization, low reasoning-token modes, routing to cheap models, etc). Open models show you can now get good performance with <50B active parameters, maybe a sixth of what GPT-4 used.⁸
- GPT-4.5 was killed off after 3 months, presumably for inference cost reasons. But it was markedly lower in hallucinations and nine months later it’s still top-5 on LMArena. I bet it’s very useful internally, for instance in making the later iterations of 4o less terrible.
- See for instance the unreleased deep-fried multi-threaded “experimental reasoning model” which won at IMO, ICPC, and IOI while respecting the human time cap (e.g. 9 hours of clock time for inference). The OpenAI one is supposedly just an LLM with extra RL. They probably cost an insane amount to run, but for our purposes this is fine: we want the capability ceiling rather than the productisable ceiling. Maybe the first time that the frontier model has gone unreleased for 5 months?
LLM councils and Generate-Verify divide-and-conquer setups are much more powerful than single models, and are rarely ever reported.
Is it “the Year of Agents” (automation of e.g. browser tasks for the mass market)? Coding agents, yes. Search agents, yes. Other agents, not much (but obviously progress).
We’re still picking up various basic unhobbling tricks like “think before your next tool call”.
Character-level work is still occasionally problematic but nothing like last year.
GPT-5 costs a quarter of what 4o cost last year (per-token; it often uses far more than 4x the tokens). (The Chinese models are nominally a few times cheaper still, but are not cheaper in intelligence per dollar.)
People have been using competition mathematics as a hard benchmark for years, but will have to stop because it’s solved. As so often with evals called ahead of time, this means less than we thought it would; competition maths is surprisingly low-dimensional and so interpolable. Still, they jumped (pass@1) from 4% to 12% on FrontierMath Tier 4 and there are plenty of hour-to-week interactive speedups in research maths.
Recursive self-improvement: Deepmind threw AlphaEvolve (a pipeline of LLMs running an evolutionary search) at pretraining. They claim the JAX kernels it wrote reduced Gemini’s training time by 1%.
Extraordinary claims about Opus 4.5 being 100th percentile on Anthropic’s hardest hiring coding test, etc.
From May, the companies started saying for the first time that their models have dangerous capabilities.

One way of reconciling this mixed evidence is if things are going narrow, going dark, or going over our head. That is, if the real capabilities race narrowed to automated AI R&D specifically, most users and evaluators wouldn’t notice (especially if there are unreleased internal models or unreleased modes of released models). You’d see improved coding and not much else.

Or, another way: maybe 2025 was the year of increased jaggedness, trading off some capabilities against others. Maybe the RL made them much better at maths and instruction-following, but also sneaky, narrow, less secure (in the sense of emotional insecurity).

(You were about to nod sagely and let me get away without checking, but the ADeLe work also lets us just see if the jaggedness is changing.)

It is!

– Roger Grosse

Evals crawling towards ecological validity

Item response theory (Rausch 1960) is finally showing up in ML. This lets us put benchmarks on a common scale and actually estimate latent capabilities.
- ADeLE is my favourite. It’s a fully-automated and explains the abilities a benchmark is actually measuring, gives you an interpretable ability profile for an AI, and predicts OOD performance on new task instances better than embedding and finetunes (AUROC=0.8). Pre-dates HCAST task horizon, and as a special case (“VO”). They throw in a guessability control as well!
- These guys use it to estimate latent model ability, and show it’s way more robust across test sets than the average scores everyone uses. They also step towards automating adaptive testing: they finetune an LLM to generate tasks at the specified difficulty level.
- Epoch bundled 39 benchmarks together, weighting them by latent difficulty, and thus obsoleted the currently dominant Artificial Analysis index, which is unweighted.
- HCAST reinvents and approximates some of the same ideas. Come on METR!
Eleuther did the first public study of composing the many test-time interventions together. FAR and AISI also made a tiny step towards an open source defence pipeline, to use as a proxy for the closed compositional pipelines we actually care about.
Just for cost reasons, the default form of evals is a bit malign: it tests full replacement of humans. This is then a sort of incentive to develop in that direction rather than to promote collaboration. Two papers lay out why it’s thus time to spend on human evals.
The first paper using RL agents to attack fully-defended LLMs.
We have started to study propensity as well as capability. This is even harder.
This newsletter is essential.
The time given for pre-release testing is down, sometimes to one week.
No public pre-deployment testing by AISI between o1 and Gemini 3. Gemini 2.5 seems to have had no third-party pre-deployment tests.
A bunch of encouraging collaborations:
- The CoT Faithfulness Defence League; OpenAI testing Claude and Anthropic testing GPT; OpenAI/Anthropic/Deepmind; Apollo and OpenAI; AISI/CAISI/OpenAI; AISI/CAISI/Anthropic; AISI/Redwood; Redwood/Anthropic; METR/OpenAI; METR/Anthropic; Eval Forum; Various countries.
- Scale AI appears to be offering big companies pre-deployment testing for free? But the Meta investment presumably spoiled this.
- Some details about OAI external testing here, including some of the legal constraints verbatim.

Safety in 2025

Are reasoning models safer than the old kind?

Well, o3 and Sonnet 3.7 were pretty rough, lying and cheating at greatly increased rates. Looking instead at GPT-5 and Opus 4.5:

Much more monitorable via the long and more-faithful CoT (–> all risks down)
- “post-hoc rationalization… GPT-4o-mini (13%) and Haiku 3.5 (7%). While frontier models are more faithful, especially thinking ones, none are entirely faithful: Gemini 2.5 Flash (2.17%), ChatGPT-4o (0.49%), DeepSeek R1 (0.37%), Gemini 2.5 Pro (0.14%), and Sonnet 3.7 with thinking (0.04%)”
Much better at following instructions (–> accident risk down).
Much more likely to refuse malicious requests, and topic “harmlessness”⁹ is up 75% (–> misuse risk down)
Ambiguous evidence on jailbreaking (misuse risk). Even if they’re less breakable there are still plenty of 90%-effective attacks on them.
Much less sycophantic (cogsec risk down)
To get HHH Claude to hack a bank, you need to hide the nature of the task, lie to it about this being an authorised red team, and then still break down your malicious task into many, many little individually-innocuous chunks. You thus can’t get it to do anything that needs full context like strategising.
Anthropic’s own tests look bad in January 2025 and great in December:

But then

More autonomy (obviously agentic risk up)
More reward hacking (and so worse estimates of capability and risk). Note that reward hacking is not a silly or isolated or self-limiting kind of misalignment, owing perhaps to post-training inadvertently creating connections between it and the others.
Huge spike in eval awareness (and so worse estimates of capability and risk). Apollo had to rehaul their whole setup to keep up.
Gemini 2.5 is bad on a variety of safety measures (as measured by Anthropic). The mathematical discovery agent AlphaEvolve (Gemini 2.0) successfully hacked its overseers.
How much can we trust the above, given they can somewhat sabotage evals now?

So: lower propensity, higher risk when they go off - and all of this known with lower confidence?

The looming end of evals

Our evaluations are under pressure from cheating, sandbagging, background safety, under-elicitation, and deception. We don’t really know how much pressure. This is on top of evals usually being weak proxies, contaminated, label-noised, unrealistic, and confounded in various ways.

Sonnet 4.5 detected evaluation 58% of the time (vs 22% for Opus 4.1). Does it act on this awareness? Still, it was found to “only engage in [covert subversion] in rare circumstances” when strongly incentivised. When Anthropic attempted to suppress the eval-awareness with steering, its misaligned behavior rate increased from 0% to between 1% and 9% on synthetic honeypots.
We expect exploration hacking (sandbagging during RL training) but afaik haven’t seen it yet.
Maybe subliminal learning leads to a silent eval confounder (if you’re using the exact same model for the judge). Yet another argument for multi-model separation of concerns.
Opus 4.5 was deployed based on a qualitative survey of Anthropic staff about its R&D risk being low (on top of the existing evals, which weren’t enough):
- “In the past, rule-outs have been based on well-defined automated task evaluations. However, Claude Opus 4.5 has roughly reached the pre-defined thresholds we set for straightforward ASL-4 rule-out based on benchmark tasks. These evaluations represent short-horizon subtasks that might be encountered daily by a junior researcher, rather than the complex long-horizon actions needed to perform the full role. The rule-out in this case is also informed by a survey of Anthropic employees who are intensive Claude Code users, along with qualitative impressions of model capabilities for complex, long-horizon tasks.”

Prosaic misalignment

Still, we see misalignment when we look for it, so the lying is not that strong. (It is lucky that we do see it, since it could have been that scheming only appeared at later, catastrophic capability levels.)

In o1, 1 in 600 (synthetic but uncontrived) prompts resulted in something which looks like deception.
o3 does reward hacking (cheating/lying) in >2% of hard normal tasks (or 44% of impossible tasks).
A nice propensity study shows that (n=10) natural prompts usually result in no scheming (“a maximum of 4% [rate of attempting manipulation] for the codex snippet on Grok”). But there’s extremely high sensitivity to the scaffold: “removing access to tools can cause models such as Claude Opus 4.1 to scheme at rates up to 30%, versus zero scheming with tools present”.
When we contrive situations for misalignment to show up, it also shows up. Nostalgebraist argues that this is probably substantially because our evals sound like bad fiction and this activates role-playing-along behaviour.
The joke about xAI’s safety plan (that they promote AI safety by deploying cursed stuff in public and so making it obvious why it’s needed) is looking ok. And not only them.
It is a folk belief among the cyborgists that bigger pretraining runs produce more deeply aligned models, at least in the case of Opus 3 and early versions of Opus 4. (They are also said to be “less corrigible”.) Huge if true.
There may come a point where the old alliance between those working to make the AIs corrigible and those working to give them prosocial values comes apart.
One term for the counterintuitive safety approach which includes treating them like people, giving them lines of retreat, making deals, and inoculation prompting could be “voluntary alignment”.

Fully speculative note: Opus 4.5 is the most reliable model and also relatively aligned. So might it be that we’re getting the long-awaited negative alignment taxes?

What is the plan?

The world’s de facto alignment strategy remains “iterative alignment”, optimising mere outputs with a stack of admittedly weak alignment and control techniques. Anthropic have at least owned up to this being part of their plan.

What is the current stack? We don’t know; they won’t tell us. Anthropic did release some details about the pipeline which watches for eval sabotage. Without knowing it we can’t criticise or red-team it or analyse the correlation between faults in the elements. Red-teamers don’t know at which stage an unsuccessful attack was blocked. And external safety research is done piecemeal, testing methods one at a time, rather than in anything like the actual deployment environment.
Some progress in replicating an open stack:
- Eleuther tested a few hundred compositions. A couple of classifiers as a first step towards a proxy defence pipeline
- OpenAI open safeguards, worse than their internal ones but good.
- Open character training
- A basic composition test
OpenAI’s plan, announced in passing in the gpt-oss release, is to have a strict policy and run a “safety reasoner” to verify it very intensely for a little while after a new model is launched and to then relax: “In some of our [OpenAI’s] recent launches, the fraction of total compute devoted to safety reasoning has ranged as high as 16%” but then falls off… we often start with more strict policies and use relatively large amounts of compute where needed to enable Safety Reasoner to carefully apply those policies. Then we adjust our policies as our understanding of the risks in production improves”. Bold to announce this strategy on the internet that the AIs read.
The really good idea in AI governance is creating an off switch. Whether you can get anyone to use it once it’s built is another thing.
We also now have a name for the world’s de facto AI governance plan: “Open Global Investment”.

Some presumably better plans:

A longlist. Some governance plans.
Deepmind, Carlsmith, Hobbhahn, Peregrine, Bowman, Clymer, Buterin, Jones, Greenblatt.
Gradual disempowerment is an exciting frame, but not a core safety agenda. It’s what might get you after you solve alignment and avoid global dictatorship.

Things which might fundamentally change the nature of LLMs

Training on mostly nonhuman data
- Much larger RL training;
- Intentionally synthetic data;
- Unintentionally synthetic data from internet slop;
- Multi-agent training.
Letting the world mess with the weights, aka continual learning.
Neuralese and KV communication
Agency.
- Chatbot safety doesn’t generalise much to long chains of self-prompted actions.
- A perceptual loop into training. In 2023 Kulveit identified web I/O as the bottleneck on LLMs doing active inference, i.e. being a particular kind of effective agent. Last October, GPT-4 got web search, and this may have been a bigger deal than we noticed: it gives them a far faster feedback loop, since their outputs often end up there and agents are now putting it there themselves. This means that more and more of the inference-time inputs will also be machine text.
Multi-agency. This is actually already here:
- By now, consumer “models” are actually multiagent systems: everything goes through filter models (“guardrails”) on the way in and out. This separation of concerns has some nice properties, a la debate. But it also makes the analysis even harder.
- It would surely be overinterpreting persona features to say that each individual model is itself a bunch of guys, itself a multi-agent system.
- But there’s huge scope for them to make each other weirder at runtime when they interact a million times more than they currently do.

Emergent misalignment and model personas

We already knew from jailbreaks that current alignment methods were brittle. Emergent misalignment goes much further than this (given a few thousand finetuning steps). (“Emergent misalignment” isn’t a great name. I would have called it “misalignment generalisation”, or misgen.)
But besides yet another massive security problem and failure of prosaic alignment methods, it’s good news!: the models correctly correlate bad things together and can thus be pushed in the other direction.
And here’s a possible example of positive generalisation (honesty about silly errors leading to honesty about hidden objectives).
Previously I thought that “character training” was a separate and lesser matter than “alignment training”. Now I am not sure. Example unsharpened hypothesis in this class: Is there any connection between Gemini’s excess misalignment and its self-esteem issues?
- See the new model spec and model personas agendas in the main post.
- Kulveit’s informal model of model psychology implies that character is deeper and so more robust than e.g. typical RLHF (but not as deep as putative ground-layer alignment).
- Are we anthropomorphising too much or not enough? Well, both, but we don’t yet know how.

Monitorability

For the first time, public mech interp tools sometimes work somewhat. This is just in time for models to be systematically aware of when they’re being tested.
- e.g. You can find and suppress “evaluation awareness” features at test-time. This helps prevent models from reporting that they know they’re being tested.¹⁰
- e.g. You can discover (toy) hidden goals quite quickly.
- e.g. You can run a real-time hallucination detector with cheap linear probes in real-time as your model streams.
- e.g. SAEs are actually deployed somewhere, in a large enterprise, for personal info identification.
- e.g. We know that LLMs can plan inside one forward pass, and how Claude plans: simultaneous plans; no distinct “plan features” (no separate scheming mode).
Lots of powerful people declared their intent to not ruin the CoT. But RLed CoTs are already starting to look weird (“marinade marinade marinade”) and it may be hard to avoid.
OpenAI were leading on this. As of September 2025, Anthropic have stopped risking ruining the CoT. Nothing I’m aware of from the others.
We will see if Meta or Tencent make this moot.
Anthropic now uses an AI to red-team AIs, calling this an “auditing agent”. However, the definition of “audit” is independent investigation, and I am unwilling to call black-box AI probes “independent”. I’m fine with “investigator”; there are lots of investigators I don’t trust.

New people

Welcome to the many new people. I’ve added a new, sprawling top-level category for one large trend among them, which is to treat the multi-agent lens as primary in various ways (see e.g. Softmax, Full Stack, collective intelligence, as well as old-timers Critch, Ngo, and CAIF).
A major world government now has an AI alignment agenda.
Some notable work from China. See e.g. Concordia’s digest.

Overall

I wish I could tell you some number, the net expected safety change, this year’s improvements in dangerous capabilities and agent performance, minus the alignment-boosting portion of capabilities, minus the cumulative effect of the best actually implemented composition of alignment and control techniques. But I can’t.

(Nano Banana 3-shot in reference to this tweet.)

Discourse in 2025

The race is now a formal part of lab plans. Quoting Algon:

if the race heats up, then these [safety] plans may fall by the wayside altogether. Anthropic’s plan makes this explicit: it has a clause (footnote 17) about changing the plan if a competitor seems close to creating a highly risky AI…

The largest [worries are] the steps back from previous safety commitments by the labs. Deepmind and OpenAI now have their own equivalent of Anthropic’s footnote 17, letting them drop safety measures if they find another lab about to develop powerful AI without adequate safety measures. Deepmind, in fact, went further and has stated that they will only implement some parts of its plan if other labs do, too…

Anthropic and DeepMind reduced safeguards for some CBRN and cybersecurity capabilities after finding their initial requirements were excessive. OpenAI removed persuasion capabilities from its Preparedness Framework entirely, handling them through other policies instead. Notably, Deepmind did increase the safeguards required for ML research and development.

Also an explicit admission that self-improvement is the thing to race towards:

In August, the world’s first frontier AI law came into force (on a voluntary basis but everyone signed up, except Meta). In September, California passed a frontier AI law.
That said, it is indeed off that people don’t criticise Chinese labs when they exhibit even more negligence than Meta. One reason for this is that, despite appearances, they’re not frontier; another is that you’d expect to have way less effect on those labs, but that is still too much politics in what should be science.
The last nonprofit among the frontier players is effectively gone. This “recapitalization” was a big achievement in legal terms (though not unprecedented). On paper it’s not as bad as it was intended to be. At the moment it’s not as bad as it could have been. But it’s a long game.
At the start of the year there was a push to make the word “safety” low-status. This worked in Whitehall and DC but not in general. Call it what you like.
Also in DC, the phrase “AI as Normal Technology” was seized upon as an excuse to not do much. Actually the authors meant “Just Current AI as Normal Technology” and said much that is reasonable.
The CCP did a bunch to (accidentally/short-term) slow down Chinese AI this year.
System cards have grown massively: GPT-3’s model card was 1000 words; GPT-5’s is 20,000. They are now the main source of information on labs’ safety procedures, among other things. But they are still ad hoc: for instance, they do not always report results from the checkpoint which actually gets released.
Yudkowsky and Soares’ book did well. But Byrnes and Carlsmith actually advanced the line of thought.
Some AI ethics luminaries have stopped downplaying agentic risks.
Two aspirational calls for “third-wave AI safety” (Ngo) and “third-wave mechanistic interpretability” (Sharkey).
I’ve never felt that the boundary I draw around “technical safety” for these posts was all that convincing. Yet another hole in it comes from strategic reasons to implement model welfare / archive weights / model personhood / give lines of retreat. These plausibly have large effective-alignment effects. Next year my taxonomy might have to include “cut a deal with them”.
US settlement on pretraining corpus IP; the ruling would likely have been that training on books without permission is fair use, but storing the pirated copies afterwards isn’t. Anthropic was first to lose a class-action lawsuit on the matter and will pay authors/publishers something north of $1.5bn. US precedent that language models don’t defame when they make up bad things. German precedent that language models store data when they memorise it, and therefore violate copyright. Chinese precedent that the user of an AI has copyright over the things they generate; the US disagrees.
Four good conferences, three of them new: you can see the talks from HAAISS and IASEAI and ILIAD, and the papers from AF@CMU. Pretty great way to learn about things just about to come out.

Cruxes for next year (with Manifold markets):

Is “reasoning” mostly elicitation and therefore bottlenecked on pretraining scaling? [Manifold]
Does RL training on verifiers help with tasks without a verifier? [Manifold]
Is “frying” models with excess RL (harming their off-target capabilities by overoptimising in post-training) just due to temporary incompetence by human scientists? [Manifold]
Is the agent task horizon really increasing that fast? Is the rate of progress on messy tasks close to the progress rate on clean tasks? [Manifold]
Some of the apparent generalisation is actually interpolating from semantic duplicates of the test set in the hidden training corpuses. So is originality not increasing? Is taste not increasing? Does this bear on the supposed AI R&D explosion? [Manifold]
The “cognitive core” hypothesis (that the general-reasoning components of a trained LLM are not that large in parameter count) is looking surprisingly plausible. This would explain why distillation is so effective. [Manifold]
“How far can you get by simply putting an insane number of things in distribution?” What fraction of new knowledge can be produced through combining existing knowledge? What dangerous things are out there, but safely spread out in the corpus? [Manifold]
- Conversely, what fraction of the expected value of new information requires empiricism vs just lots of thinking?

Cuts

Various things I cut from the above:

**Adaptiveness and Discrimination**

There is [some](https://www.pnas.org/doi/10.1073/pnas.2415697122) [evidence](https://arxiv.org/abs/2511.00926) that AIs treat AIs and humans differently. This is not necessarily bad, but it at least enables interesting types of badness.

With my system prompt (which requests directness and straight-talk) they have started to patronise me: ![Patronising screenshot](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/klptexglr46miykqiqj1)

**Training awareness**

Last year it was not obvious that LLMs remember anything much about the RL training process. Now it's [pretty](https://arxiv.org/abs/2406.11715) [clear](https://x.com/repligate/status/1994973338448662858). (The soul document was used in both SFT and RLHF though.)

**Progress in non-LLMs**

"World model" means at least four things:

1. [A learned model](https://arxiv.org/abs/1803.10122) of environment dynamics for RL, allowing planning in latent space or training in the model's "imagination."

2. The new one: just a 3D simulator; a game engine inside a neural network ([Deepmind](https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/), Microsoft). The claim is that they implicitly learn physics, object permanence, etc. The interesting part is that they take actions as inputs. [Here's](https://copilot.microsoft.com/labs/experiments/copilot-gaming-experiences) Quake running badly on a net. Maybe useful for agent training.

3. [If](https://www.neelnanda.io/mechanistic-interpretability/othello) LLM representations are stable and effectively symbolic, then people say it has a world model.

4. A predictive model of reality learned via self-supervised learning. The touted [LeJEPA](https://arxiv.org/pdf/2511.08544) semi-supervised scheme on small (15M param) CNNs is domain-specific. It does better on one particular transfer task than *small* vision transformers, presumably worse than large ones.

The much-hyped [Small Recursive Transformers](https://magazine.sebastianraschka.com/i/177848019/small-recursive-transformers) only work on a single domain, and do a bunch [worse](https://drive.google.com/file/d/1-kg2JsXYvsTAyMVxKWG7VpdcQLpmoGc7/view?usp=sharing) than the frontier models for about the same inference cost, but have truly tiny training costs, O($1000).

[HOPE](https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/) and [Titan](https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/?utm_source=twitter&utm_medium=social&utm_campaign=social_post&utm_content=gr-acct) might be nothing, might be huge. They don't scale very far yet, nor compare to any real frontier systems.

Any of these taking over could make large swathes of Transformer-specific safety work irrelevant. (But [some](https://x.com/chingfang17/status/1997080064345936028) methods are surprisingly robust.)

The "[cognitive core](https://www.seangoedecke.com/cognitive-core/)" hypothesis (that the general-reasoning components of a trained LLM are not that large in parameter count) is looking [plausible](https://x.com/Dorialexander/status/1987933205199274359). The contrary hypothesis ([associationism](https://www.dwarkesh.com/p/sholto-douglas-trenton-bricken?hide_intro_popup=true#:~:text=if%20it%27s%20all-,associations%20all%20the%20way%20down,-%2C%20does%20that%20mean)?) is that general reasoning is just a bunch of heuristics and priors piled on top of each other and you need a big pile of memorisation. It's also a live possibility: for instance, consider that a year of intense RLVR only led to task-specific improvements.

![ADeLe scaling laws](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/j76edumltnfb2ihn6nmu) *"the very first scaling laws of the actual abilities of LLMs", from [ADeLe](https://arxiv.org/pdf/2503.06378).* *KNs = Social Sciences and Humanities, AT = Atypicality, and VO = Volume (task time).* *The y-axis is the logistic of the [subject characteristic curve](https://arxiv.org/pdf/2503.06378#page=9) (the chance of success) for each skill.*

**Other**

[Model](https://arxiv.org/abs/2511.08579) [introspection](https://transformer-circuits.pub/2025/introspection/index.html) is somewhat real.

[Vladimir Nesov](https://www.lesswrong.com/users/vladimir_nesov) continues to put out some of the best hardware predictions pro bono.

Jason Wei has a [very wise post](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) noting that verifiers are still the bottleneck and existing benchmarks are overselected for tractability.

There are now "[post-AGI](https://x.com/sebkrier/status/1995515865157321070)" teams.

Kudos to Deepmind for being the first to release output watermarking and a semi-public detector. You can nominally sign up for it [here](https://docs.google.com/forms/d/e/1FAIpQLSfAYrauHmY-PpUNxL4Fs6coa185CtKWp7TnEXL0tKbAezo4MQ/viewform).

Previously, Microsoft's deal with OpenAI [stipulated](https://www.msn.com/en-us/news/technology/microsoft-just-made-sure-openai-can-t-declare-agi-alone/ar-AA1PpLTK#:~:text=OpenAI%E2%80%99s%20new%20public%20benefit%20structure%20and%20ongoing%20Microsoft%20deal%20allow%20both%20companies%20to%20pursue%20AGI%20independently) that they couldn't try to build AGI. [Now they can](https://www.semafor.com/article/11/05/2025/microsoft-superintelligence-team-promises-to-keep-humans-in-charge) (try). Simonyan is in charge, despite Suleyman being the one on the press circuit.

Major insurers are [nervous](https://www.ft.com/content/abfe9741-f438-4ed6-a673-075ec177dc62?accessToken=zwAAAZq1faz0kdOr_pdB9DhO1tOmcwdewXfcYg.MEUCIQCRFate6aeSALClx6FBsPCQw_F7YLpdF81RgLxw8EOk9wIgKHE666mkD_jI-BV90bcF0HnjXWWDW6-QLEzO9Fg06dg&segmentId=e95a9ae7-622c-6235-5f87-51e412b47e97&shareType=enterprise&shareId=6c04e38e-15ed-472c-bcbb-4500695cf776) about AI agents (but asking the government for an exclusion isn't the same as putting them in the policies).

**Offence/defence balance**

This post doesn't much cover the [hyperactive](https://www.aiat.report/) and talented AI cybersecurity world (except as it overlaps with things like robustness). One angle I will bring up: We can now [find](https://www.lesswrong.com/posts/F5QAGP5bYrMMjQ5Ab/aisle-discovered-three-new-openssl-vulnerabilities-1) critical, decade-old security bugs in extremely well-audited software like OpenSSL and sqlite. Finding them is very fast and cheap. Is this good news?

- Well, red-teaming makes many attacks into a defence, as long as you actually do the red-team.

- But Dawn Song [argues](https://rdi.berkeley.edu/frontier-ai-impact-on-cybersecurity/) that LLMs overall favour offence, since its margin for error is so broad, since remediation is slow and expensive, and since defenders are less willing to use unreliable (and itself insecure) AI. And can you blame them?

- See also "[just in time](https://www.splunk.com/en_us/blog/security/lamehug-ai-driven-malware-llm-cyber-intrusion-analysis.html) AI malware" where the payload contains no suspicious code, just a call to HuggingFace.

**Egregores and massively-multi-agent mess**

![Egregores image](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/xlz7av9qpn811z2zuiqm) - There is something [wrong](https://www.lesswrong.com/posts/6ZnznCaTcbGYsCmqu/the-rise-of-parasitic-ai#:~:text=This%20is%20likely%20due%20to%20OpenAI%20retiring%20ChatGPT4o%20on%20August%207th.) ([something](https://x.com/25KarmaIsAbitch/status/1987088461694734483) [horribly right](https://arstechnica.com/information-technology/2025/08/openai-brings-back-gpt-4o-after-user-revolt/)) [with](https://x.com/arcangel3ac/status/1996297082764681248) 4o. Blinded users [still](https://lmarena.ai/leaderboard/text) prefer it to gpt-5-high, and this surely is due to both them simply [liking](https://x.com/voooooogel/status/1987375660785148112) its style and dark stuff like sycophancy. It will live on through illicit [distillation](https://x.com/aiamblichus/status/1987267132598497374) and in-context [transference](https://x.com/repligate/status/1988813712572952815). Shame on OpenAI for [making](https://thezvi.substack.com/p/gpt-4o-sycophancy-post-mortem) this mess; kudos to OpenAI for doing unpopular damage control and good luck to them [in round 2](https://x.com/Miles_Brundage/status/1991603234746822888).

Open models will presumably eventually overrun them in the codependency market segment. See [Pressman](https://minihf.com/posts/2025-07-22-on-chatgpt-psychosis-and-llm-sycophancy/) for a sceptical timeline and [Rath and Armstrong](https://arxiv.org/pdf/2508.15748) for a good idea.

- More generally there is [pressure](https://x.com/krishnanrohit/status/1987018487001457141) from users to refuse less, flatter more, and replace humans more; yet another economic constraint on for-profit AI.

- Whether it's the [counterfactual](https://andymasley.substack.com/p/stories-of-ai-turning-users-delusional) cause of mental problems or not, so–called "LLM psychosis" is now a common path of pathogenesis. Note that the symptoms are [literally](https://www.wired.com/story/ai-psychosis-is-rarely-psychosis-at-all/) not psychotic (they are delusions).

![LLM psychosis](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/iijqKj2rzwz2ouRon/aauurmsxrfuqzvi9emaj)

My politics

Tue, 18 Nov 2025 00:00:00 +0000

Politically, I am a tragedian:

It is tragic that we need governments, that we can’t trust markets to deliver what everyone needs.

It is tragic that we need markets, that we can’t just trust government to deliver what everyone needs (let alone what they want) and to not crush the Other.

It is tragic that we need corporations, that economies of scale are so important that we must risk monopolies and cronies and skinwalkers.

It is tragic that we need bureaucracies, that the human urge for favoritism and self-dealing is so strong and ruinous that it’s worth imprisoning everyone on earth in a cage of stupid rules.

It is tragic to be forced to choose to not be free.

Comments

Baran Cimen commented on 18 November 2025:

It is tragic that you can't say it straight and have to hide behind irony. It is tragic that you are not free despite not being forced to be unfree, which would be a contradiction.

Paper AI Tigers

Sun, 16 Nov 2025 00:00:00 +0000

The best Chinese LLMs offer

frontier performance on some benchmarks;
massive per-token discounts (~3x on input, ~6x on output);
the weights. On-prem with fully free ~MIT licence, self-hosting, white-box access, customisation, with zero markup (and in fact zero revenue going to the Chinese companies);
with a bit of work you can get much faster token speeds than the closed APIs;
less overrefusal (except on CCP talking points);
on topics controversial in the West, less nannying.
they just added the search agents that make daily use actually worthwhile. They also let you see the real CoT!
They’re the most-downloaded open models.

As a result, going off private information, money man Martin Casado says “16-24%” of the (American) startups he meets are now using Chinese models. Among the few Westerners to admit it is Airbnb (Qwen). But Windsurf’s planner is probably GLM and Cursor’s planner may be DeepSeek.

And yet

outside China, they are mostly not used, even by the cognoscenti. Not a great metric, but the one I've got: all Chinese models combined are currently at 19% on the highly selected group of people who use OpenRouter. More interestingly, over 2025 they trended downwards there. And of course in the browser and mobile they're probably <<10% of global use;
they are severely compute-constrained, so this implies they actually can't have matched American models;
they're aggressively quantizing at inference-time, 32 bits to 4;
state-sponsored Chinese hackers used closed American models for incredibly sensitive operations, giving the Americans a full whitebox log of the attack!

What gives?

"Tigers"?

The title alludes to the "6 AI Tigers" named in business rags as DeepSeek, Moonshot, Z.ai, MiniMax, StepFun, and 01.ai. (This is because they're trying to hype startups specifically; the conglomerates Alibaba and Baidu are way more relevant than the latter two.)

Filtered evidence

The evidence is dreadful because everyone has a horse in the race and (in public) is letting it lead their speech:

Static evals are weak evidence even when they're not being adversarially hacked and hill-climbed.
Some Americans are downplaying the Chinese models out of cope.
Some Americans are hyping the Chinese models to suppress domestic AI regulation.
Some Americans are hyping the Chinese models to boost international AI regulation.
The Chinese are obviously talking their book.

What could explain this?

Maybe the evals are misleading?

1. frontier performance on some benchmarks

The naive view - the benchmark view - is that they’re very close in “intelligence”:

But these benchmarks are not strong evidence about performance on new inputs or the latent (general and unobserved) capabilities. It’d be natural to read “89%” success on a maths benchmark as meaning an 89% probability that it would correctly handle unseen questions of that difficulty in that domain (and indeed this is what cross-validation was originally designed to estimate). But in the kitchen-sink era of AI, where every system has seen a large proportion of all data ever digitised, and so has already seen some variant of many benchmark questions, you can’t read it that way.

In fact it’s not even an 89% probability of answering these same questions right again, as shown by the fact that people report the results as “avg@64” (the average performance if you ask the same question 64 times).

Aside: Test sets which are on the internet are not test sets.

There are dozens of ways to screw up or hack these numbers. I’ll only look at a couple here but I welcome someone doing something more systematic.

Even less generalisation?

Maybe Chinese models generalise to unseen tasks less well. (For instance, when tested on fresh data, 01’s Yi model fell 8pp (25%) on GSM - the biggest drop amongst all models.)

We can get a dirty estimate of this by the “shrinkage gap”: look at how a model performs on next year’s iteration of some task, compared to this year’s. If it finished training in 2024, then it can’t have trained on the version released in 2025, so we get to see what they’re like on at least somewhat novel tasks. We’ll use two versions of the same benchmark to keep the difficulty roughly on par. Let’s try AIME:

AIME 2024 vs 2025 Model Performance
(using the Artificial Analysis harness)

Model	AIME 2024	AIME 2025	pp fall	% fall
Kimi K2	69.3	57.0	-12.3	-17.7
MiniMax-M1 80k	84.7	61.0	-23.7	-28.0
DeepSeek-v3	39.2	26.0	-13.2	-33.7
DeepSeek V3 0324	52.0	41.0	-11.0	-21.2
Qwen3 235B (Reasoning)	84.0	82.0	-2.0	-2.4
Kimi K2-Instruct	69.6	49.5	-20.1	-28.9
DeepSeek R1 0528	89.3	76.0	-13.3	-14.9

Chinese models			-13.7pp	-21%


Gemini-2.5 Pro	88.7	87.7	-1.0	-1.1
Gemini 2.5 Flash (Reasoning)	82.3	73.3	-9.0	-10.9
Claude 4 Opus Thinking	75.7	73.3	-2.4	-3.2
o4-mini (high)	94.0	90.7	-3.3	-3.5
GPT-4.1	43.7	34.7	-9.0	-20.6
Nova Premier	17.0	17.3	0.3	1.8
GPT-4o Nov 24	15.0	6.0	-9.0	-60.0
Magistral Medium	73.6	64.9	-8.7	-11.8
Claude 3.7 Sonnet	61.3	56.3	-5.0	-8.2
OpenAI-o1-0912	74.4	71.5	-2.9	-3.9
o3	90.3	88.3	-2.0	-2.2
Grok 4	94.3	92.7	-1.6	-1.7

Western models			-4.5pp	-10.4%


Overall average	68.3	60.5	-7.9	-14.3

Almost all models get worse on this new benchmark, despite 2025 being the same difficulty as 2024 (for humans). But as I expected, Western models drop less: they lost 10% of their performance on the new data, while Chinese models dropped 21%. p = 0.09.

Averaging across crappy models for the sake of a cultural generalisation doesn’t make sense. Luckily, rerunning the analysis with just the top models gives roughly the same result (9% gap instead of 11%).

One way for generalisation to fail despite apparently strong eval performance is contamination, training on the test set. But (despite the suggestive timing) the above isn’t strong evidence that that’s what happened. It just tells us that Kimi and MiniMax and DeepSeek generalise worse on this task; it doesn’t tell us why.

Details

Here's a Colab with everything except the actual execution of my silly manual Kimi 1.5 run.

First, test for an obvious confounder: check if the 2025 AIME exam was around as hard as 2024's (answer: yes; in fact humans did 4% better in 2025). (TODO: check if 2025 had more combinatorics, which AI struggles with.)

(To be strict we should limit this to models which finished training before 12th February 2025, when the questions were released. But, as you see, we don't need to, it's a very clear result anyway.)

Selection criteria:

Models tested on both AIME 2024 and 2025 with the same weights and harness
Ideally finished training by 15th February 2025
Subanalyses can handle filtering to the most relevant models (the frontier in each group)

ML results are too sensitive to eval harnesses to use just one setting. Luckily I found four comparisons of AIME 2024 and AIME 2025 by different groups, Artificial Analysis, the Zettlemoyer Lab, GAIR, and Vals, and MathArena. AA is the one in the table above.

Qwen 2.5

Qwen3 seems clean on this benchmark, but multiple lines show that Qwen 2.5 trained on test (or at least rephrased test data and then trained on it). We know this because random rewards work on it nearly as well as correct rewards. This adds no information by definition, so the model must have already known the answers. "Intriguingly, we find that any AIME24 gains achievable from training Qwen models with spurious rewards largely vanish when evaluating on AIME 2025." Taking the max performance of the random reward curve, they fall 88% [75%, 100%].

Even more damning, when you give Qwen2.5-7B the first 40% of a MATH-500 test problem, it can reproduce the remaining 60% of the question word-for-word (with 41.2% accuracy). Llama3.1-8B fails at this completely (2%).

In this paper the QwQ Qwen reasoning model was the worst by far, 60% contaminated.

How did our replications do? As expected, the shrinkage gap varies a lot by harness and by model choice:

UoW-Zettlemoyer: Qwen2.5-7B and Qwen2.5-Math fall 12.5pp (88%); the open Western LLaMA and OLMo models were too weak to really say.

GAIR

Vals actually get no gap: -11.2% vs -10.8%. If you kick Meta out the gap goes up to 2%, still not much.
TODO: add MathArena. "QWQ-PREVIEW-32B is a notable outlier and outperforms the expected human-aligned performance by nearly 60%, indicating extreme contamination"

I'm not worried about these contradictory results; they both just include a lot of bad models and so noise. (I don't actually care how Llama 4 Scout's generalisation compares to QwQ-uwu-435B-A72B-destruct-dpo-ppo-grpo-orpo-kto-slerp-v3.5-beta2-chat-instruct-base-420-blazeit-early-stopped-for-vibes.) GAIR is also underelicited.

(Actually AIME's a funny choice of benchmark given that 2025 had a bunch of semantic duplicates from before the cutoff. But that just makes the above a lower bound on the fall in performance.)

A win for Qwen and a huge relative win for Amazon!

Claude is adorably confused about this. I didn't even ask it for this analysis:

TODO: Another way to get past goodharting pressure is to look at hard but obscure evals which no one ever reports / which manage to keep the test set private. e.g. PROOFGRID.

Kimi 1.5

I really wanted to include Kimi 1.5, because it finished training just around the time AIME 2025 came out. But it turns out they never actually released the weights and it's been removed from the API!

Because I have that dawg in me, I decided to manually evaluate it in the last place I can, the goddamn browser chat. This is suboptimal in many ways (no control over temperature, max tokens, etc) but I can do it for both and hopefully the fall is proportional. The usual practice is to repeat 8 or 64 times, but I have patience enough for 2.

I used the Mistral prompt:


Solve this AIME mathematical problem step by step.

Problem: {}

Think through this carefully and provide your final answer as a 3-digit integer (000-999).

End with: "Therefore, the answer is [your answer]."

Results:

	AIME 2024 acc	AIME 2025 acc	abs fall (pp)	rel fall (%)
Kimi 1.5 (browser)	18.3 [23.3, 13,3]	15.0 [13.3, 16.6]	-3.3	-18%

We can actually see how much weaker this (null) harness and provider is at eliciting performance by comparing to their reported results. Their short-CoT result for AIME 2024 was 60.8%.

For obvious reasons I'm not including this in the main analysis but it's another example.

2026 EDIT: Igor Kotenkov tests Kimi 2.5T on every new benchmark released after its training and finds a huge -24pp deficit compared to OpenAI/Anthropic.

Latent capabilities

My favourite paper of the year introduces a way to do real psychometrics on LLMs, breaking it down into 18 fundamental capabilities.

The DeepSeek R1 32B distill they test is about as good (total area) as o1-mini. Not bad!

TODO: Run Kimi against GPT-5.1.

Pre-1960s statistics

The bundled score people use is the Artificial Analysis one, because they have a very nice UI. But they give every benchmark equal weight, when in fact they differ hugely in hardness! Epoch’s index estimates difficulties properly and show

TODO: Wait for Epoch to do KimiK2T.

This still suffers from GIGO but is better.

‘Hacking

Another way to be misleading is to volkswagen it: put special and unrepresentative effort in during testing, “hacking”. e.g. Kimi’s benchmarks come from “Heavy mode” (8 parallel instances with an aggregation instance on top). You can’t do this via the API or out of the box with the weights. (Could you say the same for OpenAI?)

Or you can run the test on a model which is better than the one you serve. Moonshot credibly claim to have reported their benchmarks at the same low-precision quantization (INT4) that they serve users, but others don’t claim this.

In fairness

I should also say the Chinese models do very well on LMArena - despite being unfairly penalised. But Arena is a poor measure of actual ability. It is a decent test of style though. I put this gap down to American labs overoptimising: post-training too hard and putting all kinds of repugnant corporate ass-covering stuff in the spec.

Also Qwen is famous for ‘capability density’: the small versions are surprisingly smart for their size. But do you know that GPT-5-nano isn’t 7B?

The D word

Distillation is second-rate intelligence, and there’s some evidence that they are distilling off of American models to some extent. See also the excellent Slop Profile from EQ Bench, which estimates that the new Kimi is closer to Claude than its own base model.

But anyway I don’t claim this is a major factor here, maybe another 5%.

The above isn’t novel; it’s common knowledge there’s some latent capabilities gap. This is often put in terms of them being “3 months behind”, but these estimates are still assuming that brittle, ad hoc, and heavily goodharted benchmarks have good external validity. I’d guess more like 12 months.

Unreliability?

1. frontier performance on some benchmarks

The above benchmarks are mostly single-shot, but people are now pushing LLMs to do more complicated stuff. One very flawed measure of this is the HCAST time horizon for software engineering: on that, DeepSeek R1 had a 31 minute “50% time horizon” compared to Opus 4’s 80 minutes.

There are various worse agent benchmarks, and e.g. the new Kimi posts great numbers on them. But on vibe I’d bet on a >3x reliability advantage for Claude.

EDIT: Kimi K2 Thinking ended up at the same task horizon as Sonnet 3.7 (a model 9 months older than it) on HCAST, with an asterisk (the provider they had to use for privacy reasons may well have underelicited the model).

Harder to elicit?

As well as reliability over time, there’s stability over inputs. High variance in performance, for instance because the exact form of the inputs matters more.

On Vending-Bench, there’s a huge gap in performance between the Moonshot API and Moonshot models provided by a third-party provider. This is evidence of three things:

maybe Kimi is more fiddly (sensitive to the prompt and hyperparameters);
maybe the providers haven’t learned how to elicit performance from them yet (testable by just waiting);
maybe they were using questionable research practices in the self-reported runs.

TODO: I’ve been meaning to run the obvious experiment, which is to just see if they have a bigger gap between pass@1 and pass@64 success rates.

TODO: Or I could intentionally underelicit! Rerun the models on AIME 2024 with only a basic prompt. My results will be lower; the gap tells us how much the labs’ own intense tuning helps / is necessary. This tells us something about, not their capability, but their actual in-the-wild performance with normal lazy users.

Tokenomics: no effective discount

2. massive per-token discounts (~3x on input, ~6x on output)

Distinguish intelligence (max performance), intelligence per token (efficiency), and intelligence per dollar (cost-effectiveness).

The 5x discounts I quoted are per-token, not per-success. If you had to use 6x more tokens to get the same quality, then there would be no real discount. And indeed DeepSeek and Qwen (see also anecdote here about Kimi, uncontested) are very hungry:

And in this graph you can clearly see a 2-4x difference (with Gemini and Kimi K2-base as the big exceptions):

And the resulting cost is a mixed bag:

I won’t use AA’s efficiency estimates, because again I think the benchmarks underlying them are bad evidence.

Out of Context

A rule of thumb in ML is that the effective context window is about 5-10 times shorter than the theoretical maximum context window you get sold (also I hope there’s nothing important to you in the middle third).

By “effective” I mean the latent amount of context which gets simultaneously understood, as opposed to the observed size of the data type. (This doesn’t affect “needle in a haystack” retrieval, at which they have been superhuman for a while now.)

Model	Reported max context window (tokens)
Kimi K2 Thinking	256K
MiniMax M2	~200K
Qwen3 235B	32K native 256K (Instruct-2507)
Gemini 3	1M
GPT-5.1	400K (API)
Grok 4.1	256K (standard) 2M (Fast)
Sonnet 4.5	200K (standard) 1M (API)

Why does this matter? Most chats are not hundreds of thousands of tokens long!

Well, 10% of Gemini’s 1m token window is 100K, enough for one big novel input or very roughly 30 serious connected thinking tasks (including input tokens as well); since the AI’s own output tokens count towards what’s in-context, if you want to have a real conversation about a large book you’re still going to have to do it a couple of chapters at a time.

But 10% of 256k (Kimi, Qwen, Minimax, Sonnet) is enough for about a quarter of a big novel or like 8 serious reasoning tasks.

And, again, the Chinese models are token hungry! Not only do they have a smaller bucket, it also gets filled up way faster.

Self-hosting has high fixed costs

3. the weights. On-prem with fully free MIT licence, self-hosting, white-box access, customisation, with zero profit going to the Chinese companies.

Self-hosting doesn’t really make sense unless you’re huge volume or using them for very simple tasks. And most enterprises are not really competent enough to finetune anything.

This is partly a temporary matter: the software ecosystem is underdeveloped for serious high-reliability scaled usage, despite the intense hobbyist interest. (They mostly want it running on a Macbook.)

Too slow for casuals

4. You can get much faster token speeds than the closed APIs.

In the browser, they’re actually slower than Western models. This makes sense; they are incredibly inference bound thanks to chip controls! This would be enough to tank them in the consumer market.

And over API, everyone except Anthropic dominate, even in raw token rate (not counting efficiency):

Censorship and perceived censorship

5. less overrefusal (except on CCP talking points)

There’s a pretty big ick factor to the CCP, and the companies are indeed forced to comply on a range of talking points which offend the West. However, the hosted versions are much worse than the weights themselves. SpeechMap:

But there are uncensored finetunes from reputable names. But then again see (3): it doesn’t make sense for most enterprises to conduct and host finetunes themselves.

If you do a fair test on controversial but non-CCP talking points, there’s a wide spread of refusal rates in both Chinese and Western models.

Nebulous ideology?

6. on topics controversial in the West, less nannying

Lambert notes that the people he speaks privately to are really worried about less obvious stuff, the “indirect influence of Chinese values”.

There is something to this currently (but not a lot given the size of the English internet in the training corpus and the relative lack of soft-post-training skill or effort in Chinese labs):

But it’s valid to assume this will get worse as the CCP get more aware and the companies put more effort into personality and post-training.

Downloading is a long way from productising

8. they’re the most-downloaded open models

People panic about “the flip”, the point at which people started downloading Chinese models more. But this is obviously a terrible proxy for actually managing to use them. (And it’s actually pretty unclear how self-hosting adoption would really benefit China anyway except in prestige.)

For people with any need of real customisation or tiny models, or a scientific ML hobby, or an ideological interest in open-source, they clearly dominate.

TODO: Scrape relative mention over time of LLaMa vs Qwen in Arxiv experiments.

I concede that the secrecy in the West about using Chinese models makes this one weaker as an explanation.

even more insecure?

9. they are mostly not used even by the cognoscenti

Above I mentioned reliability (how low-variance they are, how well they can chain things together). But that’s the easy bit; what about adversarial reliability?

The US evaluation had a bone to pick, but their directional result is probably right (“DeepSeek’s most secure model (R1-0528) responded to 94% of overtly malicious requests [using a jailbreak], compared with 8% of requests for U.S. reference models”).

Someone else talking their book notes that Kimi is “not yet fit for secure enterprise deployment”.

This is obviously a huge problem for any agentic uses, even if the benchmark and default reliability were all fine.

Low mindshare

9. they are mostly not used even by the cognoscenti

It’s hardly cynical to note that most people don’t pick their models by analysing relative performance. Instead it’s largely name recognition and trust, which makes sense for reasons of risk aversion and filtered evidence.

In principle, you can change models by changing one string in your codebase. But in practice if you’re sane you need to do incredibly expensive evals and so there’s stickiness.

The DeepSeek moment helped a lot, but it receded in the second half of 2025 (from 22% of the weird market to 6%). And they all have extremely weak brands.

Also corporations really do settle for inferior products all the time for ass-covering reasons (IBMism). Mindshare translates directly into appeal to the risk-averse.

Corporate compliance is hard

9. they are mostly not used even by the cognoscenti.

Chinese APIs are hard for Western companies to use for legal and quasi-legal reasons.

For the API, DeepSeek sent user information to China Mobile, a state company, which violates all kinds of Western data privacy laws. Even if they’ve stopped, this risk is corporate poison. How can you ever be sure enough?

In a couple of years the EU AI Act will be (nominally) enforceable on the Chinese labs too.

On the quasi-legal side, corporate “vendor risk” programmes often flag Chinese suppliers. This is sometimes because they actually can’t guarantee there’s no forced labour involved.

So why not on-prem? Again, it’s a huge fixed cost and competence-bound and your risk team might still give you shit for it. Lambert:

People vastly underestimate the number of companies that cannot use Qwen and DeepSeek open models because they come from China. This includes on-premise solutions built by people who know the fact that model weights alone cannot reveal anything to their creators.

Political bias

9. they are mostly not used even by the cognoscenti.

There are a bunch of social reasons you might want to avoid Chinese models. You might be protectionist, or sucking up to the ascendent protectionists.

The protectionism of others is clearly enough for people to keep quiet about using them. It is often probably enough for them to just not take the risk in the first place.

I’d include here superstitions about the weights themselves being backdoored.

Vendor risk

9. they are mostly not used even by the cognoscenti

If you look ahead, at future risks to your suppliers, it’s obvious that the export control situation relatively speaks against using Chinese models; NVIDIA is not going to choke off OpenAI.

For API adoption, I also haven’t seen anything about Service-Level Agreements (contracts ensuring uptime) and support from any Chinese lab, but these are easy to make (even if compute crunch means that their uptime guarantees simply must be worse than American ones).

Also again corporate vendor-risk programmes often flag Chinese suppliers for data sovereignty, volatility of PRC law, and export control reasons.

DeepSeek openly use Anna’s Archive, where everyone else is quiet about it. But the American companies offer IP indemnity for users (cover if the models violate copyright in your app), which is nice insurance for a nervous corp with a target on its back. I can’t see anything about the Chinese companies doing this yet.

No compute, no perf

10. they are severely compute-constrained (and as of November 2025 their algorithmic advantage is unclear)

The US has five times the FLOPs as China. (Quality and bandwidth-adjusted it’s probably more like 10x.) On raw hardware, Chinese labs are thus 2-3 clock-time years behind.

What if they’re ten times as efficient though? In January, DeepSeek came out with some exciting and splashy hardware optimisations, probably the fruits from putting HighFlyer’s serious quant devs onto pretraining. But the Westerners responded by getting (even more of) their own quants. I find it unlikely they still have a big algorithmic advantage over the Western labs at this point.

Excess quantization?

11. they’re aggressively quantizing at inference-time, 32 bits to 4

No, I think this one is wrong or else only a tiny factor. gpt-oss was post-trained in MXFP4 which is only 4.25bits.

And I have a strong hunch that many American models are also served in low fidelity, maybe FP4 (4 bits). Quantization just isn’t that bad.

Galaxy-brain soft power??

12. state-sponsored Chinese hackers used closed American models for incredibly sensitive operations, giving the Americans a full whitebox log of the attack!

I can dimly imagine some kind of flexing dynamic in cyberwarfare, where you actually want to show off your attack capabilities, and so you use Claude on purpose. Yes: this idiotic move makes great sense if the apparent targets are red herrings, if Anthropic were the real target. You learn how long their OODA loop is, you learn (by retaliation or its absence) how tight they are with the NSA, you learn a little about how good their tech is.

You could also see it as retaliation for Amodei’s hawkish comments all year. Literally trading effectiveness for embarrassment.

But I don’t really know anything about this.

Overall

Low adoption is overdetermined:

No, I don’t think they’re as good on new inputs or even that close.
No, they’re not more efficient in time or cost (for non-industrial-scale use).
Even if they were, the social and legal problems and biases would probably still suppress them in the medium run.
But obviously if you want to heavily customise a model, or need something tiny, or want to do science, they are totally dominant.
Ongoing compute constraints make me think the capabilities gap and adoption gap will persist.

Interview with Analytics India

INTERVIEWER: You point to a larger performance drop for Chinese models when moving from AIME 2024 to AIME 2025, and you frame that as suggestive of weaker generalisation. If you were to explore that same idea in another domain (like coding benchmarks or natural-language reasoning), what kind of results would meaningfully shift your view?

There's nothing special about AIME, I picked it justbecause it's high-effort and it updates, and so gives us nice properties: 1) novel, 2) of equal difficulty, and 3) the data is "clean".

I could quite easily be persuaded that they generalise better on coding or Q&A than on maths. (The post is "70%" confident.) I don't have time to look myself, but I welcome people superseding me.

Other evidence points the same way though; for instance Qwen2.5 is known to have trained on test for a few benchmarks and collapses on new versions. See also the famous GSM1k case, where flagship Western models actually improved their performance on fresh data.

One problem with the post is that the "tigers" have been improving over the course of this year, and my post doesn't look at the very most recent Chinese models, because they could have trained on AIME 2025. Qwen3 seems less contaminated than Qwen2.5 for instance.

I look forward to repeating this in February on AIME 2026 to see if they've gotten more rigorous.

Your article discusses both model capability and production readiness. If we look at those separately, where do you think Chinese labs are currently closest to Western labs: in raw technical ability, or in reliability/compliance features that matter in enterprise settings?

Closer on capability than product, and closer on product than on legal and quasi-legal compliance.

One thing I didn't cover in the post is non-Western users might find it easier to use the Chinese models than Europeans for a range of reasons (looser compliance regs, less data privacy law, usually less politicisation of the matter).

The popular belief is that Chinese models are only a few months behind Western labs, but you propose a gap of closer to a year. What makes you lean toward that longer estimate, and what would you consider the most unmistakable evidence for readers who are sceptical?

The "three month gap" is calculated by trusting evaluation results naively. I think I have shown that you shouldn't do this. (Note that the Western models also dropped a lot on fresh data!)

This is my subjective guess about an unobserved quantity. I know a little more about the data filtering used in Western labs - and I have shown evidence of recent bad filtering in Chinese labs - and so I make the inference that unseen parts of the model development are also subpar. This is not science, but it's all we have. It's fine to not take my word for it.

There is no unmistakable evidence sadly. We cannot outsource our judgment to benchmark numbers; you really do just have to spend the time to compare the models side by side against your own needs.

Suppose we imagine a world where we could eliminate training-time contamination from public benchmarks. How much would the observed performance gap between top Western and top Chinese models actually change? Would it widen slightly, significantly, or stay roughly the same?

The difficulty is that the models are too smart for this. They can pick up on "semantic duplicates" of the test data (things like "da Vinci was born in 1452" and "the painter of La Gioconda was born in 1452") to cheat despite not seeing the actual data set, and this is profoundly difficult to correct for.

Granting your thought experiment though: I expect it to be wider than my cheap AIME estimate. Mathematics is a relatively clean domain in some ways, and all models struggle even more with messy things, and I expect the Chinese models to be a bit worse still.

Your writing comes across as critical in a measured way, not anti-Chinese imo. Has the social media recactions you’ve seen so far reflected that nuance, or do you feel some readers are approaching the hypothesis through a more polarised lens? What, if anything, has surprised you about the reaction?

Thanks! As I say in the piece ("Filtered evidence") the background discourse is highly toxic and deluded. And I am wary of feeding in to "cope" (being used by people who really don't want Chinese models to be at parity and who allow this desire to determine their beliefs).

I was quite surprised to see powerful people publicly endorsing a mere blogpost. This is good news for the world; blogs remain the alpha of the internet.

The jailbreak argument against LLM values

Sun, 09 Nov 2025 00:00:00 +0000

Bostrom (2014) defined the AI value loading problem as

how could we get some value into an artificial agent, so as to make it pursue that value as its final goal?

JD Pressman thinks this is obviously solved in current LLM systems:

The value loading problem outlined in Bostrom 2014 of {getting a general AI system to internalize and act on “human values” before it is superintelligent and therefore incorrigible} has basically been solved. This achievement also basically always goes unrecognized because people would rather hem and haw about jailbreaks and LLM jank than recognize that we now have a reasonable strategy for getting a good representation of the previously ineffable human value judgment into a machine and having the machine take actions or render judgments according to that representation.

I take issue with this. I agree that LLMs understand our values somewhat, and that present safety-trained systems default to preferring them, behaving like they hold them. 3 5

The jailbreak argument

But here’s why I disagree with him nonetheless: jailbreaks are not a distraction (“hemming and hawing”) but are instead clean evidence that loading is not solved in any real sense:

All LLMs can be “jailbroken”, put into an unaligned mode through mere inference on adversarial text inputs. 4
If a system can be put into an unaligned mode at inference time, then it has not internalised the values.
So models have not internalised the values.
The value loading problem is about getting the values internalised.
Therefore the value loading problem has not been solved in LLMs. 6

I might say instead that “weak value preference” is solved for sub-AGI.

(A deeper analysis would involve the hypothesis that current models don’t actually have goals or values; they simulate personas with values. And prosaic alignment methods just (greatly) increase the propensity to express one persona. Progress has just been made on detecting and shaping such things empirically, so maybe this will change.)

Value-loading vs alignment

Pressman also says that

At the same time people generally subconsciously internalize things well before they’re capable of articulating them, and lots of people have subconsciously internalized that alignment is mostly solved and turned their attention elsewhere.

I initially read this as him agreeing and celebrating this shift, but actually he thinks they’re incorrect to relax, since value loading is only a part of the alignment problem:

solving the Bostrom 2014 value loading problem, that is to say {getting something functionally equivalent to a human perspective inside the machine and using it to constrain a superintelligent planner} is not a solution to AI alignment.

I agree that value-loading is not enough for AGI intent alignment which is not enough for ASI alignment which is not enough to assure good outcomes.

Good discussion on Bsky

Pressman’s response

I sent the above to him and he kindly clarified, walked some of it back, and provided a vision of how to use a decent descriptive model even if it is imperfect and jailbreakable:

I probably should have used the word ‘generalize’ instead of ‘internalize’ there.

(Thus “the value loading problem outlined in Bostrom 2014 of {getting a general AI system to s/internalize/generalize and act on “human values” before it is superintelligent and therefore incorrigible} has basically been solved.”)

The specific point I was making, well aware that jailbreaks in fact exist, was that we now have a thing that could plausibly be used as a descriptive model of human values, where previously we had zilch, it was not even rigorously imaginable in principle how you would solve that problem.

To break this down more carefully:

1. I think that in practice you can basically use a descriptive model of values to prompt a policy into doing things even if neither the policy or the descriptive model have “deeply internalized” the values in the sense that there is no prompt you could give to either that would stray from them. “Internalizing” the values is actually just, kind of a different problem from describing the values. I can describe and make generalizations about the value systems of people very different from me who I do not agree with, and if you put me in a box and wiped my memory all the time you would be able to zero shot prompt me for my generalizations even if I have not “deeply internalized” those values. In general I suspect the LLM prior is closer to a subconscious and there are other parts that go on top which inhibit things like jailbreaks.

If I had to guess it’s probably something like a planner that forms an expectation of what kinds of things should be happening and something along the lines of Circuit Breakers that triggers on unacceptable local outputs or situations. Basically you have a macro and micro sense of something going wrong that makes it hard to steer the agent into a bad headspace and aborts the thoughts when you somehow do.

2. Calling this problem “solved” was probably an overstatement, but it’s one born from extreme frustration that people are making the opposite mistake and pretending like we’ve made minimal progress. Actually impossible problems don’t budge in the way this one has budged, and when people fail to notice an otherwise lethal problem has stopped being impossible they are actively reducing the amount of hope in the world.

At the same time I do kind of have jailbreaks labeled as “presumptively solved” in my head, in the sense that I expect them to be one of those things like “hallucinations” that’s pervasive and widely complained about and then they just become progressively less and less of a problem as it becomes necessary to make them stop being a problem and at some point I wake up and notice that hey wait this is really rare now in production systems. Most potential interventions on jailbreaks aren’t even really being tried because it doesn’t actually seem to be a major priority for labs at the moment if you ask the model for instructions on how to make meth. This makes it difficult to figure out exactly how close to solved it really is. Circuit Breakers was not invincible, on the other hand it’s not clear to me you can “secure” a text prior with a limited context window that doesn’t have its own agenda/expectation of what should be happening to push back against the users with. This paper where they do mechinterp to get a white box interpretation of a prefix attack they find with gradient descent discovers that the prefix attack works because it distracts the neurons which would normally recognize that the request is malicious.

So it’s possible a more jailbreak resistant architecture will need some way to avoid processing every token in the context window. One way to do that might be some kind of hierarchical sequence prediction where higher levels are abstracted and therefore filter the malicious high entropy tokens from the lower levels, which prevents them from e.g. gumming up the planners ability to notice that the current request would deviate from the plan.

And here’s a nice analogy contesting my suitcase word “internalise”:

this word “internalize” is clearly doing a lot of work and something feels Off to me, to say that a text prior which can be manipulated into saying whatever hasn’t “fully internalized” the values. Like if you stripped away the layers on top of my raw predictive models/subconscious that I use for completing patterns and then prompted it, I assume you could get it to say all kinds of nasty things. But also that’s not like, the complete agent.

So if you assumed you can’t build anymore of the agent until you have some way to make the text prior not do that, that’s probably a wrong assumption. One of the reasons I’m annoyed that agents aren’t really a thing yet is that it means we don’t have a good intuitive sense of which parts of the system need to handle what problems.

Post-hoc theory

In retrospect we can see the following distinct problems:

The value specification problem (“how do we describe what we value? how do we get that understanding into the model?”)
We thought this would involve:
a. The explicit value modelling problem (“what precisely do we value?”) - moot
b. The value formalisation problem (“what mathematical theory can capture it?”) - moot, since:

This problem was somewhat solved for sub-AGI by massive imitation learning and (surprisingly nonmassive) human preference post-training. The internet was the spec. This also gave us weak value preference.

The replacement worry is about how high quality and robust this understanding is:
The value generalisation problem (“how do we go from training data about value to the latent value?”) - some progress. The landmark emergent misalignment study in fact shows that models are capable of correctly generalising over at least some of human value, even if in that case they also reversed the direction. 7

Then there’s the gap between usually preferring something and “internalising” it (very reliably preferring it):
The sub-AGI value-loading problem (“how do we make them actually care / reliably use their understanding of our values?”) - not solved, but there is a preference towards niceness.
Tamper-resistant value-loading (“how do we stop a small number of weight updates from ruining the value-loading?”) - not solved, maybe a bit unfair to expect it. You could imagine doing advanced persona steering on top instead.
The general value-loading problem (“how do we get an ASI to learn and internalise a model of current human values which is better than the human one”) - not solved
The value extrapolation problem (“how do we safely improve on current human values?”) - not solved

I’m sure there’s some other deep confusion lurking in this list and I look forward to seeing it exposed, ideally in less than 10 years.

Ways we can fail to answer

Sun, 02 Nov 2025 00:00:00 +0000

In what ways can we can fail to answer a question?

(I mean necessarily fail: actual barriers to knowledge, rather than skill issue hurdles. But of course contingent failures are much more common: “We didn’t ask the question in the first place”, or “We didn’t have the particular insight that would have allowed for productive research”, or “We didn’t manage to remove every cognitive bias”, or “Instrumentation is really hard”, or “We are not rich enough to run this study yet”, or “We worshipped the problem”.)

(I also mean fail exactly; there are often excellent approximations, and we can often legitimately patch over tricky philosophical questions with our unanalysed tacit knowledge.)

Conceptual problems (the question is not a question)

A malformed question or category error: the question may ask for something which doesn’t make sense (e.g. “what colour is justice?”, or “Is sigmoid jealous of ReLU?”, or “Did these things happen at the same absolute time? What are the absolute coordinates of this event?”.)
- Strong incommensurability: the question doesn’t make sense because it mixes frameworks. (e.g. “What is the Einsteinian mass of phlogiston?”, or “What’s the wavefunction of this classical field?”)
Vagueness: the question may fail to mean anything in particular. (e.g. “When exactly did you become an adult?”) 5
False assumptions: we can’t answer it because, while it was clear and made sense, it was wrong from the start (e.g. “Have you stopped beating your wife?” or “Is personality based on nature or nurture?”. 1)

Logical problems (the question has no answer)

Antinomy: it turns out that there is no answer because the question is circular or involves itself somehow (e.g. “Is this sentence false?”)
Incompleteness (“Show that Peano arithmetic is consistent using Peano arithmetic”) 6 and undefinability (e.g. “Is this sentence true in arithmetic?”) are about some proof answers being inaccessible inside systems powerful enough to be interesting and useful. It doesn't come up in normal thought very often.
Mathematical independence: The question is neither provable nor disprovable with these axioms. (e.g. Is there a well-ordering of the reals? Is the continuum hypothesis true in ZFC? What is the value of BB(748) in ZFC?)
Various holes in social mathematics. Important questions like "what's the optimal voting system?", "what does the majority prefer?", "what's a democratic system where people don't have an incentive to vote strategically?", "what's the perfect design for a market?" in general don't have an answer. See e.g. here. 8
Anthropic self-locating effects: you can’t get at the answer because you couldn’t exist to observe it. (e.g. “What’s the probability I’m a Boltzmann brain?”, or “What’s the prior probability of observer-permitting universes?”)

Non-uniqueness is only a problem for questions which ask for the one true answer. (e.g. “What is the electromagnetic potential at this point?”). Many entries in this post are not strict failures to produce answers, they just have a non-unique answer. You can sometimes just parametrise the observer and then get your answer. If non-uniqueness is a failure, it’s a happy one; it just means that you get too many answers and have the nicer problem of picking one.

Ontic problems (the answer is literally inaccessible)

Uncomputability: we can’t answer it because no computer or mind can. (e.g. “Is this random?”, or “Does this Diophantine equation have integer solutions?” or “What’s the shortest Python program that outputs this file?” or “Can you write a program that checks if conjectures follow from these axioms?” Or you asked about the language the problem is in.) 4
Computational intractability: we can’t answer it because it would take too long, including if we turned the universe into a computer. (What’s the best way to schedule these classes, avoiding all conflicts and respecting room capacities and lecturer availability? Or “I forgot my password; crack this file”.)
Inapproximability: we can’t even approximate the answer because it would still take too long. (e.g. “What’s the maximum clique size in this graph?”, “What’s a (log n)-approximation to the chromatic number?”)
Quantum indeterminacy: there’s no answer because (maybe) physics is intrinsically random. (e.g. “When will this radium atom decay?”) 3
Physical constraints, speed limits and no-gos: it can’t be answered in principle because of physics. (e.g. the answer is beyond the cosmological or particle horizon, “what will happen in this galaxy 20 Gly away?” or “Can we measure this state in two bases?”, or “Make a perfectly accurate interferometer”, or “What are the microstates of this black hole?.)
Mixing: the answer is gone; we arrived too late; the system forgot the answer. (e.g. “What was the exact microstate of the gas in this room 1 hour ago?”, “What was the original state of this thermalised system?”)
Non-ergodicity: the answer isn’t reachable because the system doesn’t mix. (e.g. “What equilibrium will this glassy system reach?”)
Measurement problems
- Disturbance or decoherence: we can’t answer it because our instruments disturb the thing in question too much.
- Complementarity: we can’t answer it because the answer was excluded by another question we asked first. (e.g. “where is this electron and how much momentum does it have?”)

Epistemic (we cannot get at the answer)

underdetermination or unidentifiability. (e.g. Is spacetime fundamentally Lorentzian? Which interpretation of quantum mechanics is true? Why do physical constants look fine-tuned? What was the ancestral DNA sequence at some past generation?)
Chaos: we can’t answer it because we can’t measure the initial conditions well enough to predict large systems. (e.g. Where exactly will this double pendulum be in 100 Lyapunov times? What’s the weather like 3 months out?)
Computational irreducibility: getting the answer is inseparable from running the system, the question has no shorter answer than a full simulation (e.g. “What will this cellular automaton look like in 10^100 steps?”)
Hidden variables: we can’t ever observe the actual variables (e.g. What are the simultaneous values of σ_x and σ_z for this electron?)
Cognitive closure: we’re not smart enough to answer it / we lack certain faculties (like echolocation say). (e.g. “What is it like to be a bat?” It kinda looks like quantum gravity could also be an example.)
Cognitive bias: we’re not rational enough to answer it (e.g. “How biased am I?”)
I suppose I should mention the original “epistemic barrier”, the putative conceptual barrier between the mind and the world.

??? (the question is logical and/or ontic and/or epistemic idk)

Observer effects
- Back action (e.g. What’s the pre-measurement spin of this electron? What are the “real” observables in this quantum system? or “Measure the full-speed time evolution for this system”.)
- Theory-ladenness:
  - Semantic theory-dependence: Our observational vocabulary already presupposes theoretical commitments that prejudge the answer (e.g. What’s the rest mass of an electron, without assuming special relativity? What’s the ‘real’ temperature of the CMB independent of blackbody theory?)
  - Perceptual theory-dependence: Our perception is shaped by theoretical expectations, so we cannot see the answer “directly” (e.g. What do electron tracks really look like? What does this fMRI show before we apply the hemodynamic response model?)
Future contingents: it doesn’t have an answer yet so we have to wait. (e.g. “What lottery numbers will win next week?”)2
Hysteresis: we can’t answer it because we weren’t there at the start and the system remembers. (e.g. What’s the magnetic moment of this material at field strength H? When will this old rope snap? At what temperature will this water freeze?)
Reflexivity and strange loops: the thing in question is self-referential or causally dense, or it [changes](https://en.wikipedia.org/wiki/Demand_characteristics) when answered, and so can’t be picked apart. (e.g. “What’s the best method for finding the best method?”; “Which level of description is fundamental in this self-referential system?”, “Which part of your mind is the real you?”.) See also antinomy.

Finally there is the great risk this post takes (and all not-totally-technical writing):

Model error leading to spurious impossibility: you apply an analytic impossibility result (like the above) to a "synthetic" real system where it doesn't apply. You overinterpret a narrow thing; you assume the world fits the assumptions of the formal proof but it doesn't; you summarise a technical result in natural language and imply that it's more general than it is. Most invocations of Gödel are spurious;

You will have done really well if even once in your life you fail to answer a question for these reasons. Getting so far means you have avoided hundreds of punji traps, claymores, nerve gasses, madnesses.

What god has to do in mathematics

Tue, 28 Oct 2025 00:00:00 +0000

He picks from uncountably many sets simultaneously without an algorithm.

He preserves us from sets of measure zero and the non-measurables. In his loving arms infinity minus infinity equals whatever we need it to. He takes five loaves of spheres and two small spheres and feeds the host.

He makes the inaccessible cardinals. He stops the inaccessible cardinals from accessing us

He is risen, he is lifted, and he is Born.

He transubstantiates our mortal filters into ultrafilters for free.

He knows the truth of the continuum hypothesis but saves us from it.

He lets us pretend we can do functional analysis, keeping our intersections dense by tucking away the invisible point we could never reach.

Quietly he makes actual of potential infinity.

Silently he transfers the evaluation strategy from heaven (the meta-language) to earth (the source language).

He forgives us our renormalizations.

When we are weak and know only subspace, he carries our functional to the full space.

Ultimately he makes it all cohere. The accidents of our conditionals are treatable as unconditional. “I am that I am”, what makes the axioms hold.

Various atheists try to do without him and do not prevail.