Research

"Will we make the big one, AGI? If so, how? Having made it, will we regret it? And what then?"

Authors: Gavin Leech and Dwarkesh Patel (and 20 interviewees).

Total hours I contributed: 340.

I edited a volume of the best essays by the Fabric community. A distinct lineage of rationality I love.

Authors: Various. Thanks also to Olga for the great cover designs, Phil for the LaTeX, and Xavi for the maths.

I have a PhD in complaining.

I went in wanting to work on AI safety. True to form, I instead ended up with a grab-bag: approximate Bayesian inference, Covid epidemiology, metascience, the methodology of the social sciences, scientific malpractice, inductive logic, algorithmic fairness, the methodology of evaluating machine learning systems, and (of course) large language models. Some safety work in there. Broadly, post-hoc, it’s about epistemics. (Why can’t we learn that much from one study, or many studies?)

In Newell’s typology the thesis “contradicts existing knowledge; thoroughly explores an area; provides empirical data; and produces a negative result”.

It is a Gelmanian work, in that I cite him 74 times and in every chapter, and in that it is pragmatically Bayesian, in that it is self-elected police work, in that it skips merrily across disciplinary boundaries, and in that it’s stubborn and irreverent. This makes it Jaynesian.

It’s surprisingly easy to do real work in fields you have zero background in. This is partly because stats now reigns imperial in roughly all sciences but also because many fields are in some sense methodologically shallow and content deep.

All solutions to bad science have their own vulnerabilities and/or high costs. This war will never end.

It’s pretty fishy how few retractions there are in machine learning. (In every field, but especially machine learning.)

We don’t win on average (the average paper is very bad and will always be fairly bad) but as long as our filtering mechanism allows us to continue to find the 1% which actually progress us...

Total hours I contributed: 3500 on the whole deal (226 on the writeup).

1. Benchmark values are going up by X. This leads people to think that capabilities, latent performance on OOD, is going up X.
2. But this might be confounded, since the training corpus size is also going up by a factor of 50,000x, and since corpuses contain test sets. The same argument undermines many “private” evals with a short description length.
3. (Some) Labs make good-faith efforts to remove test sets.
4. But semantic duplicates of test data are extremely hard to detect and therefore to remove.
5. We show that semantic duplicates of the test data are in the (OLMo) corpus. This is an underestimate of the prevalence in closed corpuses, since one of the major synthetic data generation tasks is to rewrite training data.
6. We show that semantic duplicates of test data spuriously boost performance on the test data
7. We want to estimate the hidden interpolation share of apparent AI progress. It might be like a third.

Authors: Ari Spiesberger, Juan J Vazquez, Nicky Pochinkov, Tomáš Gavenciak, Peli Grietzer, Gavin Leech^†, Nandi Schoots^†

Total hours I contributed: 100

Full title: Tracking and mainstreaming replications in the social, cognitive, and behavioral sciences

Argues that replications are essential for validating scientific claims, but they're hard to find and don't make it into citations or courses. So we manually put together the largest database of replications in any field, 5000 experiments.


Authors: Helena Hartmann, Flavio Azevedo, Lukas Röseler, ..., Gavin Leech.

Total hours I contributed: 150.

Full title: Questionable Practices in Machine Learning

Evaluating modern ML models is hard. The strong incentive for researchers and companies to report a state-of-the-art result on some metric often leads to questionable research practices (QRPs): bad practices which fall short of outright research fraud. We describe 43 such practices which can undermine reported results, giving examples where possible. Our list emphasises the evaluation of large language models (LLMs) on public benchmarks. We also discuss "irreproducible research practices", i.e. decisions that make it difficult or impossible for other researchers to reproduce, build on or audit previous research.


Authors: Gavin Leech, Juan Vazquez, Misha Yagudin, Nic Kupper, Laurence Aitchison.

Total hours I contributed: 250.

Full title: Ten Hard Problems in Artificial Intelligence We Must Get Right

We explore the AI2050 "hard problems" that block the promise of AI and cause AI risks: (1) developing general capabilities of the systems; (2) assuring the performance of AI systems and their training processes; (3) aligning system goals with human goals; (4) enabling great applications of AI in real life; (5) addressing economic disruptions; (6) ensuring the participation of all; (7) at the same time ensuring socially responsible deployment; (8) addressing any geopolitical disruptions that AI causes; (9) promoting sound governance of the technology; and (10) managing the philosophical disruptions for humans living in the age of AI. For each problem, we outline the area, identify significant recent work, and suggest ways forward. [Note: this paper reviews literature through January 2023.]


Authors: Gavin Leech, Simson Garfinkel, Misha Yagudin, Alexander Briand, Aleksandr Zhuravlev.

Total hours I contributed: 800.

Full title: Activation Addition: Steering Language Models Without Optimization

We investigate activation engineering: modifying the activations of a language model at inference time to predictably alter its behavior. It works by adding a bias to the forward pass, a 'steering vector' implicitly specified through normal prompts. Activation Addition (ActAdd) computes these by taking the activation differences of pairs of prompts. We get control over high-level properties of the output without damaging the model's performance. ActAdd takes far less compute and implementation effort compared to finetuning or RLHF, allows nontechnical users to provide natural language specifications, and it scales really naturally with model size.

This is the first alignment method which doesn't need training data or gradient descent.


Authors: Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, Monte MacDiarmid.

Total hours I contributed: 190.

In short order we answered three questions for Atlas, a fellowship for ambitious young people: How do you find extremely gifted people? What other measures could work? What do people do in practice in elite firms and in talent programmes? See also the other two papers in the project.

Authors: Gavin Leech, Misha Yagudin, and Vasco Grilo

Total hours I contributed: 80 on all 3 papers.

My friend Hugh was convinced that linear models should be able to match decision tree ensembles in many places, and we had both seen stuff like depth-12 trees used in industry, which just don't make sense on a naive account.

We contrived a few neat stochastic processes to test a few ways that the depth of the trees could be helping with various realistic data flaws. Turns out that depth partially fixes a bunch of modelling and training errors: if you don't run training long enough, if you use the wrong link function, if you assume single responses when the response is actually composite, if you assume homogeneity in a heterogeneous population, if there are missing variables. Also a couple of simple extensions to Gaussian mixture modelling with lines.

Our linear fixes didn't work that well on randomly selected real data though.

Total hours I contributed: 50.

We try to work around the famous impossibility results in algorithmic fairness, by using legal damages as a signal about all-things-considered unfairness. This lets us use multiple definitions of fairness at once and set the weight on each in a non-arbitrary way.

A human picks a set of fairness definitions; A human gives the algorithm a set of past cases, along with the damages awarded in each case; LGFO works out how much weight to give each kind of fairness, and so produces a classifier which is relatively fair, if we trust the legal system to know this relatively well.

Authors: Dylan Holden-Sim, Gavin Leech, Laurence Aitchison

Total hours I contributed: 20.

Authors: Jan Kulveit, Tomáš Gavenciak, Gavin Leech†, Raymond Douglas

Total hours I contributed: 5

Reweighted wake-sleep (RWS) can do Bayesian inference in a very general class of models. It samples from an approximate posterior Q, then uses importance weighting to estimate the true posterior P, then updates Q towards the importance-weighted estimate of P. But the sheer number of samples you need in IW rules out any realistic-size model.

We develop "massively parallel RWS", which samples all latent variables and reasons about all possible combinations of samples. You can do this in polynomial time by exploiting conditional independencies. We get a dramatic speedup over standard RWS, which samples the full joint.

Authors: Thomas Heap, Gavin Leech, Laurence Aitchison.

Total hours I contributed: 300 (mostly on an abandoned branch).

Full title: How Robust are the Estimated Effects of Nonpharmaceutical Interventions against COVID-19?

COVID-19 policy studies mostly don't do proper validation - very few papers check their performance on holdout data, and the sensitivity checks they perform are usually really limited.

We re-ran one of the famous models, and several variations of our own, and found that the famous model's results depend quite a lot on analysis decisions (ours is a bit more robust).

Also a couple theorems about how to interpret the effects: it's not the unconditional effect of doing policy p, it's the average additional effect of p, if you implement it alongside average existing policies (the average in your dataset).

Authors: Mrinank Sharma, Sören Mindermann, Jan M. Brauner, Gavin Leech, Anna B. Stephenson, Tomáš Gavenciak, Jan Kulveit, Yee Whye Teh, Leonid Chindelevitch, Yarin Gal.

Total hours I contributed: 120.

Research results are often not reproducible and/or replicable. Intense self-criticism in psychology over the last 7 years or so has showed that only 40-65% of classic results were replicated. Call these "reversals". Shockingly, they are not incorporated into research training or undergrad education.

Authors: Helena Hartmann, Shilaan Alzahawi, Meng Liu, Mahmoud Elsherif, Alaa AlDoh, Gavin Leech, Flavio Azevedo

Total hours I contributed: about 200 to the underlying dataset.

We look at an obscure kind of machine learning, seeing if it is (or could be) safer than neural networks. We use an existing framework for thinking about AI safety, and formalise it a bit to allow the comparison.

Upsides: ILP is convenient for specification, is robust to some syntactic input changes, has greater control over the inductive bias, actually can be formally verified, and the results are pretty interpretable (you can read the model and see how it is built).

But ILP is (so far) limited to domains where you have nice neat symbolic data, it can't do architecture search, and its performance lags far behind NNs on almost all tasks. Hybrid systems of ILP and NNs look like they would lose most of what we like about ILP in the first place.

Authors: Gavin Leech, Nandi Schoots, Joar Skalse

Total hours I contributed: 100 (plus a crapton learning Metagol and learnability stuff).

Full title: The Replication Database: Documenting the Replicability of Psychological Science

Does what it says on the tin. My replications dataset formed one of several input datasets that ReD generalises.

we present the Replication Database, hosting 1,239 original findings paired with replication findings. The infrastructure of this database allows researchers to submit, access, and engage with replication findings. The database makes replications visible, easily findable via a graphical user interface, and tracks replication rates across various factors, such as publication year or journal.

Authors: Lukas Röseler (51 others), ..., Gavin Leech, ..., (44 others).

Total hours I contributed: 10 marginal hours.

Full title: Mask wearing in community settings reduces SARS-CoV-2 transmission

A puzzle: measuring individual people tended to find nice big reductions in COVID transmission from masks, like 50%. But society-level studies found random results in [-2%, 40%]. Turns out that the proxy people were using was pretty weak. So we got a much much better proxy, using Facebook's reach to get 20 million data points on where and when people were actually wearing masks.

We did a complicated regression model on 56 countries (not including treating the US states as countries), and checked it in 22 ways to make sure that our result wasn't just cherrypicking or a pure correlation. We find that masks can be confidently linked to a 6% - 43% reduction in transmission, where we can't really say what the effect of mandates was. (For comparison, the difference between summer and winter is 42%, or the effect of all government interventions in the first-wave was about 80%.)

Authors: Gavin Leech, Charlie Rogers-Smith, Jonas B Sandbrink, Ben Snodin, Rob Zinkov, Benjamin Rader, John S. Brownstein, Yarin Gal, Samir Bhatt, Mrinank Sharma, Soren Mindermann, Jan Markus Brauner, Laurence Aitchison

Total hours I contributed: 565.
16th most-discussed paper in the history of PNAS.

Full title: Seasonal variation in SARS-CoV-2 transmission in temperate climates: a Bayesian modelling study in 143 European regions

We reconstruct the ridiculously complicated causal web involved in making COVID less bad in the summer and then ignore it and estimate one scalar instead. It turns out to be big but not big enough, only about 40% reduced transmission in summer.

This provides a really important adjuster for observational studies, and updates unadjusted estimates from last year.

Authors: Tomas Gavenciak, Joshua Teperowski Monrad, Gavin Leech, Mrinank Sharma, Soren Mindermann, Jan Markus Brauner, Samir Bhatt, Jan Kulveit
Total hours I contributed: 110.

Full title: A dataset of non-pharmaceutical interventions on SARS-CoV-2 in Europe

The dataset from the second wave paper.

Authors: George Altman, Janvi Ahuja, Joshua Monrad, Gurpreet Dhaliwal, Charlie Rogers-Smith, Gavin Leech, Benedict Snodin, Jonas B. Sandbrink, Lukas Finnveden, Alexander John Norman, Sebastian B. Oehm, Julia Fabienne Sandkühler, Jan Kulveit, Seth Flaxman, Yarin Gal, Swapnil Mishra, Samir Bhatt, Mrinank Sharma, Sören Mindermann & Jan Markus Brauner

Full title: Understanding the effectiveness of government interventions against the resurgence of COVID-19 in Europe

We looked at how policy effects changed in the second wave (late 2020). This time we did it at a finer level, with the unit roughly 1/20 of a country rather than whole countries.

Policies got a bit weaker overall (66% combined reduction, compared to 80% in Spring). The best reading of this is that all the safety measures and individual protective behaviour persisted after the first wave, even when the government said it was ok to stop.

School closure was notably weaker (10% instead of 35%). This probably means that the safety measures enforced since Spring really did make them safer.

Authors: Mrinank Sharma, Sören Mindermann, Charlie Rogers-Smith, Gavin Leech, Benedict Snodin, Janvi Ahuja, Jonas B. Sandbrink, Joshua Teperowski Monrad, George Altman, Gurpreet Dhaliwal, Lukas Finnveden, Alexander John Norman, Sebastian B. Oehm, Julia Fabienne Sandkühler, Thomas Mellan, Jan Kulveit, Leonid Chindelevitch, Seth Flaxman, Yarin Gal, Swapnil Mishra, Jan Markus Brauner, Samir Bhatt
Total hours I contributed: 100.

Full title: Changing composition of SARS-CoV-2 lineages and rise of Delta variant in England

By looking at tests and sewage data from early 2021, we saw that "the English variant" of COVID (B.1.1.7) which took over England in December was itself being displaced by other nasty variants. The main worry was that one of the new variants would be resistant to the vaccines.

Authors: Swapnil Mishra, Sören Mindermann, Mrinank Sharma, Charles Whittaker, Thomas Mellan, Thomas Wilton, Dimitra Klapsa, Ryan Mate, Martin Fritzsche, Maria Zambon, Janvi Ahuja, Adam Howes, Xenia Miscouridou, Guy P Nason, Oliver Ratmann, Gavin Leech, Julia Fabienne Sandkühler, Charlie Rogers-Smith, Michaela Vollmer, Juliette T Unwin, Yarin Gal, Meera Chand, Axel Gandy, Javier Martin, Erik Volz, Neil M Ferguson, Samir Bhatt, Jan M Brauner, Seth Flaxman
Total hours I contributed: 10.

Full title: Inferring the effectiveness of government interventions against COVID-19

We used a hierarchical Bayesian model to see what worked in the first wave of the pandemic. Up to then, people hadn't been able to pick apart the individual effects of anti-COVID policies, instead using "lockdown" to name all the 20 different things that governments tried in spring 2020 (where that should really be the word for stay-at-home-orders).

We collected a big new dataset, 41 countries. We were one of the first to spot the really large effect of closing schools & unis, back when people were hoping that children were magically not infectious. Our validation was unusually big and rigorous, for epidemiology.

Stay-at-home-orders did surprisingly little (0 to 25% reduction) if you've already closed schools, restaurants, big events.

We initially did a cost-benefit analysis of each policy, by surveying (American) people on how much each interferes with their lives, but this wasn't done well enough to get into the final paper. To my knowledge this still hasn't been done (outside of secret government documents), despite it being impossible to make good decisions without it.


Authors: Jan M. Brauner, Sören Mindermann, Mrinank Sharma, David Johnston, John Salvatier, Tomáš Gavenčiak, Anna B. Stephenson, Gavin Leech, George Altman, Vladimir Mikulik, Alexander John Norman, Joshua Teperowski Monrad,, Tamay Besiroglu, Hong Ge, Meghan A. Hartwick, Yee Whye Teh, Leonid Chindelevitch, Yarin Gal, Jan Kulveit.

Total hours I contributed: 130.