Research

COVID-19 policy studies mostly don't do proper validation - very few papers check their performance on holdout data, and the sensitivity checks they perform are usually really limited.

We re-ran one of the famous models, and several variations of our own, and found that the famous model's results depend quite a lot on analysis decisions (ours is a bit more robust).

Also a couple theorems about how to interpret the effects: it's not the unconditional effect of doing policy p, it's the average additional effect of p, if you implement it alongside average existing policies (the average in your dataset).

Authors: Mrinank Sharma, Sören Mindermann, Jan M. Brauner, Gavin Leech, Anna B. Stephenson, Tomáš Gavenciak, Jan Kulveit, Yee Whye Teh, Leonid Chindelevitch, Yarin Gal.

We look at an obscure kind of machine learning, seeing if it is (or could be) safer than neural networks. We use an existing framework for thinking about AI safety, and formalise it a bit to allow the comparison.

Upsides: ILP is convenient for specification, is robust to some syntactic input changes, has greater control over the inductive bias, actually can be formally verified, and the results are pretty interpretable (you can read the model and see how it is built).

But ILP is (so far) limited to domains where you have nice neat symbolic data, it can't do architecture search, and its performance lags far behind NNs on almost all tasks. Hybrid systems of ILP and NNs look like they would lose most of what we like about ILP in the first place.

Authors: Gavin Leech, Nandi Schoots, Joar Skalse

We looked at how policy effects changed in the second wave (late 2020). This time we did it at a finer level, with the unit roughly 1/20 of a country rather than whole countries.

Policies got a bit weaker overall (66% combined reduction, compared to 80% in Spring). The best reading of this is that all the safety measures and individual protective behaviour persisted after the first wave, even when the government said it was ok to stop.

School closure was notably weaker (10% instead of 35%). This probably means that the safety measures enforced since Spring really did make them safer.

Authors: Mrinank Sharma, Sören Mindermann, Charlie Rogers-Smith, Gavin Leech, Benedict Snodin, Janvi Ahuja, Jonas B. Sandbrink, Joshua Teperowski Monrad, George Altman, Gurpreet Dhaliwal, Lukas Finnveden, Alexander John Norman, Sebastian B. Oehm, Julia Fabienne Sandkühler, Thomas Mellan, Jan Kulveit, Leonid Chindelevitch, Seth Flaxman, Yarin Gal, Swapnil Mishra, Jan Markus Brauner, Samir Bhatt

By looking at tests and sewage data from early 2021, we saw that "the English variant" of COVID (B.1.1.7) which took over England in December was itself being displaced by other nasty variants. The main worry was that one of the new variants would be resistant to the vaccines.

Authors: Swapnil Mishra, Sören Mindermann, Mrinank Sharma, Charles Whittaker, Thomas AMellan, Thomas Wilton, Dimitra Klapsa, Ryan Mate, Martin Fritzsche, Maria Zambon, JanviAhuja, Adam Howes, Xenia Miscouridou, Guy P Nason, Oliver Ratmann, Gavin Leech, Julia Fabienne Sandkühler, Charlie Rogers-Smith, Michaela Vollmer, H Juliette T Unwin, Yarin Gal, Meera Chand, Axel Gandy, Javier Martin, Erik Volz, Neil M Ferguson, Samir Bhatt, Jan M Brauner, Seth Flaxman

We used a hierarchical Bayesian model to see what worked in the first wave of the pandemic. Up to then, people hadn't been able to pick apart the individual effects of anti-COVID policies, instead using "lockdown" to name all the 20 different things that governments tried in spring 2020 (where that should really be the word for stay-at-home-orders).

We collected a big new dataset, 41 countries. We were one of the first to spot the really large effect of closing schools & unis, back when people were hoping that children were magically not infectious. Our validation was unusually big and rigorous, for epidemiology.

Stay-at-home-orders did surprisingly little (0 to 25% reduction) if you've already closed schools, restaurants, big events.

We initially did a cost-benefit analysis of each policy, by surveying (American) people on how much each interferes with their lives, but this wasn't done well enough to get into the final paper. To my knowledge this still hasn't been done (outside of secret government documents), despite it being impossible to make good decisions without it.

Authors: Jan M. Brauner, Sören Mindermann, Mrinank Sharma, David Johnston, John Salvatier, Tomáš Gavenčiak, Anna B. Stephenson, Gavin Leech, George Altman, Vladimir Mikulik, Alexander John Norman, Joshua Teperowski Monrad,, Tamay Besiroglu, Hong Ge, Meghan A. Hartwick, Yee Whye Teh, Leonid Chindelevitch, Yarin Gal, Jan Kulveit.

There was a puzzle: micro level studies (i.e. watching individual people) tended to find nice big reductions in COVID transmission from masks, like 50%. But society-level studies found random results in [-2%, 40%]. Turns out that the proxy people were using in the latter was pretty weak. So we get a much much better proxy, using Facebook's reach to get 20 million data points on where and when people were actually wearing masks.

We did a complicated regression model on 56 countries (not including treating the US states as countries), and checked it in 22 ways to make sure that our result wasn't just cherrypicking or a pure correlation. We find that masks can be confidently linked to a 6% - 43% reduction in transmission, where we can't really what the effect of mandates was. (For comparison, the difference between summer and winter is 42%, or the effect of all government interventions in the first-wave was about 80%.)

Authors: Gavin Leech, Charlie Rogers-Smith, Jonas B Sandbrink, Ben Snodin, Rob Zinkov, Benjamin Rader, John S. Brownstein, Yarin Gal, Samir Bhatt, Mrinank Sharma, Soren Mindermann, Jan Markus Brauner, Laurence Aitchison

We reconstruct the ridiculously complicated causal web involved in making COVID less bad in the summer and then ignore it and estimate one scalar instead. It turns out to be big but not big enough, only about 40% reduced transmission in summer.

This provides a really important adjuster for observational studies, and updates unadjusted estimates from last year.

Authors: Tomas Gavenciak, Joshua Teperowski Monrad, Gavin Leech, Mrinank Sharma, Soren Mindermann, Jan Markus Brauner, Samir Bhatt, Jan Kulveit

We try to work around the impossibility results in algorithmic fairness, by using legal damages as a signal about all-things-considered unfairness. This lets us use multiple definitions of fairness at once and set the weight on each in a non-arbitrary way.

A human picks a set of fairness definitions; A human gives the algorithm a set of past cases, along with the damages awarded in each case; LGFO works out how much weight to give each kind of fairness, and so produces a classifier which is relatively fair, if we trust the legal system to know this relatively well.

Authors: Dylan Holden-Sim, Gavin Leech, Laurence Aitchison