In 2011, with Ireland’s economy in the depths of recession and a general election imminent, several prominent journalists and commentators seemed to be about to launch bids for election to the national parliament as a new party called Democracy Now. They had, the argument went, been pointing out precisely what politicians had been doing wrong for many years and now was the time for them to cross the line and actually try it out for themselves. In the end, they decided against it and the politicians continue to get it wrong while they continue to point. There’s a similar line between statistical analysis that leads to policy recommendations and the implementation of those policies, and a decision to make about how close to it you want to get.
The census of everything
A line is drawn around the potential impact of any analysis at the design stage and with considerations about return on the research investment. To take an extreme example, a population census is a perfect opportunity to get nationally representative data on every topic you can imagine: standardised psychological assessment tools, income, education, and attitudes to advertising or to zoo animals. Of course, that would make a lot of statisticians very happy but would make for a census form the size of the Encyclopaedia Britannica that would take householders weeks to complete. Some collaboration with others is necessary to decide the bounds of the survey and which questions will be most useful.
Because researchers spend so much time on one topic, so much energy convincing themselves and then everyone else that what they’re doing is the most important thing in the world, there’s a risk that they’ll prioritise their research area over any potential impact. Even when they’re making a case for the relevance of their work, look closely and it might just be that they really want to keep working in the area. There’s also something cosy and familiar about staying inside the lines so they might be reluctant to look in detail at how to implement what they recommend.
There are two options: put your perfect research out there and then scoff at policy-makers’ stupidity for not doing what is so obviously right; or work with them, learn their language, and listen to their ideas from research question to final report. What you’re trying to avoid is fighting with the policy-makers, because if you’re fighting with them they’re not going to implement your world-changing ideas. It would be naïve to think that policy uses the best available evidence based on sound statistical analysis so part of the role of a statistician is to explain what the numbers actually mean, but there is also likely to be some tension.
Policies, you see, tend to be about tackling something, so if your analysis is in any way supportive of the status quo, don’t expect to get too far with the more righteous of policifiers. This is one of the issues underlying publication bias, that is, the tendency to report analysis that shows significant differences and suppress, wittingly or unwittingly, non-significant differences. People who dare to present conference papers on interventions that showed no significant impact have been described as “brave”. Sometimes it’s necessary to stick to your numbers and say “We can’t say for sure” or even “It doesn’t work”.
From the point of view of getting things done, though, knowing what doesn’t work is as important and knowing what does. Implementation science is about translating research into action, about identifying mechanisms for delivery. As noted in a recent post, vaccination is a fairly good idea and the reward of saving thousands of children’s lives out-weighs the small risk of harm to a small number. Reaching that conclusion is relatively easy for statisticians but the implementation of the policy is usually left up to someone else, and it turns out it can be quite hard. In some developing countries vaccination rates are lower than in developed countries, though the reasons for the low vaccination rates are beyond the scope of a blog on statistics. One issue is that the evidence suggesting that investment in immunisation is likely to provide a public health return is not backed up by evidence of how to achieve higher rates, because researchers have yet to ask and answer those types of questions.
There’s a gap between statistical analysis and implementation that parallels the relationship between the commentariat and the politicians. It’s a strait you might prefer to shout across, or you might want to make the leap and get into the details of how your analysis can make a difference. In the end, it’s about balancing the objectivity and independence of your work against the probability of something actually changing as a result. Just because Democracy Now never materialised doesn’t mean we don’t have any democracy now.
Power heuristic – 8th August 2014
Independence – 14th November 2014
Scientific literacy – 9th February 2015
Institute of Statistical Science Academia Sinica
December 27, 2019
London School of Hygiene and Tropical Medicine
November 19, 2019
December 19, 2019
University of Alberta
Edmonton, AB, Canada
December 18, 2019
November 22, 2019
UCLA Department of Statistics
Los Angeles, CA
December 10, 2019
The National Audit Office
December 01, 2019
The Institute of Cancer Research
Sutton, Surrey, UK
December 02, 2019
University of Glasgow
December 09, 2019
City of Westminster Council
Victoria Street, London, UK
December 31, 2019
Greater London Authority
November 24, 2019
The Department for Work and Pensions
Leeds, West Yorkshire
December 14, 2019
Universidad Carlos III de Madrid
December 01, 2019
Ministry of Justice
December 11, 2019
Competition & Markets Authority
Canary Wharf, London
November 18, 2019
University of the Incarnate Word
San Antonio, TX
November 20, 2019