Walking the line: Stats and policy development

In 2011, with Ireland’s economy in the depths of recession and a general election imminent, several prominent journalists and commentators seemed to be about to launch bids for election to the national parliament as a new party called Democracy Now. They had, the argument went, been pointing out precisely what politicians had been doing wrong for many years and now was the time for them to cross the line and actually try it out for themselves. In the end, they decided against it and the politicians continue to get it wrong while they continue to point. There’s a similar line between statistical analysis that leads to policy recommendations and the implementation of those policies, and a decision to make about how close to it you want to get.

The census of everything

A line is drawn around the potential impact of any analysis at the design stage and with considerations about return on the research investment. To take an extreme example, a population census is a perfect opportunity to get nationally representative data on every topic you can imagine: standardised psychological assessment tools, income, education, and attitudes to advertising or to zoo animals. Of course, that would make a lot of statisticians very happy but would make for a census form the size of the Encyclopaedia Britannica that would take householders weeks to complete. Some collaboration with others is necessary to decide the bounds of the survey and which questions will be most useful.

Because researchers spend so much time on one topic, so much energy convincing themselves and then everyone else that what they’re doing is the most important thing in the world, there’s a risk that they’ll prioritise their research area over any potential impact. Even when they’re making a case for the relevance of their work, look closely and it might just be that they really want to keep working in the area. There’s also something cosy and familiar about staying inside the lines so they might be reluctant to look in detail at how to implement what they recommend.

There are two options: put your perfect research out there and then scoff at policy-makers’ stupidity for not doing what is so obviously right; or work with them, learn their language, and listen to their ideas from research question to final report. What you’re trying to avoid is fighting with the policy-makers, because if you’re fighting with them they’re not going to implement your world-changing ideas. It would be naïve to think that policy uses the best available evidence based on sound statistical analysis so part of the role of a statistician is to explain what the numbers actually mean, but there is also likely to be some tension.

Policies, you see, tend to be about tackling something, so if your analysis is in any way supportive of the status quo, don’t expect to get too far with the more righteous of policifiers. This is one of the issues underlying publication bias, that is, the tendency to report analysis that shows significant differences and suppress, wittingly or unwittingly, non-significant differences. People who dare to present conference papers on interventions that showed no significant impact have been described as “brave”. Sometimes it’s necessary to stick to your numbers and say “We can’t say for sure” or even “It doesn’t work”.

Implementation science

From the point of view of getting things done, though, knowing what doesn’t work is as important and knowing what does. Implementation science is about translating research into action, about identifying mechanisms for delivery. As noted in a recent post, vaccination is a fairly good idea and the reward of saving thousands of children’s lives out-weighs the small risk of harm to a small number. Reaching that conclusion is relatively easy for statisticians but the implementation of the policy is usually left up to someone else, and it turns out it can be quite hard. In some developing countries vaccination rates are lower than in developed countries, though the reasons for the low vaccination rates are beyond the scope of a blog on statistics. One issue is that the evidence suggesting that investment in immunisation is likely to provide a public health return is not backed up by evidence of how to achieve higher rates, because researchers have yet to ask and answer those types of questions.

There’s a gap between statistical analysis and implementation that parallels the relationship between the commentariat and the politicians. It’s a strait you might prefer to shout across, or you might want to make the leap and get into the details of how your analysis can make a difference. In the end, it’s about balancing the objectivity and independence of your work against the probability of something actually changing as a result. Just because Democracy Now never materialised doesn’t mean we don’t have any democracy now.

Related posts

Power heuristic – 8^th August 2014

Independence – 14^th November 2014

Scientific literacy – 9^th February 2015