Walking the line: Stats and policy development

Uncategorized
Mar 20, 2015
Walking the line: Stats and policy development

In 2011, with Ireland’s economy in the depths of recession and a general election imminent, several prominent journalists and commentators seemed to be about to launch bids for election to the national parliament as a new party called Democracy Now. They had, the argument went, been pointing out precisely what politicians had been doing wrong for many years and now was the time for them to cross the line and actually try it out for themselves. In the end, they decided against it and the politicians continue to get it wrong while they continue to point. There’s a similar line between statistical analysis that leads to policy recommendations and the implementation of those policies, and a decision to make about how close to it you want to get.

The census of everything

A line is drawn around the potential impact of any analysis at the design stage and with considerations about return on the research investment. To take an extreme example, a population census is a perfect opportunity to get nationally representative data on every topic you can imagine: standardised psychological assessment tools, income, education, and attitudes to advertising or to zoo animals. Of course, that would make a lot of statisticians very happy but would make for a census form the size of the Encyclopaedia Britannica that would take householders weeks to complete. Some collaboration with others is necessary to decide the bounds of the survey and which questions will be most useful.

Because researchers spend so much time on one topic, so much energy convincing themselves and then everyone else that what they’re doing is the most important thing in the world, there’s a risk that they’ll prioritise their research area over any potential impact. Even when they’re making a case for the relevance of their work, look closely and it might just be that they really want to keep working in the area. There’s also something cosy and familiar about staying inside the lines so they might be reluctant to look in detail at how to implement what they recommend.

There are two options: put your perfect research out there and then scoff at policy-makers’ stupidity for not doing what is so obviously right; or work with them, learn their language, and listen to their ideas from research question to final report. What you’re trying to avoid is fighting with the policy-makers, because if you’re fighting with them they’re not going to implement your world-changing ideas. It would be naïve to think that policy uses the best available evidence based on sound statistical analysis so part of the role of a statistician is to explain what the numbers actually mean, but there is also likely to be some tension.

Policies, you see, tend to be about tackling something, so if your analysis is in any way supportive of the status quo, don’t expect to get too far with the more righteous of policifiers. This is one of the issues underlying publication bias, that is, the tendency to report analysis that shows significant differences and suppress, wittingly or unwittingly, non-significant differences. People who dare to present conference papers on interventions that showed no significant impact have been described as “brave”. Sometimes it’s necessary to stick to your numbers and say “We can’t say for sure” or even “It doesn’t work”.

Implementation science

From the point of view of getting things done, though, knowing what doesn’t work is as important and knowing what does. Implementation science is about translating research into action, about identifying mechanisms for delivery. As noted in a recent post, vaccination is a fairly good idea and the reward of saving thousands of children’s lives out-weighs the small risk of harm to a small number. Reaching that conclusion is relatively easy for statisticians but the implementation of the policy is usually left up to someone else, and it turns out it can be quite hard. In some developing countries vaccination rates are lower than in developed countries, though the reasons for the low vaccination rates are beyond the scope of a blog on statistics. One issue is that the evidence suggesting that investment in immunisation is likely to provide a public health return is not backed up by evidence of how to achieve higher rates, because researchers have yet to ask and answer those types of questions.

There’s a gap between statistical analysis and implementation that parallels the relationship between the commentariat and the politicians. It’s a strait you might prefer to shout across, or you might want to make the leap and get into the details of how your analysis can make a difference. In the end, it’s about balancing the objectivity and independence of your work against the probability of something actually changing as a result. Just because Democracy Now never materialised doesn’t mean we don’t have any democracy now.

Related posts

Power heuristic – 8th August 2014

Independence – 14th November 2014

Scientific literacy – 9th February 2015

Featured Jobs

Institute of Statistical Science Academia Sinica

Taipei, Taiwan

December 27, 2019

London School of Hygiene and Tropical Medicine

London, UK

November 19, 2019

EUMETSAT

Darmstadt, Germany

December 19, 2019

University of Alberta

Edmonton, AB, Canada

December 18, 2019

AlphaPlus

Castlefield, Manchester

November 22, 2019

UCLA Department of Statistics

Los Angeles, CA

December 10, 2019

The National Audit Office

London SW1

December 01, 2019

The Institute of Cancer Research

Sutton, Surrey, UK

December 02, 2019

University of Glasgow

Glasgow, Scotland

December 09, 2019

City of Westminster Council

Victoria Street, London, UK

December 31, 2019

Greater London Authority

London SE1

November 24, 2019

The Department for Work and Pensions

Leeds, West Yorkshire

December 14, 2019

Universidad Carlos III de Madrid

Madrid, Spain

December 01, 2019

Ministry of Justice

London E14

December 11, 2019

Competition & Markets Authority

Canary Wharf, London

November 18, 2019

University of the Incarnate Word

San Antonio, TX

November 20, 2019

Our Partners

Logo for Logo University Of Manchester
Logo for Yougov
Logo for Ministry
Logo for Ons Logo
Logo for Un
Logo for Office Depot
Logo for Mit Logo
Logo for Fca

Like what you see?

Post a job