Walking the line: Stats and policy development

Uncategorized
Mar 20, 2015
Walking the line: Stats and policy development

In 2011, with Ireland’s economy in the depths of recession and a general election imminent, several prominent journalists and commentators seemed to be about to launch bids for election to the national parliament as a new party called Democracy Now. They had, the argument went, been pointing out precisely what politicians had been doing wrong for many years and now was the time for them to cross the line and actually try it out for themselves. In the end, they decided against it and the politicians continue to get it wrong while they continue to point. There’s a similar line between statistical analysis that leads to policy recommendations and the implementation of those policies, and a decision to make about how close to it you want to get.

The census of everything

A line is drawn around the potential impact of any analysis at the design stage and with considerations about return on the research investment. To take an extreme example, a population census is a perfect opportunity to get nationally representative data on every topic you can imagine: standardised psychological assessment tools, income, education, and attitudes to advertising or to zoo animals. Of course, that would make a lot of statisticians very happy but would make for a census form the size of the Encyclopaedia Britannica that would take householders weeks to complete. Some collaboration with others is necessary to decide the bounds of the survey and which questions will be most useful.

Because researchers spend so much time on one topic, so much energy convincing themselves and then everyone else that what they’re doing is the most important thing in the world, there’s a risk that they’ll prioritise their research area over any potential impact. Even when they’re making a case for the relevance of their work, look closely and it might just be that they really want to keep working in the area. There’s also something cosy and familiar about staying inside the lines so they might be reluctant to look in detail at how to implement what they recommend.

There are two options: put your perfect research out there and then scoff at policy-makers’ stupidity for not doing what is so obviously right; or work with them, learn their language, and listen to their ideas from research question to final report. What you’re trying to avoid is fighting with the policy-makers, because if you’re fighting with them they’re not going to implement your world-changing ideas. It would be naïve to think that policy uses the best available evidence based on sound statistical analysis so part of the role of a statistician is to explain what the numbers actually mean, but there is also likely to be some tension.

Policies, you see, tend to be about tackling something, so if your analysis is in any way supportive of the status quo, don’t expect to get too far with the more righteous of policifiers. This is one of the issues underlying publication bias, that is, the tendency to report analysis that shows significant differences and suppress, wittingly or unwittingly, non-significant differences. People who dare to present conference papers on interventions that showed no significant impact have been described as “brave”. Sometimes it’s necessary to stick to your numbers and say “We can’t say for sure” or even “It doesn’t work”.

Implementation science

From the point of view of getting things done, though, knowing what doesn’t work is as important and knowing what does. Implementation science is about translating research into action, about identifying mechanisms for delivery. As noted in a recent post, vaccination is a fairly good idea and the reward of saving thousands of children’s lives out-weighs the small risk of harm to a small number. Reaching that conclusion is relatively easy for statisticians but the implementation of the policy is usually left up to someone else, and it turns out it can be quite hard. In some developing countries vaccination rates are lower than in developed countries, though the reasons for the low vaccination rates are beyond the scope of a blog on statistics. One issue is that the evidence suggesting that investment in immunisation is likely to provide a public health return is not backed up by evidence of how to achieve higher rates, because researchers have yet to ask and answer those types of questions.

There’s a gap between statistical analysis and implementation that parallels the relationship between the commentariat and the politicians. It’s a strait you might prefer to shout across, or you might want to make the leap and get into the details of how your analysis can make a difference. In the end, it’s about balancing the objectivity and independence of your work against the probability of something actually changing as a result. Just because Democracy Now never materialised doesn’t mean we don’t have any democracy now.

Related posts

Power heuristic – 8th August 2014

Independence – 14th November 2014

Scientific literacy – 9th February 2015

Featured Jobs

University of Birmingham.

Edgbaston, Birmingham, UK

March 25, 2019

The UK Hydrographic Office

Taunton, Somerset

March 30, 2019

Greater London Authority

London, UK

March 24, 2019

Cambridge Assessment

Cambridge, UK

March 24, 2019

Department for Environment, Food & Rural Affairs (Defra)

March 25, 2019

Siemens

Erlangen, Germany

April 07, 2019

Boehringer Ingelheim

Biberach an der Riss, Germany

April 07, 2019

Australian National University

Canberra, Australia

March 31, 2019

Department for Environment, Food & Rural Affairs (Defra)

March 25, 2019

The Royal Pharmaceutical Society

London, UK

March 31, 2019

University of Jyväskylä

Jyväskylä, Finland

April 03, 2019

Dixons Carphone

London, UK

April 06, 2019

Yorkshire Water

Buttershaw, Bradford

April 07, 2019

Countinglab

Reading, UK

May 11, 2019

Amazon Video Limited

London, UK

April 12, 2019

Lloyds Banking Group

London, UK

March 31, 2019

University of Oxford

Oxford, UK

April 17, 2019

Jet2

Leeds, UK

April 14, 2019

Health Education and Improvement Wales (HEIW)

Cardiff, Wales

March 26, 2019

Our Partners

Logo for Logo University Of Manchester
Logo for Yougov
Logo for Ministry
Logo for Ons Logo
Logo for Un
Logo for Office Depot
Logo for Mit Logo

Like what you see?

Post a job