How big does the ‘works’ in ‘what works’ need to be?

Running man

Anyone who knows me will know I am a keen runner. I have been so most my life with just a few ‘down periods’. The most recent down period was after a knee operation in 2013. 

I can think of three things that got me running again that time. Perhaps the most unexpected was reading an interview with Sylvester Stallone about staying fit at 60. Second was Amy McDonald’s song ‘Run’, an anthem to keeping passion in your life as you age. And, closest to home, my daughter Nadeen got me a custom made T-shirt saying ‘Runner for Life’.

Many things motivate people, and different things motivate different people.

The point of this story is that many things motivate people, and different things motivate different people. Economists have traditionally thought in terms of monetary incentives. But at New Lanark, Robert Owen motivated workers through coloured flags showing their productivity. The Soviet Union used exhortation and praise to promote model workers under the Stakhanovite movement. Public health campaigns assume behaviour change is motivated by receiving information on the benefits of good practice. Many interventions assume that new practices will be diffused through observation effects. Studies from psychology show the importance of peer effects in behaviour, as well as obedience to authority and the impact of being watched.

This wide range of factors which affect behaviour has implications for both research and practice.

A first implication is the need to look at mechanisms rather than interventions. Especially in the USA, branded programmes are common. But knowing a particular branded programme ‘works’ is of limited used. What are the mechanisms – or elements – it contains which make a difference? That is not ‘what works?’ but ‘what works and why?’.

A second implication is for expected programme impact. Not everyone responds to the same incentives. Probably few readers of this blog would have been motivated by a T-shirt to put in nearly 2,000 miles of running in a year. So, different approaches are needed for different people. As captured in the funnel of attrition, if 100 people are exposed to a behaviour change intervention, perhaps 60 engage with it, and only 20 people change their behaviour. Programme planners and researchers doing power calculations often assume all 100 will respond positively to the intervention.

So effect sizes are smaller than we might hope or expect. In the UK, Parkrun – free 5k runs held at local parks around the county – is credited with a ‘participation revolution’. Close to 100,000 runners now take part each week, which is a remarkable achievement.  But that’s less than 0.2% of the population.

Changing behaviour of 10-20% of the target population with a single intervention is a good result.

How big is a big effect? That is a matter for policymakers, which I have addressed in an article for the 3ie blog. Generally, it is not that much. Changing behaviour of 10-20% of the target population with a single intervention is generally a good result. Expect more than that as a policymaker and you are likely to be disappointed, and judge successful programmes as failures.

The evidence-based policy movement needs to manage expectations about what ‘what works’ means, if effective programmes are not to be written off as failures.

Contact us