Jonathan Mendel

« Unrest, networks and connectivity | Main | RGS impact statement - Science: So What and the impact of bad research »

September 25, 2011

Times piece: UK Government basing claim about benefit reforms on poor analysis?

David McCollum and I had a piece on the Times Science Blog this week, criticising the evidence behind much-publicised claims that moving the UK's benefits system to a Universal Credit will seriously reduce the number of workless households in Britain. You can read the piece on the Times site (paywalled), or it's reproduced below:

300,000 additional households into work due to Universal Credit – fanciful thinking?

The Universal Credit was announced with much fanfare in late 2010 – with the aim of ‘radically simplifying the system to make work pay and combat worklessness and poverty’ [1]. A key claim made in the DWPs Universal Credit White Paper is that the new measure will reduce the number of workless households by around 300,000 within two to three years of its implementation. This claim – sourced to “internal analysis using the Department for Work and Pensions’ Policy Simulation Model” – has been widely picked up and reiterated in the media, including by Nick Clegg in The Guardian [2].

Sceptical about this 300,000 figure, we used a Freedom of Information request to get details of the analysis the DWP used to generate it. Closer inspection shows that this figure was generated by inappropriate use of a fairly narrow piece of research and by making and sandwiching together broad assumptions and best-case scenarios – giving an over-optimistic prediction. There is evidence that some in the DWP were aware of how tenuous such conclusions are, but this uncertainty was hidden in what was presented to the public.

In the DWP’s analysis, 250,000 of the predicted reduction in workless households is accounted for by the assumption that 1,300,000 workless households will ‘respond’ to the introduction of Universal Credit, 500,000 of whom will be financially better-off because of its improved ‘incentives’ and that around half of this sub-group will move into part-time jobs. These figures are based on ‘research evidence and assumptions about future levels of conditionality and support’. However, the DWP document fails to note what evidence this is or to make clear the assumptions upon which these inferences are based, especially the presumption that half of this sub-group will move into part-time jobs (why half and why part-time jobs?)

The DWP's simulation draws broad conclusions about why In-Work Credit was effective, based on a much narrower study of the New Deal for Lone Parents (NDLP) which makes explicit – on p. 1 – that “findings are illustrative but not statistically representative” [3]. The aspects of the New Deal assessed were optional, involved relatively few people and were resource intensive; they offered a lot help with getting into work and staying in work. These characteristics probably help to explain why the transitions into work from NDLP lasted on average for three years (most transitions from welfare to work don’t [4]) but it would be wrong to assume that this will apply to those affected by the Universal Credit.

It is assumed that 1,930,000 people will receive a form of Better-Off Calculation under Universal Credit, that most of them will be in workless households, that 20,000 people make the transition into work every year for three years as a result of the Calculations and that these transitions last for three years on average. Again, these projections draw on research involving optional and resource-intensive aspects of NDLP. It is unrealistic to apply the same assumptions to the nearly two million people who are expected to receive a form of Better-Off Calculation under Universal Credit.

The DWP goes on to assume that 1,330,000 people will be eligible for In-Work Credit and that this will smooth the transition between benefits and work to the extent that 40,000 of them will move into employment, which will be sustained on average for three years. Again, the DWP applies research from NDLP to a much larger and more diverse group of nearly two million individuals who are believed to be in workless households.

The 300,000 figure generated by the DWP analysis is thus over-optimistic and based on unsound analysis. It should, at a minimum, come with clearly specified ‘health warnings’ regarding the uncertainly surrounding such estimates: after all, the Policy Simulation Model is acknowledged to be a ‘simplification of the real world and subject to a number of sources of uncertainty’. It appears that some within DWP did feel that strong caveats were needed: Tracked Changes in the internal analysis released to us show that an earlier draft made clear that “we think this is a plausible estimate, but recognise the true impact is very uncertain”; however, the final draft simply states that “we believe this is a plausible estimate based on reasonable assumptions”.

It is very worrying for the DWP to be complicit in producing and publicising unsubstantiated claims. These claims are used to justify reforms that will disproportionally hit the most vulnerable members of society – those who already stand to lose most from the recession and government cuts. If the Coalition really cares about getting more people into work, there is a real need to move beyond catchy but unsubstantiated headline figures – and to make better use of high-quality research in order to assess what benefit reforms might achieve and who they might hurt.


Posted by jon_mendel at September 25, 2011 06:38 PM

Trackback Pings

TrackBack URL for this entry: