September 10, 2010
A vicious circle: Science So What, BIS and the mainstream media
Alexander Holmes and I blogged about issues around research, PR, communications and policy on the Guardian Science Blog earlier this week. Here's the longer version of the post (the Guardian cut it down slightly); we have also added a postscript discussing recent debates about science funding in the UK.
The Department for Business, Innovation and Skills (BIS) is a government department whose responsibilities include the UK’s scientific and university sectors. They support research and teaching as well as overseeing efforts to encourage people to take up scientific and research careers. Under the previous government, the department ran a campaign called Science: So What? So Everything (SSW) to encourage young people via websites, media reports and special events, to be inspired by the contributions of science to their lives.
SSW was not without its problems. The campaign included a website that was expensive and inefficient: BIS spent £1.85m on a website that compared poorly with science communication sites set up for far smaller sums. The site got little traffic for a campaign of this type and there were serious concerns about the quality of some of the research that BIS was promoting. In particular, a report on future jobs in science by the Fast Future consultancy was heavily promoted during the campaign despite failing to meet some basic standards.
The department and its SSW campaign have both come under fire, with various researchers criticising them in public and private. We were interested in how BIS responded internally to these criticisms, which sought to improve their activities. So we submitted a Freedom of Information Act (FOIA) request to find out.
Good quality research depends upon robust, critical appraisal. As BIS is a major player in the UK’s research work – and as the SSW campaign was intended to promote science, technology, engineering and mathematics – we hoped their own practices would reflect the standards that contribute to the UK’s deserved reputation for excellent research. We hoped, for example, that since the department plays a role in assessing the quality of research in UK universities the research it commissioned itself would be robust. Our findings are hardly encouraging. They suggest BIS did not respond appropriately to concerns about the SSW campaign and that their way of measuring success was questionable to say the least.
The report on future jobs in science was commissioned and promoted by BIS, under the former Business Secretary, Lord Mandelson, as part of the SSW campaign. This report was garlanded with supportive statements from the then science minister, Lord Drayson, and even the then Prime Minister, Gordon Brown. However, as soon as it was released, major concerns were raised by bloggers and academics about the methodology, the inappropriate use of Wikipedia and implausible claims about nanotechnology, amongst many other problems. These serious issues were largely missed by the mainstream media, beyond a blog on the Guardian's science website and an article in the Times Higher Education Supplement (THES) that criticised the report.
The Conservative Party – then in opposition – also failed to challenge the report effectively. It even issued a press release that added further errors. For example, it argued that a worldwide survey used for the report “determined that ‘Virtual Lawyer’ is the fantasy job which people in Africa, Peru and Pakistan think is ‘likely to be the best paid’.” But as the Fast Future report makes clear, this was based on responses from only one person in Peru and one in Pakistan. It would be rather tenuous to assume their compatriots share their views. It is unfortunate that the Conservative Party’s criticisms of such a flawed document were themselves so ill conceived. More worryingly, the press release went out with David Willetts’ name in the headline along with a lengthy quote. Willetts is now minister for universities and science.
When BIS evaluated the success of the future jobs report, it used media coverage as a gauge and all but dismissed its critics. Our FOIA request shows that the PR agency Kindred (which worked for BIS on the project) focussed on mainstream media coverage when evaluating the success of the campaign around the Future Jobs report. They noted that this report achieved “178 pieces of coverage across national, regional, consumer and online media…A combined OTS (opportunities to see) of 60,985,597…An AEV [Advertising Equivalent Value] of £2,248,866.” This is a poor measure of success in science communication. Public understanding of and engagement with science cannot usefully be measured by column inches in the press, without also considering the accuracy and efficacy of the project in question.
There were also crude attempts to assess the online impact of the Future Jobs report coverage of the SSW campaign. Kindred report that the story “generated a seven-fold increase in volume of traffic to the campaign website.” The increase raised the traffic to “7,733 website hits during the six days after the launch of the activity (compared to 1,167 websites hits for the same period before the activity launch)”. For a science communication campaign aimed at millions of young people and backed by a £1m-plus budget, this trumpeted increase is pathetic.
In dealing with criticisms, BIS and Kindred focussed on managing negative publicity rather than on correcting mistakes or meaningful engagement with critics. For example, when the nanotechnology blogger James Hayton criticised the Fast Future report, an e-mail exchange supplied in response to our FOIA request argued that “James’ blog isn’t particularly well known…Not that this means his criticisms aren’t well-founded, but I doubt appeasement will be a worthwhile strategy.” The emails are so heavily redacted, it is impossible to know whether the email was from a civil servant or a BIS contractor. In deciding whether to respond to Hayton's blog, these e-mail exchanges gave considerably more attention to whether Hayton’s criticisms would appear on the Guardian science blog and how to distance BIS from any criticism than was given to the accuracy and significance of his points: “Given the reach of the Guardian blog, we believe that it is a worthwhile exercise for Rohit [Talwar, the author of the future jobs report] to provide some form of response”. The e-mail exchange states that “while tacitly looking over Rohit’s response, it needs to come from him (rather than Kindred, and certainly not BIS)”.
Responses to mainstream media criticism of BIS’s practice were no better. In response to the Times Higher Education Supplement article, BIS are quoted emphasising the “speculative” nature of the research behind the future jobs report. It is left to Talwar to claim that the approach taken is “accepted best practice in horizon scanning”. The importance of the THES was downplayed, with one email exchange citing a single tweet stating that “Jonathan Mendel [quoted in the THES article criticising the future jobs report] is a prat” as evidence that there was little interest in the story. Substantive criticisms from the THES article and elsewhere were not addressed in the documents supplied to us.
When preparing a statement on the response to the future jobs report for the then science minister, Lord Drayson, a draft saying that the “vast majority” of coverage of the campaign was positive was revised to marginalise criticism further: simply stating that the campaign “has generated a great deal of positive coverage”. Failures by BIS to uphold basic standards raise concerns about how they engage with the professions within their remit and are frankly embarrassing for those of us who work within and are keen to promote the UK’s excellent research sector.
While BIS failed to redact Hayton's name from the documents released to us and left at least one other individual easily identifiable, they redacted so much from the e-mail exchanges that it is not always clear which organisation is saying what. Given that the government is now arguing that “transparency across all departments a necessary and important part of making government more accountable” it is unfortunate that this FOIA response was excessively and clumsily redacted. We appreciate it can be important to protect the identities of individuals, but there is a real public interest in knowing whether particular statements were made by government departments or by contracted organisations. Not only have BIS been involved in commissioning and promoting substandard research, their current data-handling practices are making it harder for researchers to assess the problems with BIS’s practice.
Despite these redactions, what stood out in the documents BIS released to us was how government, PR agencies and mainstream media worked as a closed and vicious circle. The Government commissioned and promoted bad research; PR agencies promoted this to the media and the media overwhelmingly reported the government line. The project was then deemed a success because of the positive media coverage. Critical engagement with research and appropriate analysis of the tools, goals and achievements of research communication were very much marginalised and the Conservative Party’s press release only added to the misinterpretations it should have been challenging. Recent government announcements, such as changes to housing benefit, also seem worryingly under-researched in how they are presented by the government and reported by the press. Given the damaging nature of such vicious circles, there is a real need for those of us outside of these circles to find ways to criticise and challenge bad policies and practice in order to improve them.
Despite concerns about David Willetts’ role in the Conservative criticisms of SSW, we hope the new government will engage better with researchers. This means being open to a genuine dialogue and listening to constructive criticism. However, although pre-election Conservative and Liberal Democrat rhetoric emphasised evidence-based policy, we are yet to see any convincing signs that this government will engage well with those who have genuine concerns about its policies. Consider the manner in which Nick Clegg has challenged Institute for Fiscal Studies work to argue that the government’s policies are progressive (http://www.guardian.co.uk/politics/2010/aug/25/nick-clegg-budget-report-partial). We have real concerns about whether the government has even done the necessary research to know if its policies are progressive or not. Promises of engagement and evidenced-based policy seem to have become less of a priority than ensuring that mantras of cuts, austerity and reform remain in the headlines.
The recent announcements by the Secretary of State for DBIS, Vince Cable, and subsequent criticism raise concerns that this government is not setting an example at ministerial level with respect to the appropriate evaluation of evidence.
Some of the angriest responses to Dr Cable this week were over his misleading interpretation of data from RCUK on the quality of research produced in the UK science sector for which he has yet to clarify despite a robust correction from RCUK. Given the ease with which certain politicians can misinterpret research and win uncritical media coverage for these misinterpretations we are concerned that Dr Cable may have established an inaccurate headline figure that will be used in subsequent debates on science funding in the media. Evan Harris (ex-MP, and former Liberal Democrat science spokesperson) has called on his former colleague to correct this figure:
Lest that figure of 45% should stick in the mind of the public, as it is presently stuck in the craw of the scientific community, what's needed first is a proper acknowledgement from the business secretary of the true assessment of the quality of British science. It is important for Mr Cable to put the record straight.
While it is important for Dr Cable to correct his mistake, this should not be the end of the matter. The government appears to have ignored the arguments of the previous parliament's Sci & Tech committee report on science funding, chaired by a Liberal Democrat and featuring Evan Harris. This report was highly critical of planned £600m cuts by the previous government and argued strongly - and from a wide evidence base - that even in difficult economic terms it is highly unwise to cut basic research. As Harris argued after Cable's speech there are concerns that:
To believe it is possible to get more good science from less funding is the political equivalent of a belief in cold fusion: an aspiration not supported by logic or reproducibility...there is very little room to identify wastage...The real question is one for the government to address. Is there any evidence from anywhere that can be cited showing any changes to current funding allocations could reliably generate "more" from less? Even then it is not clear what the "more" is.
We would agree that this is an important question for the Government to answer: 'more for less' in research terms would be nice; however, one would not want to base major policy decisions on unevidenced, under-researched and barely plausible claims. Is there any good reason for the Government to think that more research for less is either
plausible or likely?
Despite the Liberal Democrat Deputy Prime Minister, Nick Clegg, offering a commitment to science before the election and more recently extolling the virtues of long term thinking with respect to funding the government must be judged by its actions not its words. Thus far the failure of ministers to correct glaring mistakes and to engage with important previous reports is far from reassuring. It could be disastrous if this style of government were to persist.
Posted by jon_mendel at September 10, 2010 07:23 PM
TrackBack URL for this entry:
Grief, the behind-the scenes stuff is even more of a train wreck than I realised.
Posted by: Ceri at September 10, 2010 11:03 PM