« Peter Taylor's Generation Jihad: are we in a golden period of British security? | Main | GCHQ: BBC fails to crack the code »
February 10, 2010
Dept. for Business, Innovation and Skills' poor use of research in THES article
There is a good article in Times Higher Education Supplement this week: criticising some poor use of research by the UK's Department for Business, Innovation and Skills (BIS). I'm quoted saying that a report on 'future jobs'
promoted by many including Lord Drayson, the Science Minister, and Gordon Brown...is substandard...used "unjustified methodologies" to reach its conclusions and was "overly reliant" on weak online sources and media reports, with some sections referencing only Wikipedia.Although sources are referenced...the report lifted "significant passages of text" word for word.
You can read the report on the future of the job market - written by Rohit Talwar and Tim Hancock at Fast Future Research - online (PDF). The report was used for the Science: So What? So Everything? science communication campaign, and it really is strikingly bad. Problems range from the systemic (methodologies are unjustified: for example, the survey used doesn't really answer the research question or prove much beyond 'some people said some things') to tediously basic flaws (for example, relying on Wikipedia as a citation for some points or referring to 'future jobs' which have been in existence for many years). The report comes nowhere near the standards that one would expect from work commissioned for a department like BIS - and I would have serious objections to the use of such bad social science in the name of science communication.
My interest in this comes from some a couple of papers I'm working on - looking at science communications and at the role of impact in future UK research assessment, respectively. BIS has a role in both science communication and changes to research assessment - so I do feel that their mis/use of research is significant.
When I pointed out the obvious and serious problems with the report, I was expecting BIS to simply apologise and drop it. However, they chose to defend the report - after which, THES took an interest.
Even after the THES piece, BIS continue to stand by this report: telling THES that
the report was a "speculative look" into the future intended to spark public interest in science and did not inform policy or affect funding decisions.
Responding to THES, Rohit Talwar
maintained that citing websites, news reports and blogs was "accepted best practice in horizon scanning.
Except in very specific circumstances, even undergraduate students would be advised against this type of 'best practice'.
Of course, the UK government is free to publicise and commission whatever research they want. However, it is unfortunate if they choose to focus on such poor research - even after its quality is pointed out to them. BIS's role in here is especially worrying, given their major role in universities and research.
Further reading
I found Zoe Corbyn at THES very thorough, and am pleased with the story which has come out. It's worth noting, too, that a number of other people have looked at issues around BIS's Science: So What campaign. 2020 Science offers a good take on the science communication aspects of this, while Gimpy's Blog and 10minus9 both look at aspects of the science of Fast Future's report. Kylie Sturgess used TurnItIn to find lifted passages of text in the report. It will be interesting to see how this story develops - if BIS continue to use this report, there will be plenty of opportunities for others to add their voices to this debate.
Posted by jon_mendel at February 10, 2010 10:27 PM
Trackback Pings
TrackBack URL for this entry:
http://www.watsonblogs.org/cgi-bin/mt/mt-tb.cgi/1744
