|
ISIS Press Release 20/05/08
SiS Commentary
Peer Review under the Spotlight
What matters most is the lack of public scrutiny rather than the lack of
peer review in times of corporate corruption of science Prof.
Peter Saunders
A fully
referenced version of this article is posted on ISIS members’ website. Details
here
An electronic version of this report, or any other ISIS report, with full references,
can be sent to you via e-mail for a donation of £3.50. Please e-mail the title
of the report to: report@i-sis.org.uk
You’ve probably come across the expression “peer-reviewed”
a lot recently, especially in discussions on GM food, mobile phones, or organic
farming. It’s almost always used as part of a sentence that begins “There
is no peer-reviewed evidence for …” or “There is nothing about this
in any peer-reviewed journal …”
What you’re meant to understand by that is: “there is no credible
evidence for whatever it is, and you can safely ignore anything you’ve heard
about it.” When the question is about safety, as it often is, it means the
regulatory authorities are not going to look into it.
A lot of people take peer review very seriously, or at least
they say they do. When Sir David King, then the Chief Scientific Adviser to
the British government, put forward a code of ethics for scientists, one of
his chief examples of unethical behaviour, right up there with plagiarism,
was “disseminating work before it has been peer reviewed”. You may remember Arpad Pusztai who spoke for 150 seconds in a television
programme on unpublished results indicating that genetically modified (GM)
potatoes were harmful to rats, because he saw it his duty to warn the public.
He was subjected to fierce attacks from the scientific establishment (led
by the Royal Society) that continue to the present day.
The scientific establishment’s double standard
The scientific establishment
may claim to oppose disseminating results that have not been peer-reviewed,
but there is a blatant double standard being applied, and all too often. Just
recently, a UK government research funding agency, the Economic and Social
Research Council (ESRC) put out a press release that was not only highly misleading
about farmers being upbeat about GM
crops ("UK Farmers
Upbeat about GM Crops" Debunked and Marketing Masquerading as Scientific
Survey , SiS 38); but was also based on research that had not been peer-reviewed,
according to the ESRC’s own web site.
Another recent
example came from the top mainstream journal Nature Biotechnology.
In an editorial, it criticised the Italian National Research
Institution for Food and Nutrition (INRAN) for not publishing some results
that were allegedly favourable to GM crops.
The director
of the project in question, Giovanni Monasatra, wrote to the journal to put
the story straight, and his letter was published along with a response from
the editor, Andrew Marshall. In his letter, Monastra dealt with the points
raised in the editorial and expressed his surprise that Marshall, far from
criticising Salute, Agricolura, Ricerca (SAGRI) for organising a press conference
to publicise data that were, according to Marshall, “too preliminary for peer-reviewed publication,” instead complained
that the Italian media did not give
it even more coverage than they did.
Marshall’s response
is that the data had [his italics] to be press released by SAGRI because
they were of interest to the public and political debate. Yet, in 1999, Marshall had written in Nature Biotechnology that Arpad Pusztai’s work should be submitted for peer review
before it could be considered, even when safety was at stake. The difference
is that he was then writing about results that were against the interests
of the biotech industry.
It is not at all unusual for scientific
bodies and lobby groups to issue press releases and hold press conferences
on non-reviewed material. The people who set themselves up as the guardians
of sound science either say nothing or even join in, except when it is a matter
of things the corporations don’t want the public to hear. In that case, they
suddenly rediscover their strong objection to the practice.
Peer review is a useful part of the
scientific process, but it is not as effective, as important, or as universal
as some would have us believe, and it needs to be put in perspective.
What is peer review?
One of the distinguishing features of science is that when you discover something,
you don’t expect other people just to take your word for it. You’re expected
to describe exactly what you have done and why this
justifies what you claim, and the usual way of doing this is to publish a
paper in a scientific journal.
When you submit your paper to a journal,
the editor will generally send it to be reviewed by experts in the field.
The referees, usually two or three, are supposed to read the manuscript carefully
and assure themselves as best they can that the work was done using appropriate
techniques, that it takes into account and properly acknowledges earlier relevant
work, and that the conclusions are properly derived from the data, or, in
the case of a theoretical paper, that the arguments are sound. They advise
on whether the work is interesting enough and contains enough that is new
to be worth publishing, and, even if it is, whether they consider it is suitable
for the particular journal. They may also suggest ways in which the paper
could be improved.
This peer review system is important
in science. It prevents many very poor papers from being published and it
improves many others. Above all, it helps maintain a consensus of what is
expected in a scientific paper; what you find in a scientific journal is very
different not only from the popular press but even from most papers in the
humanities or social sciences.
But peer review is very limited in
what it can do. Referees, who are not paid, vary considerably in the time
and effort they devote to the task. They are all too likely to nod a paper
through if it looks plausible and comes from a lab or a group that they know
and trust, or to reject one because they disagree with it or don’t understand
it and haven’t the time or inclination to go through it carefully. They may
reject a paper as “not interesting” when what they mean is that it’s not the
sort of thing they and their friends are interested in.
Referees do not go into the laboratory to watch the experiments
being carried out. They do not have access to the authors’ notes and raw data.
They make their decisions on the basis of nothing more than what the reader
will see if the paper is accepted. Even the most conscientious are simply not
in a position to guarantee that the results are correct or even that the work
was done properly.
Peer review could certainly be improved,
and there are a number of ways in which this might be done. For example, research
has confirmed the suspicion of many scientists that there is often bias, whether
conscious or not, and it has been suggested that referees should not be told
the authors’ names or institutions, or even their gender. But while the system
could be improved, it is hard to see how it could do much more, even if we
really want it to. The real test of a paper comes after it is published and
is open for comment by the whole of the scientific community, and that can
be a far more stringent test. Indeed, many poor or outright fraudulent works
have been exposed after they were in print. One recent example is a paper
published in the British Food Journal and given an Award for Excellence,
which provoked 40 scientists and two MPs to sign an open letter demanding
its retraction (Wormy Corn Paper Must be
Retracted, SiS 37). We shouldn’t expect
peer review to do more than it actually can, and by the same token we shouldn’t
claim to the public that it does.
Not all science is peer reviewed
If all real science were peer reviewed before it
was made public or used in decision making, then you might want to
say that peer review, rightly or wrongly, defines real science. In fact, there
are a number of ways in which science often gets into circulation without
peer review. For example, the work can be published in the proceedings of
scientific meetings or, especially in rapidly moving subjects like theoretical
physics, circulated as a preprint or posted on a web server.
On the whole, exceptions like those don’t matter much. Most of
the work will eventually be refereed or else just forgotten. In any case,
it is out in the open for anyone to see and criticise.
The absence of peer-review matters in regulation but not as much as the absence
of public scrutiny
There is, however, an area in which the absence of peer review matters a
great deal, and that is in regulation. Many products,
including pharmaceuticals and GM foods, have to be licensed. The manufacturers
are required to carry out trials, safety tests and risk assessments and submit
the results to the regulators.
Much of this work is never peer reviewed. What is more, much
of it is never published, and worse, concealed from
the public and often from the regulators as well under claims of “commercial
confidentiality”, so that other scientists are never able to comment on
it.
Companies use commercial confidentiality in much the same way
that the UK government uses the Official Secrets
Act, less as a means of keeping sensitive information from a possible
competitor than to ensure that nothing embarrassing reaches the public. And
as with the Official Secrets Act, the claim that something must be kept confidential
on commercial grounds is seldom challenged. Just to quote one example, even
after the TGN1412 trial went so disastrously wrong (see Post Mortem on the TGN1412 Disaster,
SiS 30), and it was abundantly clear that the drug would never be developed
further, the Medicines and Healthcare Products Regulatory Agency (MHRA) still
refused to release some details of the test protocol on the grounds of commercial
confidentiality. It is hard to imagine how a competitor of either Te Genero,
the company that developed the drug, or Parexel, the company that ran the
trials, could have gained any unfair advantage from the information, though it might have been of use to anyone trying to improve
the safety of trials, and more importantly, to provide effective remedy for
the victims.
The absence of peer
review is nowhere near as important as the absence of public scrutiny. To
make claims on the basis of research that you will not reveal to the scientific
community is to go against one of the basic principles of science: that we
provide evidence for the claims we make. It is all the more serious when these
claims can affect health and the environment. Such data should never be held
secret on grounds of commercial confidentiality.
Conclusion
Peer review is a useful
part of science but it is not and cannot be the dividing line between good
science and bad, between what can be relied upon and what must be dismissed
out of hand. In particular, when we are told
that there is no peer-reviewed evidence, that does not mean that there
is no evidence, nor does it mean there is no credible evidence. The scientific establishment has been deliberately
applying a double standard to exclude evidence unfavourable to industry. What
is worse, our regulators have accepted all kinds of evidence in approving
new products and processes not just without peer review, but without the possibility
of scrutiny by the public or even by the regulators themselves.
| |
|
