Brian Krueger is the owner, creator and coder of LabSpaces by night and Next Generation Sequencer by day. He is currently the Director of Genomic Analysis and Technical Operations for the Institute for Genomic Medicine at Columbia University Medical Center. In his blog you will find articles about technology, molecular biology, and editorial comments on the current state of science on the internet.
My posts are presented as opinion and commentary and do not represent the views of LabSpaces Productions, LLC, my employer, or my educational institution.
Please wait while my tweets load
Open science is a wonderful concept, but what happens when reporters start writing stories on data that has not been properly reviewed and vetted by the scientific establishment? Before this week, I had never really considered this question. Open science at its core is a wonderful utopian idea where scientists do their work in the open and publish their notebooks in real time on the web for everyone to see. The idea is that with this kind of transparency, better science will be done and scientists can collaborate more easily. Because all of the data will be on the internet and searchable, more scientists will be able to benefit from the open resource. Of course, there are numerous criticisms of open science. One being that it will be extremely easy for researchers in highly competitive fields to be scooped by competitors who have bigger labs or more resources at their disposal. However, it didn't occur to me until I saw stories popping up that open science could be abused by the media.
Almost a year ago, NASA held a press conference touting that it had found "alien" life. A group of researchers reported that they had found a bacteria (GFAJ-1) in Mono Lake that incorporated arsenic in place of phosphate in its DNA backbone. This press conference and the subsequent paper met steep criticism from the microbiology field. Arguably, the most vocal critic was Rosie Redfield, a researcher at the University of British Columbia. Redfield has since gone on to conduct an open science project to disprove the findings of the now infamous "arsenic life" paper. You can follow the progress of the project on her blog. She recently submitted the work to the journal Science for peer review and uploaded a copy of the paper to arXiv, a service that pre-publishes papers for scientists to review before the official peer review process has completed. Personally, I think this kind of science is awesome. It gets the data out immediately for everyone to review; however, I do have a problem with unscrupulous reporters writing stories on the project at mainstream news outlets before the research has been properly vetted and reviewed by the scientific establishment.
One of these stories appeared on New Scientist this week with the title, "Arsenic life does not exist after all," another from MSNBC popped up in my feed this morning, "Study counters arsenic-life claims," and while looking for the New Scientist link I found a third from ScientificAmerican, "Study Fails to Confirm Existence of Arsenic-Based Life." Reporting on data that has not gone through the peer review process as if it were truth is not responsible journalism. We see this a lot in the media though and rarely know that it is ocurring. There are plenty of press releases from universities and government agencies, especially around conference season, that tout wild claims but have no scientific literature to back them up. All that is mentioned is a conference abstract or a presentation time. As a scientist, I know to take these "findings" with a grain of salt. Without proper peer review, all data should be considered preliminary. I understand that the news cycle these days is very short and everyone is looking for a quick lead to a rockstar story, but reporting on unpublished data, be it from an open science project or unpublished data from a conference, is not doing justice to the scientific process.
Further it is quite premature to consider this argument over, once an idea has been published, it is very hard to reverse the finding. A great example of this is the recent discovery that XMRV is not the causitive agent behind chronic fatigue syndrome. Originally, researchers had shown that chronic fatigue patient samples contained DNA from a mouse virus, XMRV. However, follow up studies could not confirm this and it was finally discovered that the reagents used to conduct the initial experiments were contaminated. It took nearly 3 years with multiple publications from a diverse set of labs for the original papers to be finally retracted. It's going to take the work of more than one researcher to prove that the bacteria isolated from Mono Lake does not contain Arsenic in its DNA. It would also be beneficial to see confirmatory data come from an impartial lab. As an extreme critic of the original research, some may think that Redfield has a lot to gain by being "right." We can also expect to see further work from the original researchers as they try to counter the criticisms of their original study. The battle for arsenic life is hardly over.
This post has been viewed: 3572 time(s)
"Reporting on data that has not gone through the peer review process as if it were truth is not responsible journalism."
I have also been shocked, SHOCKED, I say, to see a paper deposited in arXiv being reported around the world by researchers and journalists alike. Nobody commented that it hadn't been accepted in a peer-reviewed journal.
Except I'm not talking about Redfield's arsenic life paper.
I'm talking about the reports of faster than light neutrinos from OPERA.
Did we hear howls of outrage from the physics community over the coverage of the story? More like chirping crickets. Physicists were right there in the thick of the discussion.
This is an example of the differing cultures of the fields. Physics has developed a pre-print culture where people stake their claims with manuscripts. Biology has developed a culture where people stake their claims with final publications. But cultures change, and it's individual cases like this one that provide a lot of the push to change.
Just because biologists normally only make findings public very near the last step in the publication change does not change that anything made public at any stage is fair game for reporting.
Dr. Redfield made her work public earlier than others would have done. Unusual, but I cannot see the ethical issue with reporting on information that she has voluntarily shared. Indeed, if her work does not pass peer review, the reporting that is going on right now can add context to the story.
The arsenic life story itself tells us that because a paper has been peer reviewed does not make it automatically credible to other researchers. Biologists on the whole weren't convinced by the claim.
That's a great point Zen. But are FTLN a good example that science reporting should aspire to? How many people stopped following the debate after the first story? How many of those people still think that FTLN exist? I'd bet a fair amount still have the original soundbyte in their head. Reporting on data that has not been fully vetted and peer reviewed is doing a disservice to the community. Certainly there are a lot of critical reviews done over on arXiv, but FTLN and this arsenic paper present a clear case where journalists should sit back and wait for a consensus and review before spewing potential nonsense that is extremely hard to retract from readers' brains once they've read it.
There ARE many peer-reviewed papers already published that tell us that "arsenic life" is almost certainly nonsense. For example, it's been understood for decades that arsenodiester bonds are vastly less stable than phosphodiesters — by MANY orders of magnitude. We're talking bout relative half-lives of less than a tenth of a second versus more than ten million years.
For example: http://pubs.acs.org/doi/pdf/10.1021/cb2000023
In any case, peer review is not sacrosanct if the data and methods are clearly presented.
The Redfield group's experiments employ straightforward microbiology, molecular biology, and biochemistry -- well-established techniques that many of us already use routinely. The data are there for anyone to evaluate. We're not talking about a conference abstract with wild claims, but a complete manuscript with full data and a reasonably complete description of the experimental methods.
The articles quote scientists with the requisite expertise who have read and evaluated the manuscript. So where'd the problem?
The *failure* here was in the peer review of the original Science paper.
I can see the rebuttal from Wolfe-Simon already, "The Refield group used the wrong growth conditions, media, etc." The argument isn't going to be over with one group providing negative results.
I think you have made the mistake of thinking Arsenic Life was ever alive. It was stillborn. There was no evidence provided in the Science paper of incorporation of arsenic into any biomolecule. There was no real evidence of growth on arsenate. Look at Supplementary Table 1 anc compare it to the description of the data in the text. This misrepresentation of the data is scientific misconduct. Why would growth on 40mM(!!) arsenate give you a growth curve indicating that there was a limiting nutrient? The arsenate was (as should have been expected) contaminated with phosphate. No additional experiments are necessary or warrented. The original paper refuted itself.
@David Sanders, just because 40 mM arsenate shouldn't be limiting, that doesn't mean that some other requirement wasn't limiting. Do you expect the growth curve to go exponentially upward forever? Brian, thank you for writing this story and calling out the main stream media as well as scientific outlets like Nature News who should know better. If people thought that the original NASA press conference was bad form (when it was communicating results that had been peer reveiwed and accepted for publication) how can you justify the trumpeting of these negative findings that have not been vetted in any way? Findings by a researcher who has a lot riding on being correct, given the shrill, vindictive and unprofessional nature of her early criticisms.
Being scooped is no problem in a non-competetive context. Pure science Wiki (http://purescience.wikia.com) is an Internet platform for the pure scientific method without any of the prestige-obsession that makes academia unscientific. The topic pages "How to avoid hoaxes in science", "Advice of ways to stop justifying" and "Advice for cheapening science" are especially relevant. http://purescience.wikia.com