Questions about the peer review process and the NASA drama

With school being out for the holiday break I can finally dust off the blog and write something original. Lately I’ve been thinking about the peer review process and it’s role in advancing scientific thought. These musings have come from two sources: a conversation with my geomorphology professor and the NASA debacle.

A few weeks ago, I came across a journal article from a planetary science journal that I wanted to read. Unfortunately, my school didn’t have a subscription to the journal and I couldn’t get beyond the abstract unless I wanted to fork over 20$. So, I asked my professor if there was another way of acquiring the article. He politely informed me that there wasn’t (aside from paying the fee) and then he talked about publishing his work on an open source site because he couldn’t get past the peer review.

I didn’t bother to ask why he had a hard time getting published. Maybe I should have. Was he trying to publish questionable material? Is he considered a science quack in the geomorphology field? Or are there other factors in the peer review process that extend beyond methodology? My professor didn’t strike me as a crank and he espoused the scientific method quite frequently. So, I ask the that question with sincerity. As an undergrad, my understanding of such things is fairly limited.

I know that the peer review process is meant to prevent pseudoscience  from being published as science. It basically insures the integrity of that discipline and the scientists involved. However, scientists and the editors at these journals are human. I have a hard time believing that they aren’t prone to being biased towards what and whom get published regardless of the validity of the science.

The second part of this post comes from the NASA drama. Just to be clear, this has nothing to do with the veracity of their findings nor with the procedure that the scientists followed. I am not a biologist and I don’t have a background that allows me to weigh in on the subject. What interests me is how science bloggers reacted when told that NASA would only address critiques in a peer reviewed setting. They basically gave bloggers the finger and said their opinions weren’t relevant.

Now, I don’t agree with how NASA handled the issue. I feel like they should have addressed the concerns in a public environment since that is how they chose to announce their findings. However, I feel like some science bloggers wanted to have their cake and eat it, too. As scientists, we’re quick to discredit anything that hasn’t been published in a peer reviewed journal. It’s the standard by which we judge someones work. However, when told our opinions aren’t valid because they’re not from a  peer reviewed source, drama follows.

So, I come to the geoblogosphere (and anyone else reading this) with these two questions: How often does the peer review process become influenced by personal biases and at what point does it inhibit science and not advance it?

Advertisements

10 thoughts on “Questions about the peer review process and the NASA drama

  1. I have yet to enter the peer-review process, though will be doing so shortly, so I don’t have good insight on it yet. From what I’ve seen, though, it’s invariably influenced by personal biases – what reviewers think is relevant and new (and efficiently written).

    I want to comment on your statement that scientists discredit anything not published in a peer-reviewed journal. I would say that I am highly skeptical of *anything* that I don’t feel has adequate proof, regardless of the publication. I have read some terrible, peer-reviewed journal articles that didn’t make a good case for the conclusions. I have also read some good papers in the “gray literature,” which is not necessarily peer reviewed. This category is often, in the remediation field, written by practitioners who know their stuff but either aren’t feeling up to the puplishable-paper level or don’t have the time. I don’t think that makes their work of poorer quality than that of the ivory tower, but like anyone else, the results have to be backed up with data.

    • Thanks for the feedback!

      I will fully admit that my comment about scientists discrediting anything not peer reviewed was a poor choice of words. Obviously, science journals aren’t perfect and some real doozies will get published for whatever reason. I guess most of this post stems directly from my inexperience with the science field. A lot of the blogs I read and the podcasts I listen to kind of deify the peer review process and I’m too jaded to believe it to be a perfect, altruistic process.

  2. The peer review process isn’t perfect, but it’s what we got. As you progress in your career and start both publishing your own work and reviewing others’ work you quickly realize that the system is only as good as the amount of work people put into it.

    It’s A LOT of work to do a good, thorough review. And even then an individual may not catch a significant error/assumption, which is why multiple reviewers are selected.

    To be blunt, yes, personalities can come into play in the peer-review process. Like anything else, humans will behave like humans. I think the set-up minimizes this but can’t eliminate it. In terms of how often it happens, I can’t say. That would be extremely difficult to measure. The vast majority of my peers w/in my field are professional about reviewing.

    I guess the bottom line with all this is that it’ll be people like yourself, those earlier in their career, that will put the work into improving the system.

    Regarding the NASA stuff, Ed Yong has a good post with a very active comment thread that you should check out: http://blogs.discovermagazine.com/notrocketscience/2010/12/10/arsenic-bacteria-a-post-mortem-a-review-and-some-navel-gazing/

    • Do you think as more professional scientists take to blogging that the peer review process will change also? I ask that because it seems some scientists don’t see blogging as an appropriate venue for academic debates. However, the NASA debacle is showing that real constructive criticism by professionals can take place in a public sphere. If this is, indeed, how the peer review process works, are we witnessing a significant change in how science will be debated in the future?

      • I do think there will be more interaction online. Whether we call it ‘blogging’ or not I don’t know. That term is horrendously vague and carries a lot of baggage. My prediction is that over the course of the next 5 years there will be more online comment/reply of papers.

        Some clever person/group will come up with a way to collect/aggregate commentary of a specific paper into a single location (perhaps using the DOI like researchblogging.org does). Right now it’s difficult to impossible to know where every comment is and then, imagine if you are the author, trying to manage each and every thread.

        I don’t blame NASA for not engaging because then they’d get accused of only engaging on one blog but not another. Somehow the free-wheelin’ decentralized aspect of the web needs to be summarized/synthesized so that substantive comments/reviews are recognized. I have no idea how this will work, but that’s what will likely happen in this decade.

        Just as it’s commonplace to travel around the world to attend conferences (something probably scoffed at by Victorian-era scientists), it will someday be commonplace to interact w/ peers online. And it won’t replace the other tools, it will be in addition to.

        This is all just my speculation of course.

      • You’re right in saying that the term “blogging” has a lot of baggage. I think a lot of that stems from the open nature that blogging presents and the subsequent crap that tends to follow. As it applies to science though, maybe it’ll give scientists a chance to publish that which didn’t make it past the peer review for whatever reason. At the very least it’ll bring some transparency to the process and a larger group of scientists can debate the merit of the findings instead of a small group making that decision. And that’s not to say that blogs will replace the peer review or other safeguards against bad science, but it’ll definitely increase the discourse.

  3. I wanted to mention that sometimes the peer review process leaves out people with good ideas who are well known in their field – I know of one case in particular. That person had a hard time getting things published in peer reviewed literature (is published in many field guides and some peer reviewed lit), but later – after review – his ideas would show up in the articles of some of the others leading the field. He finally went on to do something else, at least in part. And I don’t really know a way around that, or why in particular it happened, other than that some of the egos involved were quite large.

    • I can’t help but wonder if that’s the issue with my geomorphology instructor. As I said in my post, I didn’t ask why he couldn’t get past the peer review so I’m not going to speculate any possible reasons. I just wonder how many times conflicting ego’s have been behind someone not getting their work published.

  4. So this is a really interesting question. I have only had incredibly helpful experiences with the peer review process. I think a lot of this is because I worked in an area few people had, and by the time my advisor got done bleeding on my paper and I made the revisions reviewers were cake.
    But, I have witnessed some seemingly personal vendetta types of behavior in some instances and that is not pretty. This is particularly true in very competitive fields like Early Earth and Astrobiology. My friend was verbally attacked by a critic at an alumni reception at GSA. It was shocking.
    I have overheard faculty members discuss suggesting that only certain people should review their work either because 1) someone has been known for stealing ideas/data after rejecting a paper (this also happened to a friend of mine), 2)they have a personal beef with another scientist and think the review will be about the argument and not the data, and 3)they want people who will agree with their findings. All of this boils down to ego and as we are all human, I don’t know how one eliminates ego from the equation.

    • Unfortunately I don’t think one can eliminate ego from the equation either. As Brian Romans said in one of the previous comments, it’s what we’ve got. With that being said I’m sure the peer review process has been greatly helpful in weeding out some of the more questionable material from publication. But as with any human endeavor there’s sure to be some unhealthy, ego driven competition.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s