Wednesday, November 15, 2006

Peer review sucks?

Denyse O'Leary, the ever championing ID journalist, has been criticizing peer review in a series of articles. I would like to give some comments regarding one of these.

Writes she:
Peer review problems went "public" mainly as a result of recent high-profile scandals like Science's peer-reviewed Korean stem cell research paper that turned out be fraudulent.
Some argue that the peer review system was designed to detect incompetence but not fraud. Flawed, yes, but fraud, no.

Yep. It's true. Published research can be both fradulent and flawed. There is a certain level of honesty and competency expected and it is not always met. The question, then is, what do we do about it? You can't really just moan about something unless you have a better alternative. Denyse offers two:

1) Interestingly, the ID journal, Progress in Complexity, Information, and Design (PCID) has opted to return to the early twentieth century approach, where a senior scholar recommends a junior scholar for publication. Time will tell if this old method can be revived successfully.
I totally fail to see why this would solve any problems. Instead of having several peer reviewers, you now have one. Why would this person be any less likely to be fooled by frauds and incompetents?

2) Internet-based technologies may enable a more open and dynamic system. In a way, it can be compared to the blogosphere. The blogosphere, for all its faults, has been a breath of fresh air in media. It has restored the original concept of news as what people want to hear about rather than what gatekeepers think they should want to hear about.

I actually think that this is an idea that might have some merit, but not in the incarnation that Denyse imagines. Her version would essentially turn published research into Wikipedia where anyone can write whatever they want about anything. Yes, the "winning" concept would be the one where people get what they want to hear about. So, are we then really supposed to let the whims of public opinion decide what is valid research. I'm sure that IDers, homeopaths and astrologers would love it. Denyse, there has to be some "gate keeping" done.

So, two alternatives are offerered and none of them seem to offer any improvements. Oh, well. Back to the drawing board.

I just want to comment about one thing more that Denyse writes in her article:

Soothing comparisons have frequently been made to Winston Churchill's characterization of democracy as the worst system - except for all the others. But the convenient analogy to democracy fails. In the first place, the secrecy in which peer review operates make it a poor analogue to democracy. Second, democracy aims primarily to give every citizen a vote. The fact that some citizens vote for cranks or criminals does not mean that democracy has failed.
I suspect that Denyse has misunderstood why this so-called comparison has been made. Peer review is not analogous to democracy as she implies that scientists see it. The point of the "comparison" is just to point out that out of a variety of bad choices, peer review is the best (although this is debatable). All she has done here is to build and attack a strawman.


Post a Comment

<< Home