Peer-review research: What you need to know!

There are two things you may have realized about me, the podcast, and the blog is that I have a love-hate relationship with peer-review, but that I also hate reading bullshit advice/opinions. There is so much conflicting and misleading information on the internet now its nearly impossible to discern what to believe without fact checking. Recently this issue came to a head when the International Journal of Sports Physiology and Performance published a study bashing the Look Keo power meter. Being a pay for pub peer-review journal with a somewhat dubious reputation, many of us took the results as presented while raising questions about the results. VeloNews, however, chose to jump on the bash band wagon and reported results from the abstract (BAD MOVE) and then reported that only the SRM and Quarq had previously been tested (FLAT OUT FALSE). All this begs the question, what good is peer-review if your peers don’t spot bad research?

PEER-REVIEW: Imperfect, but better than opinion

I have no problems with people offering opinions. I offer opinions all the time. The problem with opinions is when they are portrayed as fact. Lots of snake-oil salesmen (and women) portray their opinions as expert and evidence-based, when in fact they’re neither. A good example is that pretty much EVERYTHING the Food Babe says is utter bullshit, and more than half of all Dr. Oz advice is equally non-sensical. In contrast, authentic research is peer-reviewed, but the quality of that review has been called into question. On the one hand, good research is being rejected, while shit research is getting published…this can actually happen in the same journal! Moreover, some journals claim to be peer-reviewed, but simply charge a fee and publish your paper; even if its fake. Good god man, what good is peer-review?!

Despite my own run ins with biased reviewers, I will still back peer-review over Fox News opinion bias. If you’re a fan of FoxNews I cannot see why you’re reading this post anyway, so I won’t apologize; Fox News makes up as much crap as Dr. Oz, and even edits video to spin their story. Getting back to the Look Keo paper, peer-review did in fact work. The myth is that peer-review ends with publication, but that’s just the first hurdle. Peer-review is ongoing, and the internet has broadened peer-review to include savvy consumers, as well. For example, well-known blogger DC Rainmaker did his own review of the paper and outlined several of the problems that had been thrown around over the last week. His take home message is that researchers should understand the equipment they’re testing before they evaluate it. It is pretty clear these researchers have little experience with power meters. Moreover, the Journal is not known for publishing high-quality research. Unfortunately, this can leave the consumer with feelings of mistrust in “research”. Here are my tips for evaluating news items:

  • Take responsibility for yourself! We all have a responsibility for shedding our ignorance and educating ourselves first before putting our absolute faith in advice or written material.
  • Track record matters. This one is a low bar to achieve. Just look in the news and you’ll find plenty of stories criticizing the Food Babe, Dr. Oz, and more. If someone is continuously dogged by allegations, their advice probably isn’t worth much. Contrast that with the world’s leading Journal, Nature, or other top journals like the JAMA or the BMJ. These journals have a world-renowned reputation for quality and only accept the best research.
  • Fact check. If someone says their claims are backed by research, just check one or two of those references. Embellished/fraudulent claims often cite papers that have little if anything to do with their claims. A classic example is EP-NO, which claimed to increase red blood cells and performance but cited studies on iron deficiency. If you’re not iron-deficient, an iron supplement won’t help.
  • Be patient and read. Given a little time, the facts can emerge quickly. In the above case, it was about a week before the paper unravelled. Read the forums and comments too.

This is just brief piece of advice on being an educated information consumer. Even with its short-comings, peer-review is still the best system of checks and balances and the internet has created an enormous peer-review system. Stay thirsty (for knowledge) my friend!

Written by