Saturday, August 22, 2015

Peer Into the World of Academic Publishing: Part II

When last I blogged, I was giving an overview of peer review. This time, I'm back with some examples from previous reviews I've received.

Peer review is a little like grading. Some teachers/professors give really clear, helpful feedback and offer strong explanations for why you earned the grade you did. And, well, some teachers/professors don't.

However, in this case, the stakes are higher, because journals have (often physical) limitations on how many papers they can actually select. Even as more journals become online-only and open source (free to read), there are still usually limitations on how many articles they will include within a given volume (to refer to the publishing year) and issue (the individual copies of the journal published throughout the year - for instance, a monthly journal would have 12 issues).

Peer review helps to keep that field of eligible articles lower. Unfortunately, there is little training for reviewers and reviewers themselves rarely ever get feedback on the reviews they write. So the nonsensical stay nonsensical, the terse stay terse, and the cruel stay cruel. Hopefully this also means that the good stay good.

But this may not be the case. Peer review brings with it few perks. It doesn't pay. It doesn't (always) increase your chance at getting published in a journal. It can be time-consuming and thankless.

Why then do we do it? It is seen as "service" to the profession, something people are strongly encouraged to do. People usually include journals they've reviewed for on their C.V. It can lead to bigger and better things, like being on the editorial board of a journal, which brings many perks. And it can be rewarding in other ways.

Part of my motivation for reviewing is that it feels good to give feedback to others, with the goal of helping them improve. And, though I've received some awful reviews, I've also received some really amazing ones - reviews that weren't always positive, but were always helpful. Reviewing is a way to give back.

Here are my thoughts on peer review - what is is and what it is not:

1. It can be very subjective, but that doesn't necessarily mean reviewers are wrong.

Reviews can be extremely vague and contain language like "I didn't like..." or "X didn't work for me..." It can be frustrating, when you're presenting information about a scientific study, providing detail on aspects of the study so people can objectively critique, to hear mere opinions with no concrete examples. Still, that doesn't mean the reviewer is wrong. For instance:
When I first read this, I was really frustrated. It gave me no guidance on how to write the paper differently. The introduction was written in a [background on a topic, hypothesis on that topic] format, so I didn't understand how a) the intro didn't lead to the hypotheses and b) to rewrite so that the leading was better. It didn't tell me what variables were unclear, or perhaps on the flip side, what variables were clear, so I could focus on the remaining ones. And the other reviewers had no issues with the introduction.

But after I got over my frustration and started reading the paper, I realized that I could use this feedback to improve. It's easy to think that reviewers are overly critical and to ignore their feedback. Actually digging in and getting what you can from it is challenging, but it can be very rewarding. So I reworked the intro, and you know what? The paper was much better as a result. It was accepted, and is now one of my most cited papers.

2. Reviews can range from inspiring to demoralizing, sometimes both in a single review.

My favorite example of this is the highly positive review that ends with "reject":
Once again, it's easy to get annoyed. But keep in mind that not every paper fits a particular journal. You might have written an excellent qualitative paper, but if you submitted it to a journal that only publishes quantitative work, you're likely to get a review just like the one above.

3. It can expose reviewer bias about topics, methods, analysis approaches, and even academic disciplines.

As a mixed methods researcher, I do a lot of work that is either partially or fully qualitative (that is, results not expressed in numbers, but narratives, that use methods like interviews and focus groups). Without fail, I seem to get one reviewer who does not get qualitative research, and does not like qualitative research. Here are some of my favorite examples of such:
You heard it here first, folks: quotes aren't results.
And then there's this great example from Shit My Reviewers Say demonstrating bias against an entire discipline:
4. It can make you laugh or cry - choose to laugh.

I think this is just good advice for life. If this is part of your job, you can't let every bad review get you down. It helps to find some humor in the situation. For instance, as I work on my revision, I use the Word comments feature to show where a change is needed based on a reviewer comment. I occasionally put snarky comments alongside. Obviously those are deleted before I resubmit. But they make me laugh so I keep doing it.

5. Some of the feedback will be helpful and some will be completely ridiculous.

I tried to find my very first peer review and sadly, couldn't locate it. But one comment was so ridiculous, I remember it (maybe not verbatim). The reviewer stated that my study contained serious methodological flaws. Said reviewer could not readily identify these flaws but knew they were there because I "did not confirm my hypothesis."

Okay, to repeat something I tell my Research Methods students again and again: You do not critique the validity of a study based on its findings. You critique based on its methods.

As for the part that implies a good study should confirm its hypothesis - yeah, I'm not even going into what's wrong with that. So I'll just sum up what would ultimately end up being a multi-part blog post rant with some choice phrases: scientific method, type I error, the tentative nature of scientific findings.

6. There will be times, when reading a review, that you will wonder, "Did they even read the **** paper?!"

And you know what, maybe they didn't. Or maybe they skimmed it. Or maybe they did actually read it, and weren't able to absorb this information after one read. Because most of your readers if the paper is published will probably also just read it once and absorb what they can from it, feedback from this kind of reviewer is actually very helpful.

7. It can make you a better writer, if you let it.

When you're writing or talking about something you know a lot about, you sometimes take certain information for granted. Most of the time, when we communicate about a study we're conducting, it is with other people involved in the study. But when you write up your findings for publication, you're addressing a new audience: people who know nothing about what you did and who may know very little about a topic.

Receiving peer reviews help highlight that information you may have taken for granted. Over time, with experience at revising based on reviews, you get better about thinking of what information your reader would want to know. This makes you a better writer, at least in terms of scientific writing.

8. Any given review is just one person's opinion.

So don't let bad reviews get you down. At the end of the day, it's just one person's opinion. My favorite example:
This comment was about one of my most cited papers. I receive emails 1-2 times a month and requests on Research Gate 2-3 times a month asking for a copy. I like to think its contribution is more than just small.

But this goes both ways - any good review is also just one person's opinion. You shouldn't let bad reviews eat away at you. But at the same time, you can't simply ignore bad reviews and listen only to the good ones, because the good ones are opinions too, opinions that might not be shared by very many people. Take what you can from both.

No comments:

Post a Comment