Being in the position now to mentor reviewers—and improve my own reviews—I had a conversation recently about what makes for a useful first-round review. There are a lot of perspective on this, but for me, I appreciate a focus on a few big picture issues for an author to address, as opposed to a plethora of smaller issues…
Science first, journalism later
Being a stats and research design guy, I start by reading the methods and results section of a paper. If the empirics aren’t there, the bar for a revision is high, because I do not think it is fair to ask authors to conduct a new study or collect different data for a revision. But it’s easy to miss things in empirical papers. ‘Fatal flaws’ in a paper are questions of research design and data analysis, not of theoretical framing and argumentation. Journalism is fixable in later rounds. Start with evaluating the scientific claims in the paper given the design and data presented.
Evaluate the usefulness of the research question
This one is tough, because whether a research question is useful (I prefer the term useful over interesting) is very much in the eye of the beholder. What I’m after, particularly for reviews in Journal of Business Venturing, is the usefulness of the question to other entrepreneurship scholars. Entrepreneurship research covers a wide domain. But there must be some centrality of entrepreneurship and an entrepreneurship conversation to a paper. This is where reviewers are very helpful. If three reviewers agree that the paper addresses a meaningful entrepreneurship conversation, even if I don’t agree, I’m likely to side with the reviewers.
Evaluate the argumentation
After the science and connecting the study to an entrepreneurship conversation, help me with the cogency and logical consistency of the argumentation. I’m looking for clarity of the writing, whether the hypotheses flow from the arguments and evidence provided, and whether the results support the conclusions presented. What is particularly helpful is identifying key assumptions made by the paper, or overlooked by the paper, that lead to an interpretational confounding issue. Again, journalism is fixable in later round. But if the paper is making claims or logical leaps that are not tested, or if the paper proposes a theoretical model that is not what was actually tested, these are big issues to identify in the first round.
Save the laundry list for the next round
It’s easy to generate a laundry list of issues, typos, judgement calls, and missing references. Those can be helpful for later stage reviews when you are helping clean the paper up, and after the author addresses the big picture issues. As the saying goes: “Focus on the steak, not the peas.”
Be transparent about what you know and don’t know
Given the specialization and often narrow research questions in entrepreneurship, it is fine to tell the author (and editor) that you do not know much about a topic, area, or method. I would rather have that disclosure than have to address a reviewer’s mistake. Methods comments and empirical suggestions in particular are helpful, but only if they are accurate. What is tough is when a reviewer makes a technical error in a comment about the empirics. Then I need to ‘politely’ redirect the author away from the reviewer’s comment, which is awkward. By the same token, if you are a subject matter expert in a relevant area for the paper, also let the author and editor know.
Now, there are a couple of very important starting assumptions for reviewers to keep in mind before writing the review…
Remember, it’s not YOUR paper
This one may seem commonsensical, but it happens often. The review starts reading like “Listen, I know what you did, but the better way of doing what it is that you did is to do it the way I would do it.” That’s not very helpful, and gets to the journalism and not the science. While there can always be improvements to the journalism, what I need most are insights into the science and argumentation.
AND THE MOST IMPORTANT THING WHEN WRITING A REVIEW IS…
Don’t be an ass. Seriously, if you can’t say anything constructive about the paper, just tell me, and I’ll find another reviewer.
Oh, and timeliness is always appreciated :)