Judge a paper on its usefulness, not its 'theoretical contribution'

· by Brian Anderson · Read in about 5 min · (932 words) ·

I’m banning the phrase ‘theoretical contribution’ in my decision letters and reviews.

In its place, I am going to try to judge a paper based on its usefulness, defined as whether other scholars (including myself), could make use of the paper in future work. Rather than ‘make a theoretical contribution,’ what I’m really interested in is how well the author motivated his or her research question, and did he or she do so in a way that engages and invites other scholars to read, and ideally use, the paper in their own work. It also shifts the author’s burden from jumping through—often imaginary—theoretical contribution hoops to the quality and cogency of the argumentation in the paper, and to the justification for why the paper’s approach is better than alternatives.

In my mind, dropping the phrase ‘theoretical contribution’ and focusing on a paper’s usefulness has three benefits…

It helps separate the signal from the noise

Papers increasingly cover narrow, esoteric topics. The phrase “there is a surprising lack of research in…” is used too often by authors to justify a paper that takes random (not in the statistical sense) variables and puts it them a research model. I also think that the necessity to ‘make a theoretical contribution’ is, potentially, the biggest predictor of the questionable research practices that hamper science and create lots of one-off papers with small effects and results that aren’t replicable.

Placing the paper in the context of usefulness requires an author to justify the benefit of the paper to a specific audience. That last part is important. A theoretical contribution is very much in the eye of the beholder, and reviewers come from different backgrounds and often aren’t well versed in the ‘theoretical conservation’ a paper is trying to join. Speaking to a paper’s usefulness lets authors direct attention to the specific audience he to she is trying to reach, and how that audience can use the paper to inform their work. If the author can’t identify an audience for paper, or that audience is likely very small, then the paper is not likely to be very useful.

Places an emphasis on the author’s argumentation, reasoning, and logic

Having reviewed, well, a lot of papers, and now having written quite a few decision letters, I’ve got a working hypothesis for when reviewers comment that the paper ‘fails to make a theoretical contribution.’ It happens when the writing is bad, the argumentation flawed or circular, the reasoning unclear, the research model esoteric, or the variables trivial. Putting a focus on usefulness clarifies the objective for the author, and ideally makes the paper easier to write. The paper’s purpose is to persuade a particular audience (of real people) that the paper is going to help them do something that they couldn’t do before, answer a question that they couldn’t answer before, or answer a question in a better way than they answered it before. Communicating usefulness requires cogent argumentation and reasoning…and good writing :)

Recasts the usefulness of exploratory papers, theory building papers, theory testing papers, editorials, post-publication commentary, replications, and other paper types

I see the comment a lot that ‘I don’t think the paper is appropriate for X journal’. I get the comment back on my own papers, and it’s frustrating. I don’t mean to imply that journals should not have reasonably defined editorial scopes. But, where it gets problematic as an editor, reviewer, and author, is when the journal is meant to cover a discipline, rather than covering specific types of papers. Consider Journal of Business Venturing, where I am an editor. JBV is the top journal by impact factor in entrepreneurship, and it considers a range of papers that touch on issues related to entrepreneurship and innovation. This is part of JBV’s editorial statement:

The Journal of Business Venturing: A Journal Dedicated to Entrepreneurship provides a scholarly forum for sharing useful and interesting theories, narratives, and interpretations of the antecedents, mechanisms, and/or consequences of entrepreneurship….This multi-disciplinary, multi-functional, and multi-contextual journal aspires to deepen our understanding of the entrepreneurial phenomenon in its myriad of forms.

JBV is a domain journal; so long as a paper addresses an entrepreneurship topic, all other things being equal, the paper could be published in JBV. The role of the editor and reviewers is to make the decision for whether the paper should be published in JBV, which requires a subjective assessment. So how should an editor and reviewers evaluate the useful of, say, a research methods paper? Clearly the paper is not going to make a ‘theoretical contribution’. Nor would necessarily a replication paper, an exploratory paper, an editorial, and so forth. But all of these paper types are still potentially very useful to entrepreneurship researchers. The relevant questions then for the editor and reviewers is a) would an identifiable audience of entrepreneurship researchers be able to use the paper to inform their research; and b) how big is that audience? Now, it also means that the author should specify the target audience in the paper, to make it easier for the editor and reviewers to make a usefulness assessment.

Ultimately, if the paper is poorly argued and poorly motivated, it isn’t likely to be useful. If the paper is so esoteric that only a handful of researchers are likely to get any value from the paper, it’s usefulness is also limited.

We will see how well I’m able to avoid the phrase ‘make a contribution’ when I’m writing reviews and decision letters. But I also think focusing on making a paper useful will make the papers I write more impactful.