Warren B Powell
Professor Emeritus, Princeton University
I served as Area Editor of Operations Research for eight years (1988-1996). This was a period that taught me a lot about the weaknesses of the standard way of handling papers (send paper to AE, who sends it to referees, and then do what the referees say). It took me two years to realize that this approach just wasn’t working. I came up with an entirely new approach, which I summarized in a memo available here. This web page is a brief summary of the core ideas.
It all begins with three principles:
- Let authors be authors
- Let editors be editors
- Let referees be referees
I then introduced a step that is surprisingly easy to implement, and which fundamentally changes the relationship between the author(s) and the referees:
- Allow the author(s) to respond to the referee reports before the AE makes their decision.
I explain below. I applied the ideas on this webpage for six years while serving as Area Editor of Operations Research, and they work. They are practical, implementable, and improve the process. However, there needs to be a broad awareness and adoption of these ideas, which is my motivation for creating and disseminating this webpage, which can be reached at “https://tinyurl.com/letauthorsbeauthors/”.
Let authors be authors
The responsibility of authors is to write, as clearly as possible, a paper that offers something new. For this reason, the author is in the best position to tell us what is new and significant in their paper.
An idea I was given by Mark Daskin (who adopted it from his colleague Phil Jones) is to include, typically in the next-to-last paragraph of the introductory section, a paragraph that begins with “This paper makes the following contributions.” The rest of this paragraph should be a list that contains what the author feels are publishable contributions. The last paragraph should give the organization of the paper.
Note that a contribution is not a list of what the paper has done. Implicit in each contribution is an implicit claim that it is original and significant. In other words, each contribution should contain a claim that may be verified by knowledgeable reviewers.
My experience with this process quickly revealed that most authors do not know how to write a list of contributions. The key feature of any contribution is that it should have a claim that can be verified by a knowledgeable reviewer. When the contributions did not meet this standard, I would return the paper and request a revision (sometimes I would just ask for a revision in email, that I would then forward to the AE). Often it would take two or three iterations for me to get a list of contributions that I felt I could then send through the review process.
Let editors be editors
First let me emphasize: there is more to editorial work than just picking an AE or reviewers, sending it out, and then repeating whatever the referees tell you to do. When I was active in editorial work, we called these people “mailboxes.”
Given the list of contributions, the first responsibility of any editor or area editor (but especially the associate editor) is to read the list and then make a judgment of the form “If this claimed contribution is true, does this seem to be of sufficient interest for the readers of my journal.”
Once I felt I had a valid list of contributions, I would make an initial assessment of whether the paper appeared to be publishable if all (or at least some of) the claims were true. I expected the AE to also look at the list of contributions and make the same assessment. The paper would then go to reviewers, with a request that part of their report would focus specifically on whether the claimed contributions were original and significant (that is, of publishable interest).
Finally, when the reviews are returned, the editors should first look at the reviewers’ assessment of the claimed contributions. Referee reports should contain a section specifically targeted at an assessment of the validity of the claimed contributions.
Side note: I include the “statement of contribution” paragraph in every one of my papers. It is simply stunning how often the editors just ignore it.
Let referees be referees
With surprising frequency, referees would appoint themselves as authors, and they would decide how they think the paper should be written.
While referees should read the entire paper and feel free to make any comments that they feel would improve the paper, most important should be their assessment of the list of claimed contributions. Even if some contributions are not deemed original or significant, the reviewer may confirm one or more contributions that an editor feels are enough to justify publication. This assessment is key, and should be kept separate from other statements that a reviewer may make.
I often found reports that intermingled minor edits, fixable problems, and fundamental issues that are not fixable within the scope of the paper (a revision would really be a new paper).
Allowing authors to respond before a decision is made
Assume the referee reports have now been submitted. What happens at every journal today is that the associate editor reads the reports, and makes a decision. This process makes the implicit assumption that the referees are experts. When the reports are negative (a single negative report can result in a paper being rejected), the author is immediately at a disadvantage.
The presumption that the referees are more knowledgeable than the author(s) of the paper is deeply flawed. While this may be true in mature fields where a senior reviewer may actually have more expertise than a less experienced author, my experience is that this assumption of expertise is often wrong, especially in fields such as stochastic optimization which is characterized by a wide range of subfields (dynamic programming, reinforcement learning, stochastic programming, stochastic control, stochastic search, simulation-optimization, multi-armed bandit problems, …). I cannot count the number of referee reports I had to deal with from reviewers who clearly did not have the proper background to review a paper in a particular subfield of stochastic optimization.
Even when reviewers do have the proper technical background, papers that require specific domain knowledge, or which are simply more difficult to read (this is a consistent problem with papers on stochastic, dynamic problems) may require more work than a referee is willing to devote to the paper. A reviewer can easily overlook some aspect of a paper, producing a negative report based on a flawed understanding of the research.
A very easy way to fix this is to let the author(s) of a paper read the referee reports and write their own response (possibly a rebuttal) before the AE issues a decision. Then, with both sides represented, the associate editor is in a position to make a decision based on balanced inputs. When the decision is still a rejection, the authors will feel the process has been fairer.
I did exactly this during my last six years as an Area Editor for Operations Research. I did not do it for every paper – typically it was not needed for minor revisions, and papers that were clear rejects. However, it was extremely valuable for papers that were clearly strong submissions where referees had specific objections.
This step took very little work, and would be even easier if an AE could hit a button on the manuscript system that would make the referee reports (properly anonymized) accessible to the authors, with a provision to upload a rebuttal.
Closing thoughts
This approach to reviewing puts the responsibility on authors to identify what is most important about their paper. It simplifies the review process by focusing the attention of editors and referees on what the author feels is most important. This should also help separate the fundamental decision of whether to accept a paper (including some form of revision) versus an outright rejection.
This approach reduces the tremendous amount of noise that currently exists in the review process. It helps reviewers organize their comments in a way that is most helpful to the editors, which is to separate the topics that are most relevant to accept or reject decisions, versus recommendations to authors that are primarily designed to improve the paper.