UPDATE: I’ve just added a new poll for this. Do vote away. :)
A few weeks ago, a colleague and I were discussing whether when you are refereeing a paper for a journal, you should take into account the journal’s reputation, editorial policies, etc., when you are giving your verdict regarding the paper. For lack of a better term, should you be a journal-relative referee or a journal-invariant referee? To make the issue more concrete, consider the following cases:
Case 1: Suppose that you are refereeing paper, A, for journal, X. X is one of the best journals in the field. A is a very good paper, but you personally think that A belongs in a lower-ranked journal, and not in X. Should you recommend that the paper not be accepted by X?
Case 2: Suppose that you are refereeing paper, B, for journal, Y. Y is one of the lower-ranked journals in the field. B is not a very good paper, but you personally think that B is good enough for Y. Should you recommend that the paper be accepted by Y?
Case 3: Suppose that you are refereeing paper, C, for journal, Z. Z is one of the best journals in the field. Z has a policy against publishing discussion notes from other journals. You think that C is worth publishing, but C seems to you to be a discussion note of another paper that recently appeared in another top-ranked journal, W. Should you recommend that Z not accept C by telling Z that you think that C is really a discussion note of a paper in another journal?
My colleague and I weren’t sure whether one should take into account a journal’s reputation, editorial policies, etc., when one is refereeing a paper. On reflection, I’m inclined to think that one should be a journal-invariantist, for two reasons.
First, it seems that a referee’s job is to give an editor his opinion as an expert in the field regarding the academic quality of a paper. It’s the editor’s, and not the referee’s, job to enforce editorial policies, maintain the journal’s reputations, etc.
Second, suppose that you are the editor of Y in Case 2. Imagine that you learn that your referees are recommending that you accept papers, not because they think the papers are any good, but because they think that your journal is not very good. How would you feel about that?
What do people think?