Print This Post Print This Post

UPDATE: I’ve just added a new poll for this. Do vote away. :)

A few weeks ago, a colleague and I were discussing whether when you are refereeing a paper for a journal, you should take into account the journal’s reputation, editorial policies, etc., when you are giving your verdict regarding the paper. For lack of a better term, should you be a journal-relative referee or a journal-invariant referee? To make the issue more concrete, consider the following cases:

Case 1: Suppose that you are refereeing paper, A, for journal, X. X is one of the best journals in the field. A is a very good paper, but you personally think that A belongs in a lower-ranked journal, and not in X. Should you recommend that the paper not be accepted by X?

Case 2: Suppose that you are refereeing paper, B, for journal, Y. Y is one of the lower-ranked journals in the field. B is not a very good paper, but you personally think that B is good enough for Y. Should you recommend that the paper be accepted by Y?

Case 3: Suppose that you are refereeing paper, C, for journal, Z. Z is one of the best journals in the field. Z has a policy against publishing discussion notes from other journals. You think that C is worth publishing, but C seems to you to be a discussion note of another paper that recently appeared in another top-ranked journal, W. Should you recommend that Z not accept C by telling Z that you think that C is really a discussion note of a paper in another journal?

My colleague and I weren’t sure whether one should take into account a journal’s reputation, editorial policies, etc., when one is refereeing a paper. On reflection, I’m inclined to think that one should be a journal-invariantist, for two reasons.

First, it seems that a referee’s job is to give an editor his opinion as an expert in the field regarding the academic quality of a paper. It’s the editor’s, and not the referee’s, job to enforce editorial policies, maintain the journal’s reputations, etc.

Second, suppose that you are the editor of Y in Case 2. Imagine that you learn that your referees are recommending that you accept papers, not because they think the papers are any good, but because they think that your journal is not very good. How would you feel about that?

What do people think?

In deciding whether to recommend acceptance or rejection of a paper for a journal, should one base one's decision in part on the reputation, editorial policies, etc. of the journal?

View Results

Loading ... Loading ...

Comments

  1. 1. Posted by Richard Chappell | March 22, 2009 4:39 am

    Given a forced choice, journal relative refereeing seems more helpful. But is there a reason why you couldn’t just say, e.g. to the editor of journal X, “I think A is a very good paper, but not quite up to the standards of X”, and let the editor make of this what he will?

    Case Two may be more awkward, but even there it seems it would be most helpful to let the editor know that the paper in question is on a par with (or better than) those previously published by the journal — although you have serious reservations about the paper for reasons XYZ.

  2. 2. Posted by S. Matthew Liao | March 22, 2009 3:54 pm

    Thanks for your comments, Richard. I should qualify that I don’t have a firm view about this matter. So I am happy to be persuaded either way. But let me just press a bit more my worry about journal-relative refereeing. Following an observation you’ve made, it seems that assessing the ‘standards’ of a journal could be a very subjective thing. One person might have a very inflated view of the standards of a journal and pretty much rejects everything, while another person might have a very permissive view regarding the standards of the same journal and pretty much accepts everything. The practical upshot is that the same paper could be rejected by the first referee but accepted by the second. It seems that we can count on referees to be people who are approximately knowledgeable in their field, and so we can count on them to say that this or that argument of a paper in their field is or is not orginal/good/interesting/etc. But can we count on them to apply any sort of non-biased, objective criteria to assessing the standards of a journal?

    In other words, when you say to the editor of a good journal that “I think A is a very good paper, but not quite up to the standards of X”, I think pretty much you are asking for the paper to be rejected. So out of fairness to the authors, I guess I’m wondering whether all of us are equipped to apply similar criteria as one another in assessing the standards of a particular journal. If not, shouldn’t we be more invariantist in our refereeing?

  3. 3. Posted by Richard Chappell | March 22, 2009 5:01 pm

    If we can rely on referees to appropriately evaluate whether a paper is absolutely “original/good/interesting/etc”, then we can also rely on them to evaluate whether a paper is more or less “original/good/interesting/etc” than most others that have recently been published by this particular journal.

    Indeed, the latter sort of judgment is more reliable, since it removes any arbitrariness in where one personally places the “bar” for originality/goodness/etc. That is, we might trust referees to assess the qualities of papers, without trusting them so much on the further question of what constitutes publishable quality. (Indeed, I strongly doubt that there is any one answer to that question. But then, I’m drawn to ‘scalar’ over ‘deontic’ views of normativity…)

  4. 4. Posted by S. Matthew Liao | March 22, 2009 7:40 pm

    Richard said:

    If we can rely on referees to appropriately evaluate whether a paper is absolutely “original/good/interesting/etc”, then we can also rely on them to evaluate whether a paper is more or less “original/good/interesting/etc” than most others that have recently been published by this particular journal.

    I don’t see why this follows. It assumes a) that every referee has read all the papers recently published by a particular journal; b) that they all came to the same judgments about how good they are; c) that ‘recent’ papers are a good indicator of the ‘standards’ of a journal, d) that this information is sufficient to form a reliable judgment about the ‘standards’ of a journal, etc. Take d), for example, unlike editors, referees do not have access to the full range of papers that have been submitted to the journal, both the ones that have been accepted and the ones that have been rejected. Given this, it seems that relying on referees for their judgments of this sort introduces too much subjectivity and variability into the process.

  5. 5. Posted by Richard Chappell | March 22, 2009 7:53 pm

    Right, I’m assuming that if a paper is of higher quality than most others recently published in a journal, then it’s helpful to recommend it to the editor of said journal on this basis. This seems plausible to me. (It certainly seems more helpful than just making up your own standards and applying them across the board.) But it’d be interesting to hear from some actual journal editors about what they would prefer.

  6. 6. Posted by Brian Weatherson | March 22, 2009 8:18 pm

    As someone who has edited journals of very different standards, I much prefer when referees use journal relative approaches.

    In practice, I think there’s another type of case that is pretty frequent and pretty important. A top rank general philosophy journal gets a paper that’s good, but long, and on a question of relatively narrow interest. Should this journal publish it? Probably not, especially if the paper is on an area where there are specialist journals. But the paper may well be appropriate for, say, Linguistics and Philosophy, or Kant Studien, or BJPS. That calls for journal-relative refereeing.

    And I do think this is something that the referees, and not just the editors, should have something to say about. As an editor, I have some sense of what questions about philosophy of physics, or Kant scholarship, are of general interest. But I need to have some guidance from specialists as well. If some debate that looks like a small in-house debate is actually a central question that everyone in the field is talking about, that makes the paper more interesting to an editor, and might overcome my initial judgment that the paper is best suited for a more specialist journal.

  7. 7. Posted by S. Matthew Liao | March 22, 2009 8:52 pm

    Thanks, Brian. It’s really helpful to have an editor’s perspective on this matter.

    Just some follow ups. I can see how a specialist might say something like “This debate is really important for other fields. So this paper should be published,” and I can see how this could be helpful to editors (although I would have thought that it’s the author’s job to communicate something like this in a paper). Suppose we grant this. Would a specialist really also say to an editor that his/her area of expertise is of interest only to scholars in his/her field, and therefore a paper should not be accepted? For example, would a Kantian scholar really say to an editor that “This paper is only of interest to Kant scholars. So don’t publish this paper in the journal.” In your experience as an editor, have you seen this type of comments from referees?

    Also, suppose that some journal relative refereeing should be encouraged. Should we encourage it across the board? At the moment, journals/the profession do not seem to give us much guidance on this matter and this may introduce too much subjectivity into the refereeing process. As a start, some sort of transparent list of the types of journal-relative considerations a journal would want, the merits of which can be debated in the open, would seem to be quite helpful for the editors, the referees and the authors. What do you think?

  8. 8. Posted by Doug Portmore | March 22, 2009 11:07 pm

    Matthew wrote:

    For example, would a Kantian scholar really say to an editor that “This paper is only of interest to Kant scholars. So don’t publish this paper in the journal.” In your experience as an editor, have you seen this type of comments from referees?

    As a referee, I’ve made this sort of assessment — only it was a paper on consequentialism, not Kant. It was a very good paper, but not of wide enough interest to merit publication in a journal like Ethics or Nous. Some papers on consequentialism are of wide interest (e.g., papers offering a new and important objection or defense of act-consequentialism) and others are of relatively narrow interest (e.g., papers offering a critique of a particular aspect of Hooker’s formulation of rule-consequentialism). The latter might be perfectly appropriate for Utilitas, but not for Ethics or Phil Review.

    I’ve also gotten back reports on my own submissions that said that the paper was fine for what it was, but that it wasn’t quite as sharp and imaginative as other articles that have appeared in that journal. That a good paper should be rejected because it is not as sharp and imaginative as other papers in that journal seems like a perfectly sensible judgment to make. Moreover, it seems to be the sort of judgment that editors want referees to make.

    I don’t understand your worry about introducing “too much subjectivity into the refereeing process.” Aren’t editors asking referees (people whose opinions they presumably value) to render their subjective opinions as to whether they should publish various submissions in their journal? Isn’t it the referee’s job to render such a subjective opinion? Or am I to assess only objective things, such as whether this or that argument is deductively valid?

    It seems to me journal-relative refereeing is the only way to go. Regarding your reasons to go with journal-invariant refereeing, I think that you misidentify what the referee’s job is in stating your first reason. The referee’s job is to recommend whether or not the paper should be published in the journal that he or she is refereeing it for. This involves an assessment of the paper’s quality, but also an assessment of its suitability for publication in the particular journal in question, which includes an assessment of whether it would be of interest to the typical reader of that journal and whether it is as good as the typical article appearing in that journal. Regarding your second reason, I think that a more typical case is where the paper is pretty good, but where the point it makes is a relatively minor one as compared to those made in papers typically published in the top journals.

  9. 9. Posted by S. Matthew Liao | March 23, 2009 12:13 am

    Doug, many thanks for your comments. I’ve said alot already, and I do want to give other people a chance to air their views, so let me just make two brief comments.

    You said:

    That a good paper should be rejected because it is not as sharp and imaginative as other papers in that journal seems like a perfectly sensible judgment to make.

    This example is ambiguous. There seems to be an invariant factor here, namely, imaginativeness or what I called originality (I understand that imaginativeness and originality are in some senses relative, but it seems ok to regard them as invariant for our purpose), and a relative factor, namely, how good the paper is compared to other papers in journals. I of course think that the former is a valid sort of judgment for referees to make. I’m questioning the latter sort of judgment.

    You said:

    Aren’t editors asking referees (people whose opinions they presumably value) to render their subjective opinions as to whether they should publish various submissions in their journal? Isn’t it the referee’s job to render such a subjective opinion?

    I think it’s important to distinguish between subjective judgments about the quality of a paper and subjective judgments about whether a paper meets the ‘standards’ of a journal. I understand that biases infect all of our judgments to some degree, but as I said earlier, owing to a referee’s knowledge of a particular field, I think it’s ok to rely on their subjective judgments about the quality of a paper. Do you really think that all of us would apply similar critieria regarding the ‘standards’ of a particular journal? I, for one, am pessimistic about this prospect. And, if we aren’t all applying the same criteria, I’m not sure if journal editors *should* be relying on us for these type of judgments.

  10. 10. Posted by Doug Portmore | March 23, 2009 12:44 am

    Hi Matthew,

    You wrote:

    Do you really think that all of us would apply similar critieria regarding the ’standards’ of a particular journal?

    Similar, yes. The same, no. But I also don’t think that all of us would apply the same criteria in assessing the quality of a given paper. Regardless, I think that if an editor asks me for my judgment as to whether he or she should publish the submission in his or her journal, and if how well the paper meets the particular standards of his or her journal is relevant to this judgment (as it clearly is), then I should give it.

    Note that many times I’m asked something like: “Does this paper merit publication in Journal X?” They don’t ask whether it merits publication in some journal or other. And many times I’m told that Journal X generally accept n% of those submissions that are sent on to referees or I’m told that journal X only publishes articles that are judged to be of lasting value and that I’m to keep this in mind when rendering my recommendation. If they just wanted me to judge the quality in some non-relative sense, as you suggest, then why wouldn’t they just ask me to assign it a grade. In that case, the editor could then just decide to publish papers that receive a certain grade or better?

  11. 11. Posted by Antti Kauppinen | March 24, 2009 5:32 pm

    Interesting discussion. I’ve definitely used journal-relative criteria in my own refereeing, but not without some soul-searching. I would much prefer an editor to say that out loud if that’s what she wants, and, even better, give some guidance on what the journal-relative criteria are (which would alleviate some of Matthew’s concerns about subjectivity).

    But the grading idea is intriguing. I suppose most departments have fairly detailed criteria for assigning grades to student essays. We at St Andrews certainly do, and I find them quite useful, for example in preventing grade inflation, though there is always plenty of room (and need) for judgement. So what if all or most journals agreed to, say, a 20-point grading system with pretty detailed criteria for what it takes for a paper to get a certain grade? (I know this is extremely unlikely, but let’s examine the idea.) Perhaps you’d need, on average, 18 points to get accepted for Ethics. (This could even be public information, updated each year.) Suppose I send them a paper and get a 16 without a revise and resubmit. I’d still be frustrated, of course, but I’d have a good sense of just how close I got. More importantly, I could then look up a journal a few notches down, and submit the paper there after revisions.

    In short, a journal-independent grading system would give a lot of useful feedback to the author, and perhaps introduce a measure of discipline into the refereeing process, insofar as verdicts would have to be justified by reference to agreed-upon if inevitably loose criteria.

Post a comment

Name: (required)

Email Address: (required) (will not be published)

URL:

Comments:

(Spamcheck Enabled)

This work is licensed under a Attribution-NonCommercial-NoDerivs 3.0.