March, 2008 posts on blind refereeing

1. Brief Explanation of How Blind Refereeing Works

Academic journals have varying levels of blind-reviewing. The levels enter because the editor first looks at the manuscript, then determines whether to send it to referees (usually two to four), then (should he so decide) sends it to them, at which point they read the manuscript and write a referee's report to the editor. When he gets the report he can either accept the paper for publication, allow the author to revise the paper and submit it again (at which point he may send it back to referees for one more judgment, or just accept or reject the piece), or just reject the piece.

Unfortunately, as far as I can tell, there is no consistent terminology for these. I think the most standard use is the following. Single blind reviewed journals are ones (like The Journal of Philosophy) where the author does not know the identity of referees that write reports to the editor, but where the referees do know the identity of the author. Double blind reviewed journals are ones (like the Australasian Journal of Philosophy) where the editor knows the identity of the author but the referees do not, and as before the writer does not know the identity of the referees. Triple blind reviewed journals are ones (like some Kluwer journals, I think Synthese and Philosophical Studies) where even the editor does not know the identity of an author. With these, a secretary assigns the manuscript a number, and even the editor goes by that. I guess you could have quadruple blind reviewing where the editor doesn't know the identity of the referees, but nobody does that, and it's hard to see the point.

A few years ago there was a brouhaha in the letters section of the Proceedings of the American Philosophical Association where people complained that The Journal of Philosophy could be single blind reviewed and still be considered the best journal in the field. The editor's response was especially embarrassing, given that students take logic classes in philosophy departments. He listed a set of second tier schools where some of their published authors resided, as if that was enough to refute claims of heuristic bias. If any student in a freshman level informal logic class did something like that they would get an F (and I can explain why if anybody is interested), but here you have the editor of the most prestigious journal in philosophy doing so with (I assume) a straight face.

I don't think this is so pernicious on the field, just because it is so uncommon.  A much more pressing problem is the fact that heuristic biases almost certainly lead editors to interpret reviewers' comments more or less charitably depending on the identity of the author. If this is the case, then there is a problem, because the vast majority of journals are double blind reviewed. While this is anecdotal, it is at least a piece of data that I know a number of people from top (and none from second tier) schools who have been given multiple revise and resubmits of the same piece after getting successive negative reviews. In some cases, the editor had previously gotten the writer to referee articles for him, based on a recommendation by members of that writer's dissertation committee. Well, nobody I know from OSU, even those who have published very well, has been given a second revise and resubmit for the same paper. If the editor decides that you can revise and resubmit the paper, then the norm is that he either accepts or rejects the rewritten piece, not give you another chance with another set of referees, if the initial referees still don't like it.

And again, while this is entirely anecdotal, I should note that the first journals I was able to publish in were Kluwer journals where only the secretary knew my identity. Prior to that I had two years of getting mixed referees reports and rejections by editors.

Oh well. What doesn't kill you and all that. . . . There is at least little enough heuristic bias that people from second and third tier schools can get top publications, albeit perhaps we have to work a bit harder (note how easy it is for someone with tenure to discount the effects of what are almost certain rampant heuristic biases; dear Jesus, what have I become?).

In this respect, it would be really nice to have a list of all the philosophy journals that genuinely do triple blind reviewing. I might do this after the rewrite of the book is in. This process has made me pretty interested in the ethics of refereeing. The fact that the author doesn't know the referee's identity presents a Ring of Gyges (if you could be invisible, what would you do?) scenario that can lead to moral atrociousness. I have strong views on what a referee's obligations are and I'll blog on that tomorrow. After Mark and I finish the book, and I finish and get out four co-written papers (and I don't deserve the forgiveness of my co-writers Sal Florio, Jason Megill, Sean Whittington, and the long suffering Mark Silcox in this regard) on which I'm horribly behind, I'm going to write a paper on this topic (tentative title "Three Blind Referees"), so any thoughts would be really helpful. I'll outline the main argument tomorrow.

2. Three Blind Referees

Time is short today, so I'm going to do another part tomorrow on the ethics of blind reviewing. Today I want to just define a few more terms to be able to set out the problem of what kind of reviewer one should be. [And while we are on the topic of journal procedures- also check out a recent post by Andrew Cullison defending open access in philosophy journals; you can read that on his interesting blog here.]

Referees know that the author will not know their identity. This leads to the proliferation of hostile and abusive comments (albeit, some journals, in particular the Australasian Journal of Philosophy, send a note out to all referees stating that if the review is overly negative the editor will not send it on to the authors). And yet despite anonymity, some referees are incredibly helpful to both the editor and the writer.

I propose three archetypes. On one side is the Good Samaritan reviewer. Even though the writer does not know his or her identity, the Good Samaritan goes out of her way to at the very minimum provide detailed advise about how the article could be improved, even in cases where rejection is counseled to the editor. Such referees play a very important role in improving the level of published academic philosophy, and have certainly saved the careers of young Ph.Ds struggling to go from dissertationese to mainstream published philosophy.

On the other side is Professor Angrypants. Professor Angrypants uses the anonymity as a bad person uses Plato’s invisibility granting Ring of Gyges, to take pleasure in terrorizing those who cannot know his or her identity. Rhetorically, Angrypants always presents him or herself as just objectively explaining to the editor the myriad reasons the editor should not publish the article, though the astounding level of misplaced condescention (strangely, almost always redolant of the manner in which twits and gits put down others according ot the assholery standards of the British class system) is a red light for editors in search of this. Unfortunately though, the process of blind refereeing is set up just because (in the vast majority of cases) the editor cannot be a specialist in all of the areas in which the journal in question publishes, and because the editor cannot possibly read all of the articles submitted. Thus, Professor Angrypants can resort to breathtaking sophism and uncharity and get away with it if he or she is rhetorically skilled enough. Every published author has suffered multiple times at Angrypants’ hand. Many professors are surprised when they inadvertantly learn that one or more of their colleagues is actually Professor Angrypants (albeit it is from such incidents that we have ascertained that Angrypants' behavior is almost always the result of overwhelming vanity, unjustified professional dissatisfaction, combined with other forms of jerkiness).

In between the Good Samaritan and Professor Angrypants is Joe Friday, who really does in a straightforward way present “just the facts” to the editor. The paradigm Joe Friday reviewer is very good at telling the editor what claims are being argued for and assessing the extent to which they have been successfully established according to the standards of the journal in question.

The way I’ve set this up, makes it appear that Joe Friday is the Aristotelian mean for which we should all aim. In fact I think this is demonstrably false. We should all aim to be Good Samaritans. Since this suggestion (not framed in this terminology) enraged a number of people when I made it on Leiter Reports, I’ll set out the defense tomorrow, when I have more time.

3. In praise of Good Samaritans

Here I want to defend the claim that all blind reviewers should try to be Good Samaritans. It is perhaps not so strange that this claim strikes some as common sense and provokes a great deal of defensive hostility in others.

To make my case, I want you to first imagine somebody defending the Joe Friday approach as striking them as the right Aristotelian mean between two vices, in the sense that courage is the mid-point between cowardice and foolhardiness. Such a defense does not get very far on its own though, because the Good Samaritan does not accept the view that Good Samaritanism is a vice. So Joe Friday will have to offer other reasons. For example, Joe could argue that he can do more reviews if he doesn't try to help the writer and thus contribute more to the greater good. But the Good Samaritan can respond by saying that by charitably construing the writer's claims and arguments and then giving the writer detailed advice about how to recraft the paper (or book) so that it achieves these goals he or she plays a crucial role in: (1) helping grad students mired in dissertationese learn to write published papers, (2) helping everyone's published papers be the strongest they possibly can. If everyone was a Joe Friday, the level of published philosophy would be much worse and many good philosophers would not develop their writing skills well enough to get tenure. Joe Friday will have a response, but like all such debates the participants will not convince one another.

Here's a much deeper reason why all reviewers should consciously try to be Good Samaritans. Professor Angrypants always thinks that he is really Joe Friday. Almost nobody consciously thinks, "I am going to be hostile, abusive, and uncharitable." Instead, they think that they are just presenting the facts to the editor. And yet the number of reviewers that are hostile, abusive, and uncharitable are legion.

There is a lot of cognitive science on this very issue, the ethical consequences of which are explored in the excellent papers by my ex-colleague Patrick Boleyn-Fitzgerald. In brief, anger really is a "short madness." When people are angry they systematically lose the ability to determine when anger is justified. Due to the similarity to self referential paradoxes ("this sentence is false") Boleyn-Fitzgerald appropriately calls this "the paradox of anger." Thus the kind of misdirected animosity characteristic of Professor Angrypants is such that he will be much more likely to see his behavior as not the result of animosity or even particularly angry.

Now let us return to Aristotle, who very sensibly said that the virtuous person will often need to shoot for an opposite vice than the one to which he is prone. So if I tend to be too much of a Puritanical tea-totaler I should shoot for more drunkenness in order to achieve true temperance (and vice versa if I tend to be a drunk). Without admitting that Good Samaritanism is a vice, this insight can be used against the defender of Joe Friday. Everybody suffers from the short madness of anger and as a result does irrational and destructive things, but research (cited by Boleyn-Fitzgerald) shows that if one prepares oneself ahead of time by noting that people are going to do irritating things (i.e. submit badly prepared papers) and tells oneself that one's job is to help people, then the irritating things do not provoke anger. So if one rationally wanted to be a Joe Friday (and not a Professor Angrypants who thinks he is being Joe Friday), then one should consciously strive to be a Good Samaritan. So we must all try to be Good Samaritans.

Fitzgerald uses this paradox to argue that anger is never justified. The only way one could have a justification (in the epistemic sense) that one's anger was justified is if one was not angry. The above reasoning is similar.

I don't think this argument will change anything, even if I get off my butt and write it up as an actual paper. There are too many Professor Angrypants in our field. Most of them are extraordinarily bitter for personal reasons (such as thinking they should be at a better ranked institution or that their articles should be cited more) and take it out on writers. Unfortunately, almost none of them realize they are doing this, and they get very defensive when confronted with it. Like all evil and abusive people, they see themselves as good guys doing the right thing. Moreover, given the nature of academic philosophy it will never be possible for the most conscientious editor to always catch Angrypants.

One thing that could change though is that journals and presses could explicitly tell reviewers that they expect them to be Good Samaritans even when counseling rejection. Of all the journals I've reviewed for and published in perhaps Australasian comes the closest to already doing this. And clearly Mind has been the worst in this regard; about five years the editor publicly rationalized the shoddy paperwork that leads them to keep papers for over a year by saying it's not Mind's job to help young scholars get tenure (long turnaround times are murderous when you are under the publish or perish gun). This is the classic Angrypants claim that he is just being Joe Friday. But if I am right that we need to all try to be Good Samaritans, then it is the job of Mind to try to help people get tenure. Even rejected authors should get helpful non-abusive comments in a timely fashion.

Comments

[Mark Silcox]

Actually, I did get one revise and resubmit (from the Canadian Journal DIALOGUE) where, when I rewrote the paper, one of the original referees decided that the rewrite was still not good enough. The journal editor decided to ignore this person's (admittedly hugely crank-ish) protestations and publish the paper anyway. Pretty decent treatment for a mere OSU grad, I reckon. Hardly the norm, though.

[Jon]

I don't think that's *too* uncommon. While you are right that the way the editor responds to inconsistent reviewer's comments is probably the primary manifestation of heuristic bias towards or against the writer, in the case you describe it does sound like the editor just thought one of the reviewers was being a crank (and it sounds like he/she was).

I think what happened with your paper is a lot more common than the journal editor letting you make a second set of wholesale revisions and then sending it out again to a new group of reviewers. For doing this necessitates another round of work for the editor. Again though, any experience we bring to bear will be anectodal.