As far as I can tell, I was asked to do this critique because I was cited in the paper. I imagine that it's different in a field like history where citations are everything, but in my field people are not so thorough about the literature: the literature review simply has to include a few relevant papers, rather than being comprehensive. As I've noticed in my (overly thorough grad school) lit reviews, people often cite a source which gives a broad array of other sources rather than citing all sources individually.
This paper cited me in three different places, which was very flattering until I saw that the cites were concerned with something other than the central purpose of my paper. People don't pay enormous attention to the literature, as far as I have noticed, so only the author would bother to make any note of who should be cited where. In two places, they cited me because of my literature review, rather than citing the original authors; in that case, I just referred them to the one major researcher in the area who has many papers with many different coauthors, figuring that's something that someone else would know. In the third place, they cited me in a place which didn't make any sense, making me wonder if they had read any part of my paper besides the literature review. In this case, I really wanted to say that the implication of my paper would be for them to write a totally new paper, rather than the paper that they had written, but I kept my mouth shut. Naturally, I had a small conflict of interest: I can use all the cites that I can get, even if they are inappropriate. If they like my literature review, please go ahead!
I've reviewed two papers so far, and each time, my reaction has been the same. "This is on a topic that I really don't know anything about it," I exclaim to myself. "Someone else could deal much better than I could with all the intricate details, I'm sure. I'll probably miss a lot." And so I start reading it, and feel a bit intimidated. As I went on, though, I discovered (for both of the papers; I am sure it's not typical) that I could not only understand what the paper was doing, but also how terrible it was. It truly did not add anything at all. The first paper as much as admitted it with its title, which was something along the lines of "Some thoughts about [An Intricate But Somewhat General Subject]" and it lived up to the title: it was certainly some thoughts. They would have been very good course lecture notes. I actually wrote that in the review. Now I wonder if that was too mean.
In this case, the paper was poorly written, and I waded through quite a lot of the paper before I figured out what they were trying to say: the paper could have easily been a few pages shorter without having less content. Even after I waded through the paper and read it a few times, I wasn't sure of either its purpose or conclusions. I think the conclusion was that researchers should think more. My review ended up being several pages long, about 25% of the length of the original paper, of almost solid criticism. I did give a window of hope about how they could rework the paper, and I hope that they do, but they have dauntingly far to go.
An excerpt from my review:
The paper needs to be more clearly focused and structured: stating the questions of interest, their practical implications, how they will be answered, and then giving the results and explaining away alternative explanations.
It occurs to me now that that's the definition of a paper. I'm sure the draft that I am working on right now could use some of that too.
I know a professor who says that he takes about 15 minutes to review a paper and is much more generous that he used to be in his recommendations. I think that's probably a good approach: 15 minutes would be way too little for me, but to jump to the very heart of the paper. I spent a lot of my review nibbling around the edges looking at things like the lit review. Looking at the heart of the paper before I've looked at the tangential parts is a little scary. I can stand on firm ground to say that they should cite A and not B, since that's easy to establish, and also whether they are following basic conventions of writing like telling the reader what you're trying to do. It's even not so difficult to say whether their claims are supported by their evidence, and if they left out any logical steps; in this case, I had the (hopefully rare) situation where they gave lots of evidence but didn't reach any conclusions at all.
It's much harder to look at the central purpose of a project that someone has spent months on and figure out if the project makes sense and if they met their goal. In my mind, that's the purpose of a professor or academic advisor.
If you're doing a review in 15 minutes, that's the issue that you look at, and it's the most important question of the whole review. It's not that I'm intimidated to answer the question, but rather I feel like I might be off the cuff or missing something. Emotionally, it's a difficult statement to say that someone has just wasted their time. Especially if there is nothing to salvage.
So both times, I've read the paper over and over, convinced that the authors must have hidden some material which makes the paper good, and been surprised when I failed to find it.
The other day I went to a seminar at my university and their research was a textbook example of what (they taught us) you're not supposed to do. They're people who I think are quite smart, but I'm not even sure if there is something to salvage in their research at all. I stayed quiet while people debated minor points, and at the very end I asked about whether they considered ways to get around [the flaw], maybe an argument they could make to mitigate the problem. They were, of course, aware of the problem, but for whatever reason didn't have the ability to do better. It's not a minor flaw: any academic in a different field could find it.
It's a bit of a circle: the people in the "top" schools (which has a social meaning as well as an academic ranking meaning) are socialized to accept certain high standards and have almost a visceral urge to stay away from certain mistakes, and they end up being able to meet the high standards because they have access to the state-of-the-art knowledge and resources. People without such access don't have such a strong socialization to stay away from the flaws, and have fewer resources anyhow, so have a harder time meeting the higher standards, and perhaps have less knowledge about tricky alternative ways to get around the mistakes when there's truly no choice.
Happy Christmas to everyone who celebrates it.
No comments:
Post a Comment