reviewing

Defending propositions: an index for reviewers

by

This is the third post in the propositions series. If you don’t know what I mean by “defending propositions”, you can read the introduction here.

The proposition

I recently wrote about reviewing papers for the first time, and how happy I was when I recently discovered Publons. Publons lets you track your reviewing, essentially making visible all the effort you put into it. Here is an example of a statistic I’ve extracted from my profile:

Example statistic from publons.com
Example statistic from publons.com

This idea resonated with me because of a proposition I defended a year before, and that I will discuss today.

Introducing an r-index for reviewers as a counterpart of the h-index for authors would lead to more, better and timelier reviews.

Supply vs demand

During my PhD I noticed that the supply vs demand for publications was very different than that for reviews. For publications, supply was clearly higher than the demand: many papers would get rejected, and even if accepted, some papers would never be cited. For reviews, however, the story seemed to be the opposite! It could take months to receive reviews for your paper. And when you would finally get the reviews, regardless of the decision, a few would be very short and uninformative.

I wondered whether this discrepancy could be a result of the reward system. A new publication is rewarded by your name on the paper, by citations, and perhaps even increased chances of getting a position. A new review is – to the outside world – at most a line on your CV. Reviewing is something that is expected to be there, but I doubt it has ever been a deciding factor in a hiring decision. Combine that with well-meant advice from colleagues, “say no to everything, just focus on your publications!”, and you’ve got a supply vs demand problem.

Metrics

The h-index is meant to measure the productivity of a researcher. Although it has flaws, it is being used — consciously or perhaps subconsciously — to evaluate quality. As a result, it is something researchers can try to optimize for. And for a higher h-index, you need start with getting more publications. Do you already see where the supply is coming from?

What if there was something similar for reviewing papers? If such a – yet to be defined – reviewer index became an important metric, more people would want to review papers and submit their reviews on time, just as they do with papers.

Quality is a bit more difficult. One way to do this is a system where reviews could be seen and discussed by others. This doesn’t mean that the reviewers’ identity needs to be revealed. See for example the reviewing process of ICLR.

Or maybe we can step away from reviews altogether, as Ludmila Kuncheva proposes
here. (The post also features one of my favorite stories about publishing before the internet). Reviews would simply be replaced by citations. Then maybe we wouldn’t need new metrics, and could just go back to looking at the h-index instead.

Firsts: reviewing a paper

by

After a conversation on Twitter about when to start reviewing papers, I decided to post about my own experiences with reviewing, when and how and when I started, and what I wish I would have known in advance / done differently.

I was in my third PhD year that I started wondering about being a reviewer. I heard of another PhD student being asked to review, and got worried why that has not happened to me yet. But shortly afterwards, I was invited for two journals, and then more and more invitations (including more important journals where I have not published) followed, so the worry disappeared. In this post I share some experiences about what helped me with reviewing, and what I could have done better.

Getting invited

  • I discussed the issue with more senior academics, who were already reviewing for a while. They gave me some pointers about how they got started, but they also made a mental note of me being a potential reviewer. Then, if they were an editor, a program committee member, or simply unavailable to review themselves, they could mention my name.
  • I started submitting journal papers, which meant that I had to create accounts with publishers. Editors were then able to link my name to different keywords, even if my own papers were not published yet. In fact, most of the invitations I received were from journals that I had never submitted to, but that shared a publisher with a “submitted-to” journal. In retrospect, I could have created the accounts even before I was ready to submit, and indicated that I was available to review.
  • I maintained an online presence with the page at my university and Google Scholar. My guess is that editors who did not know me personally, would scan these pages to get an idea about my research and experience. Having a blank webpage (with only your name and contact details) might send the wrong message in such a case.

Accepting the invitation

Once I got an invitation, I accepted it! The paper was on the topic of my PhD and the invite gave me a feeling of accomplishment (and relief), so I felt ready to do it. It could also happen that you do not feel ready, for example because it is unusual for PhD students to review papers. My advice is to go for it anyway. Somebody else thought you have the expertise you do it, so don’t listen to the impostor inside you, and give it your best shot! Reviewing taught me a lot about my own writing, so I’m happy I didn’t wait until I was “grown up” to start doing it.

Reviewing

Then it was time for the actual review. I spent quite a lot of time on this, because I was doing the review by myself, and I wanted to make sure I didn’t miss anything. I thought the paper was not very strong, but I felt I couldn’t rely on just my opinion to say that. So, I discussed the approach with my supervisors, who agreed with me.
Then I read a few “how to” guides and started writing what I wanted to be a constructive review. I also went over the reviews I had received for my papers, as both good and bad examples of reviews. This was more insightful than the guides, so in retrospect, I should have asked colleagues for more examples to learn from.

An alternative would have been to ask a supervisor or colleague to go through the review process with me from start to finish. At the time, I probably thought this would be asking too much, but now I feel that many academics would be happy to do this.

Keeping track of reviews

I created a folder called “Reviews” with a subfolder with the name of the journal, and stored the submission and my review of it. This seemed sufficient at the time, as most people listed only the names of the journals on their CV. But it might be a good idea to keep track of what happened to the paper afterwards, should you need that information later.

Update 4th of October 2016: I found an excellent place to keep track of reviews, share reviews and even rate reviews written by others: Publons! I am still in the process of adding reviews to my profile, but you can already check it out here

Mastodon More Mastodon
%d bloggers like this: