This is the third post in the propositions series. If you don’t know what I mean by “defending propositions”, you can read the introduction here.
The proposition
I recently wrote about reviewing papers for the first time, and how happy I was when I recently discovered Publons. Publons lets you track your reviewing, essentially making visible all the effort you put into it. Here is an example of a statistic I’ve extracted from my profile:
This idea resonated with me because of a proposition I defended a year before, and that I will discuss today.
Introducing an r-index for reviewers as a counterpart of the h-index for authors would lead to more, better and timelier reviews.
Supply vs demand
During my PhD I noticed that the supply vs demand for publications was very different than that for reviews. For publications, supply was clearly higher than the demand: many papers would get rejected, and even if accepted, some papers would never be cited. For reviews, however, the story seemed to be the opposite! It could take months to receive reviews for your paper. And when you would finally get the reviews, regardless of the decision, a few would be very short and uninformative.
I wondered whether this discrepancy could be a result of the reward system. A new publication is rewarded by your name on the paper, by citations, and perhaps even increased chances of getting a position. A new review is – to the outside world – at most a line on your CV. Reviewing is something that is expected to be there, but I doubt it has ever been a deciding factor in a hiring decision. Combine that with well-meant advice from colleagues, “say no to everything, just focus on your publications!”, and you’ve got a supply vs demand problem.
Metrics
The h-index is meant to measure the productivity of a researcher. Although it has flaws, it is being used — consciously or perhaps subconsciously — to evaluate quality. As a result, it is something researchers can try to optimize for. And for a higher h-index, you need start with getting more publications. Do you already see where the supply is coming from?
What if there was something similar for reviewing papers? If such a – yet to be defined – reviewer index became an important metric, more people would want to review papers and submit their reviews on time, just as they do with papers.
Quality is a bit more difficult. One way to do this is a system where reviews could be seen and discussed by others. This doesn’t mean that the reviewers’ identity needs to be revealed. See for example the reviewing process of ICLR.
Or maybe we can step away from reviews altogether, as Ludmila Kuncheva proposes
here. (The post also features one of my favorite stories about publishing before the internet). Reviews would simply be replaced by citations. Then maybe we wouldn’t need new metrics, and could just go back to looking at the h-index instead.