Universities have become increasingly sophisticated in the way they collect and analyse data. Research publications are counted, citations are tracked, grant income is monitored and institutional performance is benchmarked against national and international competitors. Senior leaders often have access to dashboards showing research outputs by faculty, department, discipline and even individual academic.
Yet there is one activity that sits at the heart of scholarly publishing that remains surprisingly invisible.
Peer review is one of the foundations upon which the credibility of academic research depends. Every published article that passes through a reputable journal will have relied on experts volunteering their time to evaluate the quality, originality and validity of the work. Despite this, many universities have little idea how much peer review activity their staff undertake, who benefits from it or how much institutional effort it represents.
This creates an interesting paradox. Universities know a great deal about the outputs of scholarly publishing but often know very little about one of its most important inputs.
Beyond The Workload Debate
Whenever peer review is discussed within universities, the conversation often turns towards workload models. Should reviewing be formally recognised? Should academics receive workload credit for undertaking reviews? Should peer review be considered when evaluating academic contribution?
These are legitimate questions, but I suspect they are not the most interesting ones.
The more important issue may not be whether peer review is counted. Rather, it is what universities lose by not measuring it at all or, at least, collecting data about it. In focusing exclusively on workload recognition, we risk overlooking a potentially valuable source of information about the scholarly publishing ecosystem.
The purpose of collecting peer review data need not be to create another performance metric or another target for academics. Instead, it could provide a better understanding of an activity that is fundamental to research but currently sits largely outside institutional visibility.
What Data Could Be Collected?
Universities already collect information about publications, grant applications and research income. Extending this to peer review would not necessarily require anything complex.
For example, institutions might record the journal being reviewed, the publisher, the discipline, the approximate time spent, the date completed and perhaps whether it was a first review or a subsequent review round. Many academics may already keep some of this information for promotion applications, annual reviews, their CV, their web site or for professional recognition.
Individually, these data points may appear unremarkable. Collectively, however, they could create a dataset unlike any other that currently exists. For the first time, universities would be able to quantify the scale of their contribution to one of the most important quality assurance mechanisms in research.
More importantly, they would be able to move discussions about peer review from anecdote to evidence.
Understanding The True Cost
One of the first insights such data could provide is a clearer understanding of the institutional resources devoted to peer review.
Academic time is one of the most valuable resources available to a university. Every hour spent reviewing a manuscript is an hour that is not spent teaching, supervising students, writing grant proposals, conducting research or undertaking administrative responsibilities. This does not mean peer review is unimportant. Quite the opposite. It simply means that it consumes resources that are currently not captured.
If institutions understood the volume of reviewing undertaken by their staff, they could begin to estimate the scale of this contribution. They could identify which faculties devote the most effort to reviewing, whether certain disciplines carry a disproportionate burden and how this activity changes over time.
Such information could also contribute to more informed discussions with publishers. Universities know how much they spend on subscriptions and article processing charges. Perhaps they should also understand the value of the expertise they contribute to a given publisher.
Mapping The Global Peer Review Network
The real potential emerges if institutions begin sharing data.
Imagine hundreds or thousands of universities contributing anonymised information about peer review activity. The resulting dataset could provide an unprecedented view of the global research ecosystem.
We might discover which countries contribute the most reviewing effort and whether this aligns with publication output. We could examine whether nations that publish large volumes of research are also providing a proportional share of peer review labour. We might even compare reviewing activity with expenditure on article processing charges and other publishing-related costs.
These are important questions because peer review is often described as a collective responsibility of the scholarly community. Yet we currently have limited visibility of how that responsibility is distributed. A global dataset could reveal whether the burden is shared equitably or whether some institutions, disciplines or countries contribute significantly more than others.
Revealing What We Cannot Currently See
Large-scale peer review data could also provide insights into the operation of journals and publishers themselves.
For example, it may become possible to understand how many reviews are typically conducted for articles in different disciplines, how often manuscripts undergo multiple rounds of review and how reviewer demand varies across fields. Researchers could analyse trends over time and identify areas where reviewer capacity is becoming constrained.
More intriguingly, such data might reveal patterns that are currently invisible. If a journal publishes very large numbers of articles but relatively little associated reviewing activity can be observed, that may prompt questions about how its editorial processes operate. Equally, journals that invest heavily in rigorous peer review could demonstrate this more transparently.
The objective would not be to rank journals or identify winners and losers. Rather, it would be to improve understanding of a system that currently operates with very little transparency.
Does the Data Already Exist?
One of the more interesting aspects of this discussion is that much of the data may already exist. Publishers and journals routinely collect information about the peer review process as part of their editorial workflows. They know how many reviewers are invited, how many accept, how long reviews take and which institutions and countries reviewers come from.
This raises an obvious question. If the data already exist, should universities be collecting it independently, or should publishers be encouraged to make aggregated and anonymised information available? Doing so could benefit the entire research community.
Universities frequently negotiate subscription agreements, transformative agreements and article processing charges with publishers. Yet those negotiations are largely financial. If peer review data were available, universities could also understand the scale of the intellectual labour they contribute to the publishing ecosystem. That may not change the relationship between universities and publishers, but it could make it more transparent.
Researchers could study how peer review operates across disciplines and regions. Publishers themselves could demonstrate the robustness and transparency of their review processes.
Perhaps the challenge is not the absence of data. Perhaps it is that the data is fragmented, held privately and rarely shared in a form that allows the wider scholarly community to learn from them.
The Risks of Measurement
Any proposal to collect additional data inevitably brings risks.
Academics are understandably wary of new metrics. The higher education sector already has extensive experience of measures that were originally intended to provide information but later became targets. Peer review should not become another activity that staff feel compelled to maximise in order to satisfy institutional reporting requirements.
There are also practical considerations. Data collection would need to be simple, proportionate and respectful of the confidentiality that underpins the review process. Universities would need to ensure that the purpose of collecting information was to understand the system rather than evaluate individual performance.
These concerns are important and should not be dismissed. However, they should not prevent us from considering the potential value of a dataset that could significantly improve our understanding of scholarly publishing.
A Governance Opportunity
Universities increasingly emphasise evidence-based decision making. Strategic planning, resource allocation and policy development are all expected to be informed by data. Yet one of the most important activities supporting academic publishing remains largely absent from institutional datasets.
Perhaps this is understandable. Peer review has traditionally been viewed as a professional responsibility undertaken by individual academics rather than an institutional activity. However, when thousands of academics collectively contribute thousands of hours to sustaining the scholarly record, it becomes difficult to argue that universities have no interest in understanding that contribution.
The more I think about peer review, the more surprising it seems that we know so much about publications and citations while knowing so little about the process that helps validate them. Universities have spent decades building systems to measure research outputs. Perhaps the next opportunity lies in understanding one of the most important inputs.
Perhaps the biggest surprise is not that universities do not collect these data. The biggest surprise may be that publishers already possess much of this information, yet the scholarly community has remarkably little visibility of it. If peer review is fundamental to research quality, should its operation remain largely hidden, or is it time for greater transparency?
Final question
The question is not whether peer review matters. The entire scholarly publishing system depends upon it.
The question is why we have spent decades measuring publications, citations and research income, while paying so little attention to the activity that helps determine which research is ultimately published.
If peer review is fundamental to research quality, perhaps it is time for the data surrounding it to become visible as well?
About the Author
Professor Graham Kendall is Vice-Chancellor of GlobalNxt University and an Emeritus Professor at the University of Nottingham. He has held senior leadership positions including Vice-Chancellor, Provost, Pro-Vice-Chancellor and Deputy Vice-Chancellor across the higher education sector.
His interests include university governance, research strategy, artificial intelligence, research evaluation and scholarly publishing. He has published more than 300 peer-reviewed papers and has been actively involved in academic publishing, editorial activities and peer review for more than 25 years.