Walk into almost any university and you will find people who are busy.
Calendars are full. Meetings follow meetings. Reports have to be written. Committees need papers. Academics are teaching, supervising, applying for grants and trying to publish. Professional services teams are recruiting students, developing partnerships, responding to regulators and supporting everything else the university does.
There is certainly no shortage of activity. But activity and progress are not the same thing.
That distinction matters because organisations can become very good at measuring how much they are doing without asking often enough whether all that activity is taking them where they actually want to go.
Activity is easy to count
Universities measure an enormous number of things.
We count publications, grant applications, research income, student applications, enrolments, international partnerships, events, outreach activities, meetings, projects and almost anything else that can be turned into a number.
There are good reasons for doing this. Leaders need information. Boards need assurance. Regulators require data. Universities also need some way of knowing whether strategies are being implemented and resources are being used effectively.
The problem comes when the measure of activity starts to stand in for the outcome we originally wanted.
Consider international partnerships. A university might report that it signed 20 new agreements during the year. That sounds impressive and can easily appear as a positive result in an annual report.
But what happened next?
Did students move between the institutions? Did academics collaborate? Were joint programmes developed? Did anybody publish together? Was research funding secured? Are the agreements still active two years later?
If only two of the 20 agreements resulted in anything substantial, then 20 may not be the most useful number.
The institution was certainly active. Whether it made progress is another question.
Outputs are not necessarily outcomes
The same issue appears in many areas of university life.
Submitting more grant applications demonstrates activity. Securing more research funding is an outcome.
Running more recruitment events demonstrates activity. Recruiting students who are well suited to the programmes, remain enrolled and ultimately graduate is closer to the outcome we presumably wanted.
Holding more committee meetings shows that people met. It tells us very little about whether better decisions were made.
Even research publications, which appear to be an obvious output, need some thought. Counting papers tells us something useful, but it cannot by itself tell us whether the research was important, whether it influenced the field or whether it contributed to the university’s broader research objectives.
None of this means that activity measures are worthless. Often, they are necessary intermediate indicators. The difficulty arises when we stop asking what they are meant to indicate.
A metric that was originally introduced as a proxy for something important can gradually become important in its own right.
When the measure becomes the objective
Once a university decides that something will be measured, behaviour will inevitably respond.
If the number of international agreements becomes a KPI, there is an incentive to sign agreements. If grant submissions are measured, there is an incentive to submit more grants. If the number of events matters, more events will be organised.
People are responding rationally to what the institution has said is important. But the number can begin to drift away from the original purpose.
Why did we want international partnerships? Presumably not because we wanted signed documents. We wanted collaboration.
Why did we encourage grant applications? Presumably because we wanted to fund good research, not simply increase the number of applications passing through the research office.
Why do we hold meetings? Presumably because something needs to be discussed or decided, not because a meeting itself represents an achievement.
Leaders need to keep returning to the purpose behind the measure.
Otherwise, the danger is that the organisation becomes increasingly effective at delivering the indicator while making surprisingly little progress towards the objective that gave rise to it.
The things that matter can be difficult to measure
There is another problem. Some of the things that matter most to an institution are difficult to put into a spreadsheet.
How effectively does the leadership team work together? Do people trust their managers? Can staff challenge a decision without worrying about the consequences? Does the institution learn from mistakes? Are decisions made at the appropriate level? Do academics and professional staff feel that their expertise is respected?
These things affect institutional performance, but they are not easily reduced to a simple number. That can create a bias towards measuring the things that are easiest to count rather than the things that are most important.
A dashboard containing 30 numerical indicators may look rigorous. But a dashboard is only as useful as the questions behind those numbers. There is also a temptation to believe that because something has been measured precisely, it must be important. Precision and importance are not the same thing.
Measuring things also creates work
There is a further issue that universities sometimes overlook: measurement itself has a cost.
Every KPI has to be defined. Somebody has to collect the data. Somebody else may have to check it. Reports are produced. Performance is discussed. Variances need to be explained. Actions might then be assigned and monitored.
Multiply that across departments, faculties and central functions and the workload can become substantial. Ironically, an organisation can make itself busier by trying to measure how effectively it is performing.
That doesn’t mean universities should abandon measurement. They couldn’t sensibly be governed without it. But perhaps every new measure should carry an obligation to justify the work required to produce it.
And perhaps introducing a new KPI should sometimes mean removing an old one. Universities tend to be much better at adding measures than retiring them.
What would happen if we stopped measuring it?
One question I think leaders should periodically ask about every major performance indicator is:
What decision will we make differently because we know this number?
If there is a good answer, the measure probably has value. If the information identifies a problem, changes the allocation of resources, influences strategy, triggers intervention or helps the institution understand whether an objective is being achieved, there is a reason to collect it.
But if the number appears in a report every month or every quarter and nobody does anything differently as a result, we might reasonably ask why it is being collected.
A second question follows naturally:
What would happen if we stopped measuring it?
Perhaps nothing. And if nothing would change, that might tell us something useful.
Busy organisations can still stand still
Being busy can feel reassuring. There is visible activity everywhere and plenty of evidence that people are working hard. But a full calendar is not a strategy. A long KPI dashboard is not evidence of success. And an institution can complete an impressive number of activities while making surprisingly little progress on the things that matter most.
This is not an argument for measuring less simply for the sake of it. It is an argument for being clearer about why we measure something in the first place.
Universities should certainly know what they are doing. But they also need to know whether all that activity is actually getting them anywhere. Perhaps that is the question worth asking the next time we look at a dashboard full of green indicators:
We know everybody is busy. But are we actually making progress?
In a future edition of this newsletter, I will ask a related and, in some ways, even more important question: should we be measuring inputs or outputs? If you want to read my thoughts on that, please consider subscribing so that you are alerted when that edition is posted.
I publish two free newsletters each week
Governance & Research in HE
A weekly brief for higher education leaders and researchers examining governance, leadership and research in practice.
https://buff.ly/GhsZqVl
Publishing with Integrity
Examining scholarly publishing through an ethical lens, challenging assumptions and supporting academic career development.
https://buff.ly/WiG1yFb
If you subscribe, you will be alerted whenever I publish a new edition.
About the author
Professor Graham Kendall is an academic and university leader with extensive experience in higher education leadership, governance and research. He has held senior university roles in the UK and Malaysia and is an Emeritus Professor at the University of Nottingham. His work and commentary focus on higher education, leadership, research and scholarly publishing.