Research impact assessment: the 400-year-old conundrum

From the UK and US to Australia and China, the last decade or so has seen increasing efforts to find ways to assess researchers’ contributions to human progress. But how can this be measured?

Elsevier’s recent study of 400 researchers, academic leaders and heads of funding bodies sheds fresh light on the issue. Back to Earth: Landing Real-world Impact in Research Evaluation shows that many in the research community want an acceleration of effort in overhauling academic assessment based on improved methodologies and ecosystem-wide consensus.

Still, it’s worth remembering that this is not a new debate. Francis Bacon, the 16th-century English philosopher, lawyer and politician who is credited with being an early proponent of empiricism, argued that science was a public good that should serve progress. His argument was based on the immense value of three inventions: printing, gunpowder and the magnetic compass.

Bacon did not use the word “scientist” because that word — and the modern concept of the role — was popularized only in the late 19-century United States.

That was a time of furious and often ill-tempered public debate about motivations, corruption and increasing opportunities to exploit and commercialize the results of academic research.

A plea for pure science

Perhaps most famously, the US physicist Henry Rowland used an address to the 1883 annual meeting of the American Association for the Advancement of Science (AAAS) to issue a “Plea for Pure Science S’ouvre dans une nouvelle fenêtre.” He complained that with inventions like the telegraph and electric lights, obscure newcomers were stealing ideas from the past and being “lauded above the great originator of the idea, who might have worked out hundreds of such applications had his mind possessed the necessary element of vulgarity.”

Rowland was not against progress; rather, to protect the integrity and reputation of science, he wanted to keep a respectable distance between the research and the application. But as public funding for science ramped up after World War II, policymakers became keener to push science and its application closer together.

This was especially apparent in the United States, where the Manhattan Project to develop the atomic bomb — and other military advancements based on new technology —helped convince policymakers that federal funds for scientific research could make a real difference.

A report to President Franklin D Roosevelt from Manhattan Project pioneer Vannevar Bush helped to usher in the new era. Titled Science — The Endless Frontier, the 1945 report made a series of recommendations, including that the government set up and fund a national research foundation to oversee peacetime research and award basic science grants to universities. The National Science Foundation that followed in 1950 followed that model and set the US and the world on a path that would change science policy and funding forever.

The gap between promise and delivery

After several decades of increased public investment in research, however, some politicians began questioning what they were getting for their money. One prominent example of a call for change was by British prime minister Margaret Thatcher, who as a former chemist, knew something about balancing both sides of an equation. In 1986, UK officials started asking universities for regular summaries of their research activities for grading. Those with the best perceived quality received more central funding in the next round.

Some early efforts at systematized research impact assessment looked at academic metrics, partly because such statistics were available and could be readily compared and sought to reassure those holding the purse strings (and those who voted for them) that they were at least paying for the best possible academic results.

However, those assessments rarely followed through to examine whether academic quality produced societal progress (partly because such statistics were unavailable and could not be readily compared).

Fast forward to today, and research impact assessment is commonplace, although still an imperfect art. That’s not surprising: although Francis Bacon declared that discoveries “confer benefits without causing harm or sorrow to any,” today’s debates on sustainability mean that some research that advances the economy could be seen as causing overall harm — a real conundrum for impact assessment.

Despite this, many countries are conducting broad impact assessments, aided by the wealth of data available in our digitized world that has facilitated measurement techniques that would previously be unthinkable. Organizations such as the Coalition for Advancing Research Assessment (CoARA) S’ouvre dans une nouvelle fenêtre and the Network for Advancing and Evaluating the Societal Impact of Science (AESIS) S’ouvre dans une nouvelle fenêtre are providing essential forward momentum.

Today’s impetus for change

However, Elsevier’s survey of the research community shows that many want to move further, faster. Barely half of the 400 respondents — who came from 10 countries, including the US and UK — think current research assessments incentivize work that can make a meaningful difference to the wider world. Two out of three say academia has a moral responsibility to incorporate real-world impact into standard research evaluation.

In Bacon’s day, some said improving the mind — as opposed to improving the world — should be the main aim of research. That view seems almost obsolete today: In the survey, just 1% opposed using real-world impact on research evaluation. And 55% said they thought academic leaders were best placed to advance the quality of impact evaluation, although they felt the public and funding bodies had an important role, too.