Advancing responsible research assessment: NICHD’s story

Disclaimer: Chris Belters' participation in this article and webinar does not constitute an endorsement of any vendor, product, or service. All opinions are his own and do not necessarily reflect those of NICHD, the NIH, or the US Government.

The idea that the assessment of researchers and their outcomes should be “responsible” is nothing new. But factors such as the increasingly competitive environment that researchers operate in, and the growth of open science and accountability, have given the topic a new resonance. In this article, Christopher Belter, Program Analyst for US biomedical research funder Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), explains how his agency is responding to these shifts. In the second article of this two-part series, we will look at the steps that Elsevier is taking in this area, not only within its business and solutions, but via the work of its International Center for the Study of Research (ICSR).

Chris devoted his first weeks in the role to thinking about how NICHD can improve assessment. The result? A new conceptual research assessment framework is now out for review with NICHD’s program officers.

And, as Chris notes, he didn’t have to look far for inspiration: “I realized that to get closer to this idea of responsible metrics, we needed to borrow from evaluation science. The evaluation framework that will help us map what we do and how we measure it.”

Resources versus results

Chris knew that drawing up a framework would involve striking the right balance between “responsibility” and “feasibility”.

He explains: “Irresponsible use of data, metrics, and indicators puts an enormous pressure on researchers to do the right kinds of research, produce research outputs, publish them in the right places, and do all kinds of things that aren’t really appropriate for the healthy conduct of science. So, it was always clear that whatever assessment method we developed at NICHD, we would have to ensure it didn’t make the situation worse.”

Accuracy has been another big focus. “It’s not only about making sure that data sources and indicators are as accurate as possible, but that we are using the right indicators for the things we are trying to measure. We have a very broad field of research, so we can’t take a one-size-fits-all approach.”

Chris also had to consider the feasibility of any assessment measures proposed: “We currently have 3,700 active projects associated with 400 different research funding opportunity announcements that produce close to 10,000 publications every year. Many of our staff focus on administering our grants. Our assessment and accountability team is small. And, as a funding agency, we don’t have the kind of access to researchers that folks at the university- or laboratory-level enjoy.”

To develop his framework, Chris followed a five-step approach. “One of the core principles of assessment is to evaluate a program based on the good that it is trying to do in the world. So, we needed to work out what we ultimately wanted to achieve and then work our way backward.”

STEP 1:  Establish the long-term outcomes

NICHD has two – develop new research and improve health.

STEP 2: Set intermediate-term outcomes

Chris and the team brainstormed a series of outcomes to help them reach those long-term outcomes

STEP 3: Develop short-term outcomes

Chris notes: “We wanted to capture the kind of levers we can start pressing now.” He adds: “What’s nice about the short-term outcomes we’ve chosen is that they are relevant to both our long-term outcomes.”

STEP 4: Map these against your organization’s activities

For Chris, this involved dividing NICHD’s various programs into key categories, which the framework will help them assess the impact of.

STEP 5: Design relevant indicators

This step is still under development. Chris explains: “Now we have our draft framework, we need indicators for each individual goal and indicators that track our progress from one goal to the next. Then we will have a framework that not only captures what we need to measure but puts those measures in context and ensures we are interpreting them in the best way we can.”

He adds: “Research is a complicated process, and while I don’t think we can ever really prove direct causality between our framework fields, I don’t think that should stop us from trying to make that link. That’s what prompts us to ask the right kinds of questions.”

How the framework works in practice

To illustrate how the framework can support NICHD’s various programs and activities, Chris has developed examples for two classic NICHD funding initiatives - Impact for a clinical research project and Impact for a sociological research center, both illustrated below.

For Chris, developing the draft framework is only the start of the process. “Now we need to finish it, actually use it, and make sure that we reassess it on an ongoing basis.”He adds: “I think it’s going to help us be a bit more responsible in what we are trying to do. Ultimately, it’s about making sure it can measure the things that we care about – not just the things that we can actually measure.”

Chris shared NICHD's plans for a research evaluation framework during the Library Connect webinar Advancing responsible research assessment: implications for librarians and their users, held in May 2021 (timestamp 03:50 - 27:05). View the webinar. 새 탭/창에서 열기