Digital dashboard tackles unconscious bias in academic recruitment

The topic came up when Margaret met with Elsevier’s Chief Academic Officer, Dr Nick Fowler 새 탭/창에서 열기, during his visit to Australia in 2020. When they touched on the use of research indicators in recruitment and promotion, Margaret’s gender radar was triggered:

There is this view that using indicators and data to generate candidate shortlists is less biased than traditional methods, such as reviewing CVs. But I think indicators can be just as biased, and can discriminate against women, in particular.

She points to the h-index as one of the biggest culprits:

It relies on length of career and other factors, including patronage from senior colleagues, yet women are more likely than men to take career breaks for family reasons and/or be nominated less frequently for awards and international collaborations.

However, her years in academia and university leadership roles, along with her policy work for the Australian government, have shown Margaret just how valuable indicators can be.

People really like using them — I do too when I want evidence of outcomes or performance in an area or discipline — but I am very careful when they are applied to the evaluation of individuals.

So I asked Nick whether we could use indicators to make the initial selection process for vacancies fairer; for example, look at people’s careers more inclusively and reward other behaviors, such as being a generous collaborator or mentor.

The application is currently being tested by Margaret and other research leaders in the higher education sector. And while the project team gathers their feedback, work is already underway on the next stage of the collaboration — the development of a graphical CV for researchers that provides an enriched view of their careers and achievements over time.

The recruitment dashboard — creating a more even playing field

How the application works

An institution provides the ICSR with a search query defining the field of expertise required for the vacancy. This is used to generate a dashboard of suitable candidates, which contains more than 30 indicators on five themed tabs:

• Career Status

• Is Innovative

• Leader in the Field

• Multi-Dimensional

• Social and Education Oriented

Figure 1 shows the Is Innovative tab for candidates. Dynamic text at the top provides a breakdown of inferred gender, including an “unknown” category for researchers whose gender is impossible to infer within the confidence threshold. For Margaret, even this relatively simple data point adds great value:

Knowing the gender breakdown of the candidates on your shortlist can help to raise early red flags. For example, if I’m recruiting for a biology position and I see more men than women, then I know I should check the data because women generally outnumber men in that field.

Each indicator has a histogram showing the distribution of candidates by gender, with a “slider” filter that dynamically changes the data. For example, Margaret can reduce her shortlist by increasing the percentage of publications with funder acknowledgement that candidates must have published. And whenever a filter is adjusted, all histograms in the dashboard are updated, including the figures that break down the proportion of gender in the shortlist. This allows Margaret to see how her weighting affects other indicators, the overall number of researchers on her shortlist, and the ratio between men and women.

A Results table at the bottom of the dashboard enables institutions to select and download indicators for their chosen shortlist of candidates. The candidates are not individually identifiable by gender.

Generating new insights with new indicators

For Margaret, while all tabs and indicators have their value, there are a few she turns to regularly. For example, histograms on the Social and Education Oriented tab show the number of candidates’ advisees and their percentage breakdown by gender.

“These are some of my favorites,” she says. “I want to recruit people for QUT who will mentor the next generation of researchers. The advisee indicator shows how nurturing the candidates on the shortlist are overall, as well as how nurturing they are towards men and women, in particular.”

She adds: “While these indicators still need refining, people are already getting excited about them as they credit something that is rarely recognized.”

The co-author information on the Multi-Dimensional tab is also proving popular at QUT (see figure 2).

“The indicators show that women tend to have a higher percentage of women co-authors than men do, Margaret explains. “That’s something we wouldn’t see if we were looking at CVs alone.”

When considering men for a role, Margaret looks to see whether they have a healthy number of women co-authors relative to the field. “That tells me they have a diverse lab and are thinking more inclusively and broadly,” she says. “And if I see a high-profile woman who isn’t publishing with other women, that suggests they aren’t necessarily good at supporting women colleagues.”

For Margaret, the ability to work in a team and collaborate at all levels is crucial for potential QUT employees. “I was recently selecting a new head of school, and one of the school’s professors pointed out that while a man candidate had a really good research record, the data on the dashboard showed that he rarely published with more than one person. He said, ‘That’s not what we are about here. We are collaborative and want to work across disciplines.’ He had a point!”

Margaret also finds the 5-year field-weighted citation index (FWCI) on the Leader in the Field tab useful. The FWCI normalizes impact by field. An FWCI value of 1 indicates the average global impact for that research area, while an FWCI of more than 1 indicates higher than expected citations, and vice versa.

Margaret reveals: “I was on the selection committee for a prestigious national prize with a shortlist of very high-performing researchers. We evaluated them in a variety of ways, but there was one woman candidate with a high h-index, and the committee members thought it was down to her field, where everyone has high publication rates. Using the FWCI, I was able to show that, even within that field, she was a very strong performer.”

Addressing a pressing industry need

For Margaret, initiatives like the dashboard are much needed given the ongoing rise in indicator types and uptake. “We don’t want people relying on indicators that entrench explicit bias at the same time as we are trying to change behaviors.” Importantly, the dashboard can also result in a more cost-efficient and effective recruitment process for institutions.

And to those who might have concerns over the potential for positive discrimination, she says: “This initiative is just about broadening your pool beyond the networks of your team or external recruiters. We are not diluting excellence — we always want to appoint quality people. The dashboard just helps us make good choices about how we recruit them.”

From Margaret’s perspective, these choices not only create a fairer recruitment process, but they also help to build a better future:

There are so many examples of where diversity has led to better outcomes; for example, in tackling implicit biases in research questions. A broader network brings the best minds to a problem. And universities need to reflect the societies they serve — that’s something we haven’t done historically, at least in recent times.

For researchers, benefits include a more transparent and equitable recruitment process. Margaret notes: “I’ve found that people don’t mind entering a competition if they feel it’s a fair or level playing field. And this dashboard will help us to retain talent. If early career researchers feel they aren’t being recognized, promoted or rewarded, they will go and do something else. So, for me, the dashboard is not just about recruitment, it also has the potential to help us promote and support the right internal people.”

The graphical CV — telling the story of the researcher

The second strand of the collaboration between the ICSR and QUT is the creation of a graphical CV for researchers. Although the project is still in its early stages, figure 3 shows a conceptual view.

According to Angela, the goal is to create a CV for each candidate that displays selected indicators over time, helping recruiters estimate their career trajectory. Although the CV will be automatically generated using data from a variety of sources, researchers can annotate the document: for example, provide more detail on career breaks or achievements.

For Margaret, this kind of in-depth, more holistic view of the researcher is invaluable from a recruitment perspective: “I’m looking for people who are good at what they do. So if there’s a career break, this CV helps me understand what they did before that break, and whether it was any good.” She adds:

Whatever a candidate’s background, I want people who can climb a mountain and reach the peak.

“The aggregated indicators you see in traditional CVs often average things so that those peaks are excluded,” she explains. “For example, I know people who have had a good h-index for a very long time, just based on one or two blockbuster PhD papers. Seeing the trajectory of a career like this is much more helpful. There is so much richness in there.”

Next steps

The project team plans to continue developing the CV concept with input from Margaret and her colleagues at QUT. It will also continue gathering feedback from the academic community to optimize the dashboard’s performance, and usefulness, to QUT and other universities and research institutions worldwide.

According to Angela, she and the ICSR team are also exploring additional ways the dashboard can be generated. It currently relies on the institution supplying a customized search query, but in the future, it might be possible to create a dashboard based on a SciVal Topic, or even select a researcher who fits a vacancy’s requirements and then ask the dashboard to find lookalikes.

Angela adds: “We have so many ideas, and many of them come from Margaret. We’ve really appreciated her honest feedback — every time she suggests a pivot in direction, the dashboard improves.”

Margaret is also eager to continue the collaboration:

I could have found a data scientist to do this work for me, but the ICSR is at the forefront of this field, and far ahead of where any academic would be — it understands what’s possible in terms of the data and is familiar with institutions’ needs and best practice. Having this kind of think tank available is a great opportunity.

For Margaret, the collaboration has been a rare opportunity to return to her academic roots:

This has really felt like a proper research project. By working with the ICSR team, which is at the cutting edge of this kind of work, I’m learning so much.