Emerald academic culture survey 2020:
Impact evaluation

In August 2020, we commissioned a global survey to gather views on change within the academic sector. The survey was sent to a random selection of 172,033 academics, librarians and students within Emerald’s literati community. A total of 1,274 literati from 188 countries responded.

The survey report covers attitudes to research impact evaluation, cultural challenges within academia, openness and transparency, and the role publishers can play in furthering change within the research ecosystem.

Use the 'On this page' and 'In the report' menus to quickly see and jump to content you're interested in, or scroll down to see what's here.

academic culture survey 2020

Impact evaluation

On this page, we reveal the findings of our Academic culture survey 2020 in relation to impact evaluation. Here we investigate how change ready we are as a sector to move beyond traditional impact metrics and embrace fairer and more meaningful research evaluation.

Download a PDF of the full findings for this topic

Key themes

There is a growing desire within the research community for a broader number of metrics and indicators to measure the quality of individual research contributions. As a signatory of DORA and in line with our Real Impact Manifesto to move beyond metrics and celebrate impact commitment, we have rolled out various initiatives to create awareness of the limitations of metrics and drive real impact.

One area we have focused attention is helping researchers demonstrate the influence of their research on practice, policy and society. In collaboration with industry experts, we are developing a suite of resources that will help researchers tell their impact story. Support materials that are readily available include an Impact Literacy Workbook and Institutional Healthcheck Workbook.    

To guide our efforts to further research impact, we are continuously listening to the research community and probing further into the barriers to, and opportunities for, change. Our surveys and reports in these areas over the past three years have been an attempt to stimulate debate and bring conversations to the fore.

How change ready are we?

In this year’s survey, we found the desire for a broader impact metric had grown when compared to the previous year, with 20% of the research community calling for journal impact factors (JIFs) to be dropped all together, up from 13% in 2019. However, in terms of how research quality is measured at their institution, JIFs were perceived to play an important role – 71% selected JIFs as the way research quality is measured at their institution, up from 58% in 2019. 

According to respondents, the biggest challenges to change include ‘Incentives for career progression still aligned to traditional impact metrics (i.e. publishing in ranked journals)’ (56%), closely followed by ‘Difficulty in tracking research impact beyond academia’ (55%), and ‘Lack of clarity on what measures would replace rankings to assess quality’ (49%). 

Driving change

In terms of what individual researchers were willing to do to broaden the impact of their work and push for change, ‘Publishing open access and sharing links to supporting datasets to get more ‘eyeballs’ on my work’ came out on top, with just over half of researchers selecting this option. More opportunities for collaboration between industry and practice was believed to be the best way to make change happen, with 63% supporting this choice, up slightly from 2019.

On a scale of 1–10 where 1 is not at all important and 10 is very important, how important is demonstrating impact of research on society to... (average score out of 10)?

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

You personally Your university Funders Policymakers Society
Overall 2020 8 8 8 8 8
Overall 2019 8 8 8 7 7
Overall 2018 8 7 8 7 7
 
How is the quality of your research impact currently measured (percentage time chosen in top 3)? 

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

Journal citations and impact factors Tenure or career advancement Funding opportunities Measurable change in practice, policy or behaviour Provable effects of research in the real world Improved societal, health, economic or environmental outcomes Mobilised knowledge that affects decision-making in applied settings Other (including bottom 3 chosen options)
Overall 2020 71% 33% 31% 24% 23% 19% 18% 35%
Overall 2019 58% 30% 35% 59% 59% 61% 49% N/A
 
On a scale of 1 to 5, where 1 is not at all important and 5 is very important, how important are the following factors in helping achieve broader impact with your work (net important, scored 4–5)?

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

I want to make a difference to society I want to improve my reputation I want to advance my career I want to increase funding opportunities I need to meet institutional or funder requirements
Overall 82% 71% 67% 61% 60%

 

Do you expect the priority of measuring real-world impact to change in your institution in the next 12 to 18 months?

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

Yes No I don't know
Overall 2020 36% 41% 23%
Overall 2019 40% 28% 32%

 

How strongly do you support the idea of changing the way research impact is measured? 

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

Very open and I have already driven change in the way I approach research Very open but I have not yet driven change in the way I approach research Fairly open to change Neither open to change nor against it Fairly against change Not at all open
Overall 2020 33% 36% 22% 7% 2% 1%
Overall 2019 38% 31% 24% 5% 1% 1%

 

How supportive / interested are those in your broader institution in driving change when it comes to other ways to measure research impact?

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

Very open and they have already driven change in the way I approach research Very open but have not yet driven change in the way I approach research Fairly open to change Neither open to change nor against it Fairly against change Not at all open
Overall 2020 13% 19% 29% 23% 11% 5%
Overall 2019 16% 19% 30% 20% 10% 5%

 

What main change would you like to see in the way research quality is measured? 

Survey results 1Survey results 2Survey results 3

Survey results (table)
 

 

I want to make a difference to society I want to improve my reputation I want to advance my career I want to increase funding opportunities
Overall 2020 20% 53% 35% 7%
Overall 2019 14% 68% 43% 9%

 

Which of the following do you consider to be the biggest 'challenges' of changing the way research impact is assessed? 

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

Incentives for career progression still aligned to traditional impact metrics  Organisation resistant to change / entrenched culture Lack of funding for open research Regional drivers (discrepancies around impact ‘readiness’) Lack of clarity on what measures would replace rankings to assess quality Difficulty in tracking research impact beyond academia Lack of case studies or tools to help Lack of case studies or tools to help    Challenges with non academic collaboration
Overall 2020 56% 47% 39% 14% 49% 55% 13% 30%
Overall 2019 60% 34% 26% 9% 44% 41% n/a n/a

 

When it comes to your research, what types of change would you consider implementing? 

Survey results 1Survey results 2Survey results 3

Survey results (table)

 

Publishing non-traditional content (short form, policy notes, blogs etc.) if the rewards mechanisms for this were in place Publishing non-traditional content (short form, policy notes, blogs etc.) if the rewards mechanisms for this were in place Publishing open access and sharing links to supporting datasets to get more ‘eyeballs’ on my work Saving published work to my institutional repository (green open access) Publishing my research with a publisher that auto-deposits my Author's Accepted Manuscript (AAM) on my behalf I’m not considering changing my processes / methodologies I would like to make these changes, but feel unable to do so due to my institution Something else
Overall 2020 46% 44% 51% 34% 28% 10% 22% 5%
Overall 2019 48% 51% 29% 24% n/a 9% n/a 4%

 

In your opinion what are the best way(s) to enable change to happen (percentage of times chosen in top 3)?

Survey results 1Survey results 2Survey results 3Survey results 4Survey results 4Survey results 5Survey results 7

Survey results (table)

 

Greater understanding around impact literacy training More opportunities for collaboration between industry and practice Communications support Greater adherence to DORA principles More publishers making research open access More opportunities to debate the issues in a public forum Other
Overall 2020 31% 63% 24% 17% 52% 35% 8%
Overall 2019 35% 61% 23% n/a 35% 22% 7%

Your words & comments


Under pressure

When asked, ‘What main change would you like to see in the way research quality is measured?’, suggestions included:

Quality over quantity: “Pay attention to the quality of a researcher’s work, rather than quantity of research. Not all research should be equally weighted. Philosophical refection takes time. Yet, quantitative work can have quick outcomes. But their impacts are different. Current evaluation drives most researchers to do quick work, especially when they have heavy teaching load”

Female teacher, India

Changes to incentives: “Changing the incentive structure for the career and performance evaluation beyond the publication and impact factors”

Female, Researcher, SSA

The Emerald view

Supporting research that can make a real difference is crucial to our progress on global issues such as climate change and poverty, says Tony Roche, Executive Vice President of Publishing and Strategic Relationships at Emerald. In this context, he calls for the development and recognition of a broader range of research evaluation metrics, in addition to narratives that support the impact journey. 

Tony Roche

Drawing on our latest survey, it is encouraging to see researchers, institutions, funders and policy makers placing greater emphasis on the societal impact of research. While this is now widely accepted in principle, poorly aligned evaluation and incentive structures are clearly blocking these aspirations. Bibliometric indicators and citations have a role to play, and academic rigour need not be sacrificed for research to better connect with real world impact. 

There has been positive movement in some national evaluation systems, with open routes of dissemination increasingly preferred, however the participation of the intended beneficiaries in society is still limited, and mechanisms to mobilise knowledge remain poorly developed. 

This year’s survey also highlights cultural challenges within research that must be addressed through the research evaluation process as driver of change, to incentivise responsible research practices for the benefit of all. 

Giving voice to the underrepresented

Emerald is committed to action through co-creation, to bring the voice of the beneficiary as well as underrepresented researchers themselves more directly into the research and publication process, and we will hold ourselves to account to measure progress here.

As a participant within the global research and scholarly comms ecosystem, we work with over 30,000 researchers each year, and through our own commitments to diversity and inclusion we can ensure that the research we publish is more representative and reflective of the needs of society. 

Supporting research impact

Our commitments extend to working with policy makers and funders, so that a wider array of indicators and metrics, as well as the narratives to support the impact journey, are developed and recognised through evaluation processes themselves. This clearly requires coordinated efforts and a willingness to work together, so that research can perform better in its critical underpinning role to support societal progress in areas such as climate change mitigation, environmental degradation, poverty and illiteracy.

Openness & transparency

Read other results from our academic culture survey

View content

Academic culture

Read other results from our academic culture survey

View content

Join us in making a difference

For over 50 years, championing fresh thinking has been at the heart of the Emerald business. Our goal is to help those in academia or people in practice work together to make a positive change in the real world.

Publish with Emerald

Publish with us

Find the right home for your research across our journals, books, teaching cases and open access options.

See how
Social people

Talk to us

Do you have a question about open research at Emerald, or would you like to find out more about our open research routes?

Get in touch

Sign up for our newsletters

We'll be releasing a newsletter every quarter; fill in the form and we'll send you a link as soon as the next one comes out.