Q&A – unpacking the secrets of the hidden REF

What’s the hidden REF and how did it get started?

Simon: It all started because people kept coming to me asking how I helped get research software engineers (RSEs) recognised and could I help them do the same for data stewards, research managers or other roles in the research space. The point is these roles are largely hidden, so we needed something that people could submit ideas to and then we can sift those to see what hidden roles are out there. The obvious way to do it was through what has become the hidden REF.

We started with the categories from the Research Excellence Framework (REF) and chose the ones that aren’t related to publications, you’ve got website content, software, data and so on. The one that really struck me was software. There were 191,000 outputs submitted to the last REF, but 186,000 of them were based on publications, and only 38 were in the software category.

I just thought 38 out of 191,000 is not representative of how much software is out there. When we survey across disciplines, we find about 70% of research is fundamentally reliant on software. A huge amount of software is used in research, but only a tiny amount is being submitted to the REF.

I was unsure whether the categories were fit for purpose, so we decided to design the first round of the hidden REF as one where people can submit ideas on how to change the categories or add new categories to better represent the actual work being conducted by researchers. 

What kinds of categories have been submitted?

Simon: We’ve had lots of interesting categories come in, such as professional services and technicians – central supporting roles – no research can happen without those people, and people want to recognise that.

There are digital outcomes that people want to include in the competition, not just data and software, but also things like standards for data and software. People’s careers are completely based around what appears to be little tweaks of the REF categories. But it’s a difference that means their work can now be recognised and due to this they can justify their existence.

How does the competition work?

Simon: Anyone who works for a university or research organisation can submit to the competition. The submissions are purposefully short, just a few hundred words. We’re trying to raise recognition of the diversity of work in research rather than conduct the most stringent assessment. It’s more about starting a conversation about research and identifying what outputs we need to recognise.

The closing date is 26 February. After this, we will have panels of volunteer experts who review the submissions and identify which ones are the most impressive.

What outputs should be submitted to the competition?

Simon: We took the REF categories and deleted all the publication-based ones. We don’t allow publications because they are already well catered for in research assessment. Even within the committee that runs the hidden REF, that was an unpopular move. It’s funny, we are so addicted to publications!

If you’re going to submit a journal article into a competition, we already have the REF for that. I am interested in how people find the idea of taking publications completely out of this space. What’s left in research when you do that? How are we going to identify impact?

We are so hooked on the easiness of counting the number of publications, multiply it by the impact factor and supposedly you’ve calculated your value to research. It’s a very easy way of identifying impact, but there are plenty of industries around the world that don’t have such easily countable metrics. Why is academia so focused on publications as the only metric?

At the end of it, I hope we will have a showcase of the ‘hidden roles’ in research, along with people that are working in these newly recognised roles.

Is the competition open to all researchers?

Simon: The competition is for the UK academy at this point. Although the competition is UK focused, the conversation we are having is international. Ultimately, there are hidden roles in research in all other countries.  

Why is Emerald supporting the hidden REF?

Tony: Our whole vision is about helping to broaden participation in research and publishing, as a facilitator towards real impact in society. As a publisher, our core competencies include helping authors to optimise the reach of their work, so I can’t get away from the fact that we publish research. But we are guided by impactful content that can contribute to positive change in society, rather than the pursuit of ‘REF-able outputs’.  

The plan isn’t to publish more journal articles as an end in itself, if anything it’s about creating new forms of digital content that engage new audiences in different ways, appended to or associated with the underlying research so that it is useable without detriment to the rigour of the research itself. The research ecosystem is complex, and research culture slow to change, but if there is a willingness to collaborate then we believe that we can play a supporting role in driving changes to the way that research is evaluated, and by showcasing the many underrepresented through the research we publish.

We are also committed to driving the debate forward, to broaden the participation with people like Simon and the hidden REF team, as part of the wider movement. It’s the research community’s movement, but we want to be a facilitator, because it aligns to our real impact vision, and because we have creative and technical skills to support this.

Through our learning, we are also developing tools and services to support the research community in its approach to responsible research and publication. The foundation for all of this is built upon close collaboration with our communities, to co-create solutions that really support their needs, whether that be through publications, open research, or tools.

What are the challenges to broadening your remit as a publisher?

Tony: We experience the same challenges that Simon reports around research evaluation, that there is inertia in the system and misaligned incentives. Many researchers say to us that they would like to better communicate their work to society more broadly, but their priority is to get the journal article (or the book) published. Once their work has been accepted via peer review, they rarely feel there is enough incentive or reward for them to invest in using digital media more creatively, to reach beyond the walls of the academy.

We are coming from a similar vantage point to Simon and the hidden REF team, we recognise the underlying challenges and want to do something that influences research evaluation to drive wider participation. As a result, much of what I talk about at conferences and panels, relates to the need for action. The sector needs to move beyond good intent, and beyond pointing out the problems and collective wringing of hands about things that are deemed too big to solve. It’s ultimately about how we are going to have an impact in driving new forms of research evaluation, and more broadly driving positive changes to research culture itself.

Funders and policy makers are unlikely to listen to the voice of the publisher in isolation. We are committed to supporting a ground swell of researchers themselves, to influence those who hold power positions around evaluation in terms of what is recognised and rewarded in different roles and different forms of contribution. Progress is being made, but remains painfully slow.

How hopeful are you that there will be change?

Tony: We are optimistic that things will change as today’s ECRs come through in support of the other drivers of change. What we have seen with Open Access, despite the commitment at the higher institutional level or funder level, and this is an inconvenient truth really, often the researchers aren’t actually that engaged. If you put it in binary terms – open or closed, they will always say open, but actual commitment to transparent, reproduceable, open research behaviours is very different on the ground. That’s not because we as a publisher want to maintain the status quo, indeed we see our future as one within a fully open environment. We are investing to support the move to open science behaviours, that go way beyond open access to outputs, for example through Emerald Open Research, our SDG-focused gateways for social science communities.

No one is saying we will arrive at the destination in 2020 or 2021, a bit like DORA, but we see a more collective effort emerging, involving multiple groups such as the hidden REF. As these challenges and opportunities gain more traction, the people who hold the power structures will increasingly commit and change. That’s what we are here to support.
 

Contact Simon and his team to submit an entry to the hidden REF, provide support or offer your views on these issues.