Helen >> Hello and welcome to this episode of the Emerald Publishing Podcast. In this episode, we're here to discuss the Research Excellence Framework, or REF. In March 2020, as the UK went into lockdown amid the global coronavirus pandemic, the REF team at Research England made the decision to put REF2021 on hold until further notice. This means that the original submission deadline of 27th November 2020 will no longer apply. So what does this mean for the research landscape? How is the delay affecting academic and support staff? And what are the wider implications of the Covid-19 crisis on the funding ecosystem? Here to talk about this, we have Mark Taylor, Head of Impact at the National Institute for Health Research
Mark >> I'm Mark Taylor I'm Head of Impact CCF for the National Institute for Health Research. We are a health and social care research funder, we put in about 1.3bn pounds a year to universities and hospitals up and down the country. We're not directly involved with REF but we follow it with great interest and it does affect the work we do.
Helen >> and Chris Hewson, Social Sciences Faculty Research Impact Manager at the University of York. Chris >> I'm Chris Hewson, I am the faculty impact manager for the social sciences at the university of York my remit is REF impact; I manage the ESRC impact acceleration accounts and also some of the policy coordination business engagement functions in the social sciences.
Helen >> Both Mark and Chris sit on Emerald's Impact Advisory Board; a group designed to discuss the higher education research cycle and champion real impact. We met up in early June to talk about postponement of the REF and we kicked off the conversation talking not just about the technical and administrative implications of this decision but also talking about how the sheer amount of effort that people put into cycle like this and that subsequent delay, what that emotional fallout might be on staff and faculty.
Mark >> The issue about REF is it's not just a technical issue it's one of the cultural change as well and therefore there's an emotional issue as well. The efforts of this put in by teams across universities when REF demands high standards, or high standards are demanded by the universities themselves is definitely under discussed, so any changes to REF because of Covid I think need to be talked about in terms of a cultural lens or an emotional as well as a technical one or an audit one if I can put it that way.
Helen >> and do you think that fed into what motivated REF to delay this exercise?
Mark >> I think it's a good question, I'm not sure that anyone looked at it from that point of view; they certainly looked at the fact that university research was going to change and had to change because of Covid and that the way the universities were going to run for short medium or long time is going to be radically different, and the such it would be unfair to continue without any change what is a very high pressured intense exercise but I would imagine that the Chris on the ground would have a much sharper view of it. Chris >> Yeah, I mean I think what's happened really is at the current juncture this is more of a pause isn't it, and I think that they're looking into a four month pause and we carry on everything just gets shunted 4 months. So, if the clock doesn't start again in July for a sort of putative 2021 March deadline then that becomes a delay, and a delay is slightly different from a pause. A pause is just the world freezing and then carrying on again. I think a delay would send lots of different messages back to universities about how this might proceed, there may be another consultation, there may be a splitting of the of the deadlines that the various bits of the REF. I think going back to Mark's previous points, the issue here is that universities deal with the REF as a as a compliance process but also something that could perform a useful function in the background. So whilst people are very happy to pause the compliance bit the idea that the impacts and impact support and how universities manage impact could be delayed, the issue you've kicked down the line somewhat is a little bit more problematic. For instance, universities might have tied internal funding structures into promoting certain things at certain times that fits into the REF, and you could say that's instrumental but the REF in its mutual sense just provides some arbitrary structure that people can then organize their universities around. So anything beyond this initial pause that turns into a delay could cause slight problems for how universities are planning impact activities in July, 18 months to 2 years down the line.
Mark >> I sometimes wonder though whether, apologies for the analogy because it might sound odd at first, is responding the Tokyo Olympics or the premiership football league in the downtime people unwind and the REF exercise whether it's a compliance exercise or something which is used more internally, requires a lot of efforts across departments where teams build up their expertise, that energy, and all of a sudden now it's paused and whether you can restart, just like that click of a finger after 4 months, whether they're going to be some issues about getting the energy back into the system, I'm really quite intrigued to see how universities will cope with that because as I said I think there's an emotional angle to this as well as just technical.
Helen >> What are institutions doing this pause, Chris, perhaps that's the question for you? Chris >> I think they're literally just pausing. The REF is 3 different things: it's Outputs, Environments, Impacts, so outputs, there's no push at the moment to change the cut-off date for outputs it's just the way the universities have to go around try to sift what the best research is, and then you submit it, you strategize around it, and work out where it fits. A lot of that can be happening anyway. Environment is a bit like the student's essay, something that always tends to get sort of done a little bit of a last minute. In a sense you have to because any work you put in ahead of time tends to get changed right up until the census point anyway which is a cut-off point for the state you need to submit. Impact is the is the tricky one because there are opportunities to benefit from any delay and universities would be foolish if they weren't thinking in those terms. There's always a case study that's, you know, slightly undercooked and actually if they did move the evidencing dates back slightly they could be viable case studies. So universities are weighing up all these scenarios. The wild card in all of this is you can never strategize around how much money you're getting to get because that amount of money changes. And part of the issue around a minor delay turning into a more serious delay is once you go beyond 3 or 4 months that time table for the money being doled out at the beginning of the UKRI's next financial year changes, so there are certain universities that are more worried about that than others, maybe universities didn't do so well in 2014 and don't want the 2014 formula to carry on for an extra year. So there are lots of different conversations going with institutions, you'll find the different bits of the institutions have different views on this. I thought it was quite interesting when the survey went round, and actually what the survey does, it tends to filter the views of various stakeholders in the university to the office that's managing the REF for the university, back to the REF team at UKRI. My sense is, the feedback they're getting there will be very different from just asking rank and file university staff academics, professional services what their views are. So the the process of collecting information trying to work out what the next step should be, creates its own strange dynamic which might not necessarily be representative of the sector as a whole.
Mark >> I think it's an extremely good point that by definition, all the exercises so far, the consultation exercises or the recent questionnaire, by definition has its own in built filter at every institution and a university is not a single organization in that sense, each cycle each department will have different views that will then get filtered into a single view of the institution half the time, and I think that is problematic when trying to work out what the effect of REF actually will be. I think the funding model as Chris says is going to be difficult to work out in the future, I think it it's well known that some research intensive universities have relied a lot on international students international student fees to bolster the research they do, taking into account that most founders don't pay full overheads, so if you have a funding model based on the 2014 REF, at the same time as its national student fees going down with no further the overheads, there could be a vicious circle brewing here which I think would be bad for everyone.
Helen >> So how can REF ensure that some institutions don't gain an unfair advantage, or some institutions don't suffer under these kind of different funding structures? Chris >> The answer there is they can't really. I mean there's certain things they can mitigate, I think the exceptional circumstances process will have to be revamped whenever the REF restarts anyway. One point which I think's been talked about a little bit on social media and it could explode if people aren't careful is the sense of moving the census dates because, if anybody doesn't know, the amount of staff you submit to the REF, that works out your formula and the amount of papers you submit, is related to the amount of staff in that unit of assessment on the census date, July 31. There's a view that universities won't look to restructure even though they can see a massive deficit in the future, maybe they've got an issue with recruiting Tier 4 students, as Mark said, or even home students, but they won't look to start making people redundant or moving people around until August 1. If you then go and move that date, it puts the date further forward that universities will have to make those decisions. So actually, maybe the answer is to move that dates and call universities bluff on some of this because some of the decisions will have to be made anyway once it's very clear what the students' intake is going to be.
Helen >> How likely do you think is that they will move it? Chris >> I suspect they won't. I suspect at the very at the very most, they might move the census state for impacts and keep the other dates the same, mainly because the particularly with the environment statement, you would play havoc on how work was being planned and worked out by universities anyway, and it would seem slightly unfair once you start moving census dates, you then start having to move other dates, people start writing more stuff, but are people writing more stuff in lockdown? I can guarantee there will be equality and diversity issues attached to that to who could be writing things in these times as well, so I don't think you want to be opening that can of worms. I think the sector is very well aware of some of the issues that could crop up to be honest.
Helen >> So as working for a funder Mark, do you want to speak about the impact of the delay of Covid-19 more generally on funding?
Mark >> As I said, NIHR funding is separate to the funding that's doled out based on the REF process so the effect for us is difficult to actually sit down and work out at this particular time. I think the issue with REF for every other funder is that it is a massive cultural change programme in some ways, and it affects how universities think about impact because it's the major impact assessment exercise every number of years so there is a concern I guess, that's perhaps too strong a word, if the delay is continued, if it's longer than what's already been discussed it means that peoples' focus, everything's on pause for the REF, we are waiting for Godot, we are waiting for REF, and it makes it harder for other funders to gain traction in conversations they have with the universities about the impact assessment exercises or work they may want to do. NIHR at the moment are engaged in a large excised creating a value framework; a framework that will help explain the impact of NIHR funding and there will be a level, a large amount of external stakeholder engagement with including universities up and down the country and that's going to be very important and hopefully very powerful but if universities up and down the country are saying 'yeah that's fine, but REF's next, we've got to wait for REF and get through REF first' and if that continues then that affects our work in that particular way. So we're not directly affected but we have always been indirectly affected by REF which is why we have always submitted every time there has been a consultation exercise for our view of what REF might look like, our view of how REF might move forward because I think, as another funder, by definition we are indirectly affected and I would suggest any major funder, whether NIHR, Wellocme Trust, any of the association of medical research charities would feel similar. Chris >> This sort of cuts both ways as well doesn't it, because NIHR, MRC, Wellcome, they are going to be looking to how best to use the funding they have to support Covid-19 related research and all the other bits of research that aren't directly related to Covid-19 but that emerge from it. It's the same people within that space who are the people submitting to Panel A in the REF, it's also the same people in that space, who are the people who submit to Panel A in the REF who have to get evidence for their case studies in Panel A in the REF. So the effects of Panel A are going to be slightly different from the effects of Panels B, C, D. I've had conversations with people, there have been mooting on the grapevine, that there are people in Panel A that want the whole thing shelved for a year, which you know on some level would be reasonable. But then you have the question of okay there's other people ready to go, some people aren't; do you still splitting main panels off and sequence them in a slightly different way? I think that's not feasible. Then once you start delaying a little bit longer the question comes in of, well you need to start including some of the impacts emerging from Covid research otherwise that's going to be emerging in REF 2027; we might be worried about alien invasion by 2027 and Covid might be something way in the past. But there's a worry then you start bolting that onto a REF process that doesn't include Covid-19, what does that mean? Does that mean that universities who happily have lots of biomedical research suddenly starts doing better out of the REF because in a sense they deserve to because they're doing this useful research but what does that mean in terms of some of the overall league tables, and how people use and abuse league tables? My personal view is you have to look at the impacts of some of the some of the research around Covid, even the really early stuff, but you have to sort of firewall it off from the main part of the REF somehow. Maybe have a section in the environment template or something like that.
Helen >> That must also be a consideration for you in social sciences because there's a lot of things that social sciences can speak to in research around Covid-19 from a social sciences perspective that could also be quite impactful? Chris >> Yeah and the issue there is that some of those projects are already started, the ESRC have already given various bits of urgency grant money out, there's impact accelerators giving money out to various institutions on projects related to Covid-19 like anything else, and it's similar to GCRF a few years ago where people are desperately trying to retrofit their research around it, whilst at the same time there are people with very little time and capability you actually want to be engaging in this agenda a little bit more. So the problem you have with a REF submission is, you got a bunch of people who were taken away right when you need them around to write cases, you've got people who had lots of things planned right to the deadline who suddenly can't do a whole bunch of stuff between March and July a lot of those will be in Panels, the Arts and Humanities...
Helen >> Yes, the impact on Panel D with the theatre closures... Chris >> Yeah, as much as anything else, and then you've got lots of people in between some of whom have more capacity than they had before and some a lot less capacity in terms of both research and their personal circumstances, so it is a bit of a mess and it is going to affect some - if you aggregate this out it will affect some institutions more than others because it's bound to; these things are always fundamentally unfair but I think it's a little bit like the questions around starting the football season again: Whatever you do and whenever you start this, the rules have changed. Yes, you can argue it's not a level playing field that's fine, but the rules would have slightly changed, and it won't be fair. Luckily the REF doesn't have promotion and relegation! So the question might be, are there ways of levelling some of this out in terms of how the money is distributed? If you remember last time, universities didn't really know how much money was to be distributed and what the mechanisms for distribution were, particularly around the differences between 4 star and 3-star research so you could do scenario planning around that. So those questions haven't been answered, so maybe you could give slightly less for 4 star recognizing that 3 star is good... I don't know but there are probably ways of post-hoc ironing some of this with the allocation mechanism.
Mark >> It might be that REF is not the best mechanism to smooth things out; actually, we might be asking too much of the REF structure to do so. We already know they're going to be shortfalls in the system because of Covid-19, we've already talked about the drop in international student fees, the issue with overheads overall, there's a debate going on about reducing or making it more difficult for students in England to go to universities in Wales and Scotland, which has caused a bit of an uproar. Using REF to sort all of that out, I don't think it's probably the best mechanism. REF should do what REF does and the government should find ways all of... patching is the wrong word but making things good elsewhere through other funding mechanisms. REF is not for Christmas it's for life; there will be another REF exercise at some stage, if we unbalance this one by building on top of it to take into account unique circumstances which hopefully by the time of the next REF exercise won't exist anymore, then we put an instability in the system for years to come. As far as possible REF should be REF and other funding mechanisms should take into account those shortfalls that Chris has so eloquently described because they are there, they're going to be painful and we need to be careful we don't shoot ourselves in the foot here by funding in a way that rewards a short term imperative but doesn't take into account long term talents and long term expertise. Chris >> I would agree with that and what you got to remember: universities have been planning for the REF for several years, but there's not a lot you can do in terms of changing the environments of your departments or producing much more new research so a lot of this is a scraping around the edges. There are not going to be universities whose REF submission is suddenly hauled below the water line because of because of Covid-19. It might be the case where it's 'all hands to the deck' to use another nautical term just to get the actual submission in, which is a different question around staffing capability. Mark's entirely right, there's different ways of allocating QR money, you know there was QR policy-related money that came through last year - I'm not quite sure how they calculated that per institution - but some of these are of course linked to HEIF as well as being linked to the REF formula. So there are different ways universities are going to do this, and it's going to have to happen because there's always R&D money that's allegedly going to be pumped into the system and I think just linking that to REF would be foolish and wouldn't be the way to go.
Helen >> Do you think and then as Covid-19 goes on and effects this current REF cycle what kind of lessons do we need to take from it about evolving, and what kind of lessons do you think REF can take to adapt into the next cycle?
Mark >> What a great question. I'm not quite sure. The problem with 'lessons learned' is that that tend to be the immediate reaction rather than someone looking back in a few months' time about what really we should be doing and what has been learned. I'll keep going back to the fact this is more than just an audit system: the effects on universities in a normal year of REF is... it's a hard thing to do in my view. I'm always in awe of people who undertake and support REF within university because I think it's an exceptionally difficult job. There may be lessons about lightening the load, so to speak, but even then, I'm not quite sure what that means in reality. There's always been talk about trying to make this much more of a toolkit or dashboard exercise where you hit a button and numbers come up and I would hope that's not a lesson that comes out of this because I think that's mis-learning. I think one of the positives about REF from my point of view is the impact case studies. They have been incredibly useful. I've taken them from 2014 and I worked on some of them because you can see how NIHR have funded some of that work because it's declared in those impact case studies. So I would hope that what doesn't happen is that REF changes to become a numerical audit, hit a button and get a spreadsheet response, which I know, as I say, has been discussed before. Let's not lose the case studies in any way we move forward.
Helen >> Absolutely. I think that it's clear to me that what the situation is really shown is how potentially fragile the ecosystems of universities are and how interlinked everything is. So, as you're talking about the number of international students potentially affecting that the relationship between REF and the international students, between funding bodies like NIHR.
Mark >> It is an ecosystem and when you shake one bit show effects occur elsewhere in the beast and that this is what I mean about REF is an exercise that needs to go forward and needs to be complete but how we move forward is a much bigger question than includes but is outside of REF. No-one really knows how we can recover from Covid. When a vaccine if a vaccine comes - there are therapeutics if a vaccine doesn't come. Whether the virus mutates... It looks like it's going to be a couple years before things get back to vaguely normal. We already know the moment that certain research has had to pause because you can't do face-to-face work so my understanding is ethics committees at universities up and down the country are not issuing ethical approval for some research not because it's wrong or bad at the moment it's just not possible to do which may have a knock on effect for the next REF in six or seven years' time. All those things need to work through. So REF is important but there's an underlying set of issues that need to be discussed, debated and solved, which I would hope our government is beginning to discuss with universities up and down the country as we speak through this podcast.
Helen >> We would hope so. Do you think that REF is actually able to give a realistic submission date right now? I mean with things carrying on for the couple of years and social distancing, working from home, can the REF submission process actually start? Chris >> Well this is a point we didn't make before actually and it was talked about the university's process and how that gets stymied by some of these issues but of course REF panels need to meet and they traditionally met physically and been sequestered away and hammered issues out in a small room for several days. That might not be able to happen for quite a while. Also, I think you'll find people pulling out of panel membership as well and beginning to be replaced which would cause an issue. So the wider question and I don't probably think it's one anybody's really in the mood to answer - is doing something like this every seven years, all at once really an optimal model anyway? As we said before, there is no necessary relationship... You may write an environment statement in a certain way, but there's no necessary relationship to impact case studies you submit that - let's remind everybody, the university decides which case studies make it look the best - so there's nothing particularly representative around some of this stuff. I think one of the sort of dirty secrets about impact case studies is sometimes a strong impact case study might not necessarily represent research, or a group of researchers who are central to how that unit of assessment operates. So there are certain issues that need playing out there. As Mark says, these things are very useful for seeing how funding has been used and the impacts of funding, even many, many years down the down the line, but it's not a very scientific way of working that out, and part of me wonders if UKRI decides it wants to look at the impact of individual investments in slightly more detail, that's going to have an effect on how the REF decides impact work in the future because you just start putting too much burden on institutions. One thing we found is that the burden from the funding councils has gone down in proportion to the burden of the REF going up, by and large. So it will be interesting to see whether the funding councils get the information that they want from this round because there were a few questions, there were also some very strange report saying, 'well, you know you get one REF star for every 500 spent by the British Academy but it would cost you 75,000 with the BBSRC for every REF star.' Ridiculous stuff simply because people weren't mentioning grants that were linked to impact cases.
Mark >> It does raise a very good question about why people undertake these sorts of exercises in the first place. On the impact side, outside of REF you tend to do impact assessment because you're accountable to somebody and you want to show that you're doing the right things. You do it because of advocacy. So medical research charities do impact sessions because they want to tell people who donate 'we're worth donating to. Give us some give some cash.' The analysis angle, what works and what doesn't, through an impact assessment but then the allocation: where we put the money. Now REF is a great big allocation exercise in that sense. Whether this is a time during the pause to actually think in terms of 'well, are we doing this just for allocation?' Should we be looking at impact assessment in a broader ecosystem sense? Looking at the analysis angle of what works in research? I don't know about that but you know the problem with REF being so big is that - it's that oil tanker analogy sorry, I use analogies far too often. It's got a direction of travel. I'm not quite sure it can change, even within the next seven years, radically because you know it's going in a certain direction. We just need to accept it. But I still think that this is a decent time have it a debate and a conversation about what is REF for overall? Chris >> Yeah, funders often ask for case studies but you understand why they're doing that because they want to show the value of their research and it links to research impacts and communications angles. The REF is not a very good mechanism for showing what the inputs are, how they operate in universities and what the outputs are. You've got a sense of what the inputs are, a sense of what the outputs are. The middle bit...? It's a little bit opaque. Obviously, there's all manner of gaming that goes on around it as well. The question then is could you move to a situation where you in effect had a permanent REF and there was something submitted on a yearly basis?
Mark >> The issue there is and you used the term there and it is a perfectly correct term to use - is it becomes gameable. Any form of audit or it impact assessment or review eventually becomes gameable, and that's just human nature. There's not much you can do about that unless you change the criteria on a regular basis, which means you remove the gaming nature of it but by definition you can't track back over ten, twenty years because you keep changing the criteria. So you're in this rather bizarre game where you can't win either side in that sense. There has to be a mechanism by definition in how we allocate money - this huge amount of cash, billions of pounds and REF is the way that we choose to do it at the moment. Some more thought about how it affects the ecosystem and whether we're necessarily asking all the right questions or whether, Chris, as you say, we try and change it. It's still REF but not as we know it. I think these are decent debates to have. From my point of view, I've always wondered why does everyone have to do it at the same time? So rather than do it over a seven-year cycle, it becomes some sort of yearly thing that you put in to. Is it possible to do this where you randomly choose fifteen universities and you have a really deep dive into those fifteen, and the next year you do another fifteen? So it's yearly but not everyone does it yearly. Would that come up with better or more interesting information?
Helen >> And would it prevent people gaming the system?
Mark >> Well possibly. What perhaps doesn't help is that allocation tends to be on a set budgetary cycle and therefore that doesn't work in that way, but to sort of 'impact purists' terrible term it just might be more interesting. It might be more interesting for the people involved; might be more interesting for the information you get. It just might be a better way of doing it. But we are where we are. It's the system. REF, KEF and TEF they are the games and that's what needs to be done.
Helen >> So do you think that this deadline should be extended or not? Do you think that the pause should become a delay, or do you think that they should stick to the 4-month delay window? Chris >> It's difficult to say. My personal view is they should submit outputs and environment in March and move impact further down the line. Whether they will do that, I don't know. I think there's enough evidence to suggest that certain bits of the submission will be unduly affected if we go for a March deadline and keep the July cut-off date for impact. You know, you could potentially keep the March deadline and move the cut-off date for impact anyway and still have the same March 2021 deadline. The problem, as we touched on before, is you've got two things going on with universities... There's a certain sunk cost fallacy are some activities. People want to do things in a certain way because that's what they've put their effort into and also a lot of people, they're not taking a strategic view, they just want it done as quickly as it humanly possible and out of the way so they can think about other things. To be fair that might actually be the right approach to this, but it might not be the right approach to all of the REF. So, I think there's probably a lot to play out. I think a lot will happen when they work out the panel A membership and whether those people are still committed to being in the main panel and sub-panels. Because if people start dropping out left, right and centre, it will take the rest of the REF with it, I think.
Mark >> Yeah, I think that's a fair point.
Helen >> I think that is a good place to round it up. I'd like to thank you both very much for coming on. I think that was a really interesting discussion.
Mark >> Thanks for having me. That was an absolutely fascinating discussion. Chris, look forward to seeing you soon after the lockdown. Chris >> Thanks, Mark. That was an excellent conversation. I think time will probably tell on how accurate some of our prognostications were.
Helen >> Thank you both for a fascinating chat.
Mark >> Thank you. Chris >> Thank you.