Are you in? podcasts

Are You In?: How close are we to reaching more responsible research assessment across the globe – part 1 transcript

Florence Theberge, Emerald Host (FT): Despite growing support for reforms to existing research evaluation approaches, changes in practice are slow and incremental. This is in part due to the complex nature of the task which will require coordinated actions from multiple stakeholders. In this 2-part podcast, we will first discuss the current challenges to the research assessment system, and the initiatives behind the call for change. We will then explore the tools and frameworks available to implement these changes and the roles of the various stakeholders involved.

To lead on this discussion today, I am joined by a Panel of experts on the topic of Research Assessment and Open Science.

Dr Elizabeth Gadd is a Research Policy Manager at Loughborough University, with an extensive research background in scholarly communication and open research. She chairs the International Network of Research Management Societies Research Evaluation Group.

We also have with us Dr Noémie Aubert Bonn – an early career researcher who is currently undertaking post-doctoral research at Hasselt University and Amsterdam UMC where she pursues her PhD work exploring different stakeholder’s perspectives on the impact that research assessments have on research practices and integrity.

Finally, we are joined by Dr. Dasapta Erwin Irawan, an assistant professor in hydrogeology at the Institut Teknologi Bandung. He is actively participating in promoting the open science movement and the importance of science communications to Indonesia’s research ecosystem.

FT: Thank you all for joining me today for this podcast on research evaluation. It is a real pleasure to have you all here and discuss this topic which is fair to say has raised a few debates over the years.

Indeed, there is a general understanding that the current research assessment processes are not fit for purpose and may in fact hinder the research landscape and a researcher career progression. Starting with you Lizzy, could you explain what are the main issues with the current research assessment practices?

Elizabeth Gadd (EG): Yes, thank you Florence and thank you for inviting me to speak today. There's a lot; how long have we got? [laugh] I think there are 4 kinds of main issues really and I've written about this. They're all kind of interlinked issues and I think the first issue that we have with research assessment is what I call the construction problem. So, the design of our evaluation approaches is often really poor. So, we've got journal impact factors and they take the mean of a skewed data set. You should never take the mean of a skewed dataset so they automatically kind of unhelpful indicators. University rankings, they use composite indicators without any justification for the various weightings of the different elements of that indicator. They use sloppy surveys that wouldn't pass peer review. You know there's all these sorts of construction issues with our evaluation approaches. So, from step one there are poor and unhelpful data sources that are used, are heavily biased towards the Global North etc. etc. So there's construction. Then the second issue for me is the validity problem. So, the approach that's used is poor. Yes, it's constructed poorly but it's then used as a poor proxy for the dimension that it's claiming to represent. So, a journal's “citedness” and its impact factor is not a valid indicator of the quality of an individual article within that journal or of an individual researcher. So, the validity is wrong. The rankings are largely based on research data and they seek to identify the top universities in the world. Well top at what? What if you're a teaching university? They're not assessing which are the best teaching universities because they don't really assess teaching qualities. So, there's a validity problem. The third issue is around application. So, it wouldn't really matter having these poor evaluation approaches if they were just kind of back page curiosities in the research press. But of course, the problem is that they are then applied to individual researchers and to universities etc. to allocate funding, to give out jobs, and in the case of global rankings now we've just had the announcement in the UK that they're going to be used to provide people with no questions asked visas to the UK. So, there's poor application of these measures and then that leads to what the fourth problem with our current research assessment practices to my mind which is the impact problem which is essentially the downstream effects of all of those issues construction, validity and application on the diversity of our research communities, the favouring of the Global North, the allocation of funds to already favoured people or entities rather than the best person for the job, etc, etc. So, there's a lot of issues and I think often they're not kind of teased out in any systematic way but I think it's helpful to because people can perhaps knock down one of these arguments. “Well, we know the construction is poor but they're the best we've got” but it's not just the construction that's the problem, it's then the validity, the application, the impact of those measures that we need to kind of fully understand.

FT: What about you, Noémie?

Noémie Aubert-Bonn (NA-B): What I can add is what I heard from the perspective of researchers. Through my research I did interviews and focus group with researchers to understand what are the problems of research assessment. And what I found was a bit less defined but I found issues for example in the fact that we only assess the outputs of research. So, there's almost no attention given to how the research is conducted, to any of the activities that are part of researchers’ everyday life but are not part of the outputs of their research. That's completely left out of research assessment. Another problem that I found was that to be successful in their careers then researchers face this kind of tension where they feel that they have to respond to these indicators which have the construction problems, which are not necessarily valid, which have issues of application like Lizzie mentioned, but in responding to these indicators researchers feel that they undermine the quality of their research, and that they're not able to perform research with the highest research integrity. And finally, Lizzie already mentioned that there's issues of diversity. Well, this is a problem that really came up quite often in my work where current research assessments are very uniform. So, they expect everyone to fit the same profile, to have a very similar CV and this is the CV of a successful researcher. There's a profile for a successful PI and there's no diversity in that. So, you have to publish a lot, you have to get a lot of funding, you have to be a leader but there's no real place for team members, for diversity of skills, for diversity of even past careers, past experience etc. It's a bit like trying to have a football team and the only people that you hire are the strikers. So, we really have a problem in the way that careers are being organized in science.

EG: Florence can I come back on that because I think Noémie makes some very good points there. I read something yesterday in the press. It's around the Chinese universities now starting to pull out the university rankings and a Chinese colleague described it as “our current research assessment practice is as cutting the feet to fit the shoe” and I thought that's exactly what's going on here: we're cutting our feet to fit the shoes rather than cutting our shoes to fit the feet so we're not kind of measuring by mission, not by all the sorts of things that we value as Naomi so well-articulated there, but around this kind of poor standard, and then seeing you know, hacking away at our research careers and our research creativity and our research’s diversity to kind of fit this very poor limited shoe that we've designed, although that was designed about a hundred years ago you know and we are still using.

FT: And what is the situation in Indonesia, Dasapta?

Dasapta Erwin Irawan (DEI): So, as we speak right now, I am following a webinar held by our Ministry of Education saying that all of the indicators of the universities in Indonesia should follow or should agree with the criteria that has been used by most of the major university ranking in the world, right, such as QS, or Times Higher Education ranking. So, our ministry has mentioned that statement, and it has been going on maybe, maybe more than five years now, I can say, it's 10 years, right now. Overall, the criteria of the advance University in Indonesia are not based on the principle of science. So, the principle of science is not very much in front of the conversations. Most of the conversation will be criteria or the output criteria that has been used by the university ranking. So, I think that would destroy the critical thinking and also the value of science itself, because the conversation are not really on, on the science, but only on the, on the end products, such as international papers published in international journals, in reputable journals.

Another point is, since Indonesia is not an English-speaking country, although English education has been given to many of the children since early age, but we are not English-speaking countries. So, it's really strange if those papers is separated based on the language. So, for instance, if I wrote, if I write my research, in Indonesia, it will be given a lower score, then if I write papers in English, which is all of those reputable journals are asking the authors to write in English, right?

FT: Those are all very interesting points and I love those analogies that I am sure will help our audience understand the extent of the problem. But, despite these issues there seems to be a momentum for change. In 2021, the European Commission published its scoping report “Towards the reform of the research assessment system”. That same year, the UNESCO recommendations on open science was adopted by its General Council and soon afterwards the Paris call on research assessment advocated for an assessment system where research proposals and researchers are evaluated on the basis of their intrinsic merits and impact rather than on the number of publications and where they are published.  

So, Noémie, would you be able to elaborate on those initiatives that are behind this momentum for change and what those initiatives promote?

NA-B: So, I think, I think we're seeing now that in the past ten years - I would say 10 years it's more than that - but let's say from the start of when DORA came about and started obtaining signatures for the DORA statement. We've seen more and more awareness of the imperfection, the big problems behind research assessment and how we really need to change this. So different countries, different research institutions adopted new methods of assessing researchers but that was very slow. There was a lot of action, a lot of documents being put out, a lot of statements, manifestos etc. but the change was very slow and there's a few reasons for that. On the one hand, there's the first mover disadvantage: any institution who does the first move may risk losing status or recognition internationally because they change their assessment methods and then their outputs are not the same and there's always a risk there. There's also a risk of the researchers losing mobility opportunities because their profile is very different. So, there was a bit of fear in there. And it's also it's a systemic change. It's something that does not just imply research institutions or publishers or funders, it's something that's really multi-actor. So, it requires the coordination of a lot of different stakeholders together. So, in the past few years and I think this is a very very positive change. We've seen more international collaboration, more consortium that put together different stakeholders to move these statements that we've built in the past ten years and put them into action. So, you've mentioned the EU reform for research assessment, the UNESCO. There's also the G7 that just started another group as well, the Global Research Council. So, we have a few of these international initiatives that are really moving things forward. The one that I'm most involved with at the moment is the EU reform and the agreement that is being put forward. They will release the agreement very soon probably by the time this podcast is out it will be released already. Their goal is to create a coalition of different stakeholders that help each other and support each other to change their research assessment. So, it’s really about creating mutual learning, creating support because changing research assessment is very difficult. It's resource demanding. It requires a lot of effort and courage. So, it's putting these things together to help move the commitments that we've agreed on towards action.

FT: Dasapta, are those initiatives widely promoted in Indonesia?

DEI: I don't think DORA and the Leiden manifesto is known by general researcher in Indonesia. So, I can give you example, if I write a grant, the funder has already clearly written that the end product must be papers in reputable journals, in a prestigious journal, right? So, as a researcher, I write a proposal to comply to what funders request, right? And then it's been going on and on and on. So, the focus is not the science, but the end product as I said before. While DORA and Leiden Manifesto promoting something that that it might be looked strange from the public in Indonesia, because the statement is not based on journal right the assessment is not based on the journal where the science or the papers is published. It's not very common in our way of thinking in Indonesia. So, I can say that DORA and Leiden Manifesto is not known in general research ecosystem in Indonesia.

FT: Lizzy, do you have any additional points based on what we just heard?

EG: Yeah, just a couple of observations on this. I think it's fantastic for the work that's going on in all of these spaces. I suppose I am just slightly nervous; we've had a lot of kind of European action in this space often driven by Open Science and that is great. I know the latest Towards Responsible Research Assessment Coalition is broader than the Europe. They are inviting all actors but I think I was just slightly concerned that there is this sense that again it's driven by - this particular initiative, and others like it - are driven by the Global North again and that the Global South I've heard the view that they feel as though they're just trying to catch up on the traditional measures and then the goalposts are being moved again by the Global North, ostensibly in the Global South's favour but it's a bit. I think these initiatives really genuinely do have to be international in focus and include all of the players and all of the cultures because they are so broad and wide ranging. That's why I love the Global Research Council work because it really genuinely, it does have a proper engagement from a very wide range of nations and cultures and approaches and it’s difficult, it's really difficult to kind of get agreement but I think that international agreement beyond just one region is going to be really important. As I say, I know it's great that the European work is happening and I know that they're looking to expand but I think there is this “not-invented-here” thing that we are all a little bit prone to. “Well, I didn't come up with that, I wasn't involved in that design” and we do actually have to kind of ensure that everybody is included in those conversations. The other concern that I have I suppose around some of this work is that it is been driven by Open Science and the move towards Open Science which is a good thing. We want Open Science and yes poor research assessment is inhibiting the adoption of open research practices but we can't just improve research assessment in order to enable research, open research practices. That leads to all sorts of additional problems. To my mind you know, open research is a hygiene factor. We should all just be doing it anyway, like adhering to proper ethics systems and health and safety standards. Trying to kind of incentivize Open Science through moving away from Journal Impact Factors and moving towards Top Factors, you know transparency and openness factors in journals, it's just kind of maintaining the status quo really instead of looking for cited journals, we're looking for open journals, instead of rewarding “citedness”, we're trying to reward openness and I think that could potentially be a little bit problematic if it's not kind of exposed. It's great having open research as a driver for more responsible research assessment practices, great, but we have to just look beyond that and think very carefully about the kind of long-term consequences of just trying to improve open research and engagement with open research through responsible research assessment because there are bigger issues at play here.

FT: Before, I go any further, Dasapta, earlier you mentioned the issue of language bias. Could you explain in more details this language barrier and how this may lead to Indonesian researchers being discriminated by current national research assessment policies for publishing their findings in any language other than English?

DEI: So, I would see this matter in from two point of view, right, in the point of view of the author as knowledge, to produce knowledge, right, as an author, we are pushed to write in English, which is not our first language. So, that will be strange if, for instance I have research in flooding in a nearby village but the people that read my paper first-hand is not the people that suffer the flood itself, but maybe some peer reviewers that live in London. And then the other one is from the point of view of readers. So, I did some informal research throughout the years to my students, undergraduate to PhD students, I asked them what keywords what language that you use to type your keywords in looking for literature, right, and surprisingly they say English. So, this is also strange because the study area their study area are mostly located in Indonesia. So, local researchers using Indonesian language, we should find those papers right written by Indonesian author in Indonesian language but they missed those papers because they only use keywords in English because our university keep promoting to use certain databases, commercial databases that they buy with a lot of money, right, so that's the problem with the language barriers.

FT: This concludes the first part of this podcast on research assessment. You could find a transcript of our conversation and more information about our guests on our website. Please stay tuned and join us for the second and final part of this podcast where we will discuss the tools and frameworks available to implement the changes to the current research assessment systems and the roles of the various stakeholders involved.

Are You In?: How close are we to reaching more responsible research assessment across the globe – part 2 transcript

Florence Theberge, Emerald Host (FT): Thank you for tuning in to the second part of this podcast on research assessment where I am joined by Dr Lizzie Gadd, Dr Noémie Aubert-Bonn and Dr Dasapta Erwin Irawan to discuss the challenges and opportunities for change to the current research assessment practices.

So, in the first part of this podcast we’ve discussed the responsible metric principles that are advocated in DORA or the Leiden Manifesto, among others, but institutions often struggle to apply those principles to design alternative research evaluation mechanisms.

Lizzy, through your work with the INORMS Research Evaluation working group, which I’ve mentioned earlier, you’ve helped developed tools to support institutions in their effort to design better value-driven research evaluation process. Could you tell us more about this work and others’ work in this field?

Elizabeth Gadd (EG): Well, this is exactly where the INORMS Research Evaluation Group have been turning their attention in terms of the design of a framework for responsible research assessment called SCOPE. SCOPE is an acronym. It's a five-stage kind of process although it was very iterative obviously. It's a mechanism for designing new or testing existing evaluations to see if they adhere to principles or responsible metrics. The first stage is firstly “S” start with what you actually value about this thing that you're evaluating as so often we start with what data have we got that we can use to measure- the streetlight effect. Why are you measuring here? The person has lost their wallet and they start looking under the streetlight and somebody says, “why are you looking for your wallet here?” and they say, “well, this is where the streetlight is!” rather than where they actually lost it. So, we start with the data we have rather than with the thing that we actually value about the entity that we are evaluating. So, then that's a long step. It's really hard. We don’t often sit down and think about what it is we actually value about research. We just kind of launch in with our bibliometrics or how much money we've got etc. So, the second stage of SCOPE is “C” contexts in which you want to evaluate. So, considering all of the different contexts in which you want to evaluate. Because often we have these conversations about “Is the journal impact factor a good measure or isn't it?”. Well for what? Who or what are you evaluating and why are you evaluating? If you're just looking to use bibliometrics to understand country's output production over the last fifteen years, actually bibliometrics is quite helpful for that. If you're looking to assess which of these 2 candidates are best for a job, bibliometrics could be extremely unhelpful for that because there's all sorts of reasons at that level of granularity why one individual researcher might have more citations than another even within the same discipline. So, it's teasing out what context in which you want to evaluate and making that part of our discussions around what is what isn't responsible research evaluation. The “O” of scope is then looking at your options for evaluating. So, you're not starting with your options “I want to see which is the best university. What options have I got?” There were options in stage 3 of the scope framework and it's including both the quantitative and the qualitative measures that we have for evaluating the particular entity in that particular context given our values. So, it's encouraging, but as a rule of thumb only use quantitative indicators for quantitative things like students and money and citations, and qualitative indicators for qualitative things you know, excellence, value, impact those sorts of things. Then the “P” of Probe is really important and this is where a lot of our research valuation falls down. And that is to probe deeply the evaluation approach that you have designed. We encourage folk to look at what might the foreseeable unintended consequences be because it's not always easy to predict those. But to look at who might systematically be discriminated against by this measure that you've developed. Year on year is the same the same group of people, the same demographic going to be rewarded here, to look at the cost benefit. To design the perfect research valuation approach, which doesn't exist by the way just for avoidance of doubt, to design the very very best quality research evaluation approach takes a lot of time and money. What actual benefit are you going to get from that evaluation approach? So, there is a kind of a cost benefit exercise to be done at this stage. And then the last stage of SCOPE is to “E” evaluate and then evaluate your evaluations. So, you're not just kind of thinking right done and dusted move on next project but you're looking at it systematically over time. How is this working for us? Do we still value those things? Is this measure systematically discriminating against a certain group that we hadn’t foresee? So, it’s that kind of evaluation approach is being live and being a living thing and not just a kind of one-off process. And SCOPE works under 3 principles which I think are really important and the first is to evaluate only where necessary so we know that research, researchers, research institutions are over-evaluated at the moment and it's causing mental health difficulties. It's causing all sorts of problems and we measure because we can often. And individual researchers are just as guilty at this, right? You know, we measure for pleasure as I call it. We like to see how big our H-index is and how we compare with our peers and our colleagues and we like to see how we're doing relative to the rest of the world in SciVal and all those sorts of things. We don't always have to evaluate. Certainly, if we're trying to incentivize activity and I think open research is a good example here if you're wanting to incentivize open research, you don't have to measure it. There are other ways of incentivizing open research. So, evaluate anywhere necessary. The second principle is to evaluate with the evaluated. This principle of co-design of evaluation approaches and co-assessment and co-interpretation of those assessments with the evaluated communities. So, we're not doing evaluation to people or to institutions but we are evaluating with the evaluated. It just makes it a more meaningful outcome. The third principle is to evaluate with your expertise. So often as I mentioned earlier, we develop evaluation design approaches which just would not pass peer review. They would not pass an ethics process and yet we are academics, we have research qualifications, we know how research works, we know what good assessment of various forms, given our different epistemic differences, looks like. We should not leave that at the door neither should we assume that just because we are good at physics that we are therefore qualified to do a good evaluation. We have to involve those that have the necessary skills in evaluation to deliver it. So, that is SCOPE and I 've used it in all sorts of settings all over the world. It's being used by the Research England and the joint funding bodies in the UK to redesign the research assessment exercise. It's being used all over by small institutions, individuals and larger funding bodies too, as a kind of high-level framework around the stages that we really need to include if we're going to have half a chance at doing this well. The other kind of tool that's out there is SPACE which is developed by DORA and obviously I'm not so qualified to speak about SPACE but that is a system, a framework that could be used by individual institutions to self- assess and see how ready they are to evaluate responsibly and they can kind of plot themselves on this kind of matrix and see where they might want to improve their evaluation approaches.

FT: So, we talked about the initiatives behind the call for change for a more responsible research assessment, and the frameworks available to design better value-driven research evaluation processes, but how would the implementation of those various initiatives look like in practice, starting with you Lizzie?

EG: It depends because there's so many different entities that want to evaluate and things that they want to evaluate but the way that we actually make it work, the way that we've kind of workshopped this and developed it, I suppose, is to work with institutions and organizations to firstly, I suppose focus on this values piece. You may be familiar with the Humane Metrics initiative which is a fantastic initiative. Again, another global initiative which has got European and US participants where they have really developed a fantastic methodology for understanding helping individuals, understand what it is they actually value with a focus on the Social Sciences and Humanities. But this kind of piece of work around what it is we do actually value about the thing that we're evaluating is a lengthy process. And the way that we kind of get to the bottom of that with the SCOPE process is to get a group of those stakeholders together, those who are being evaluated, those who are doing the evaluation, and to try and tease out what it is we actually value about that thing. And we have super values which might be a single word like “collaboration” or “creativity”, and then we have kind of values which is how those supervalues play out, and then we might have some subvalues and some very specific kind of practices or processes that are important in the embedding of those values. And we kind of just get to the bottom of that question first, then we think about you know, the context, and usually that is known by the evaluator. They know they want to create a KPI or they know they want to identify individuals for funding and those sorts of things but having that very clearly stated. And then you move on again with that same group or with another group of stakeholders representing all of the parties, early career, different demographics, the different protected characteristics, etc, etc. to actually explore what the options might be for evaluating that value in that context, and just really exploding it out. We have a conversion/diversion model, what's the art of the possible in this space? And then we kind of narrow it down again to what actually feasible given our probe stage, I suppose around what the cost benefit might be, and what tools are already out there. This is, we have to make it work. So, the Probe stage is next and then you kind of you do your evaluation and run a final evaluation process. So, looking at, how did that work? Did we meet our evaluation aims there? Who was discriminated against in the end? Or I hadn't foreseen that, and then you kind of cycle back around again. So that's how that's how we make it work in practice.

FT: Noémie, do you want to add anything?

Noémie Aubert-Bonn (NA-B): I just want to come back on a point that Lizzie mentioned about… Well, I mean, first of all I'm a strong supporter of SCOPE. I think it's brilliant. And the way it's done is so concrete. I mean, anybody could follow it. It's a procedure. It really brings the assessors to reflect on why they're assessing so this is extremely important. It's one of the things that I found at the end of my thesis when I conducted all this work on research assessment, on the potential problems of research assessment, then I realized but why are we assessing? I mean, what do we want? What do we value? Do we want to create capacity for researchers? Do we want to create individuals who have strong research skills? Do we want to impact society? Do we want to advance knowledge? Do we want to create knowledge with high research integrity? Do we want to create quality of knowledge? This is not even clear in the type of assessments that we have and I think the SCOPE can bring that up in the assessment process. So, I'm really in favour of using it. In terms of how it would look like to implement these things in practice. Well, I think it's a gradual process always and it's something that will come little by little. It's not that from one day to the other you change all your assessment procedures. It will require training also of the assessors so that they understand how to do it because one of the big problems is that the assessors are not necessarily trained on how they should assess people, and quite often the assessors are us. It's the researchers. So, then we're facing a new assessment method then we're not sure what to do. And so, I think it's really important to consider the aspect of, we need to provide training. We need to provide support, awareness raising, so people are aware of how these changes should be implemented in practice.

FT: Dasapta, from what you said previously it seems that Indonesia is still a long way from moving away from more traditional research assessment processes but perhaps there are some signs that this is changing? For example, looking at your own work, as a researcher, you are using a whole range of media to promote your research?

Dasapta Erwin Irawan (DEI): In our assessment regulation, including assessment of research grants, assessment of promotion like me. I am in the process of promoting my rank into Associate Professor right. So, the regulation is clearly state that I need to have at least two papers in prestigious journal. And what is defined as prestige, one, is the journal must be indexed in Scopus or Web of Science, that is no question ask, you only have to publish in a journal that is indexed in Scopus and Web of Science. And the second one is the journal need to have metrics you can use metric from journal impact factor and also from SCImago Journal Rank. So in my application I keep saying that…Because I cannot control people to not use current metric, right and to not use current way of our government, setting up the regulation in measure in assessing research, right? I cannot force people to do that. But what I can do is try to widen their mind, widen their perspective, that even if you go with those metrics, you need to have more work or more ways to promote your papers is right itself, right. So, for instance, if I aiming to publish in Nature, I need to have more diverse way to promote the Nature paper to a wider audience, right. It's kind of my responsibility to the society. So, I introduced the term narrative CV or translated articles is also research output. So, researchers need to do more work just to deliver the message, right. However, those alternative products it's still not going to be used but at least I need to promote it as way for the research itself to gain more credit and more acknowledgement from common public, common people right, right? So, they can be more famous than they already are. Because if you publish in certain journals, you maybe, you are only famous abroad not in Indonesia, right? And the last thing in my application, I need to promote additional work that that would require less time for the researcher to work on, right? So, for instance, if, if my paper is published and those press journal, actually, I have nothing to lose to do another additional work, right? Because the paper is published, right? But I need to have additional products that don't consume more time, of my time, right? So, I propose several simple ways for them to promote their work in more popular way in Indonesian language such as blogging, such as maybe drawing cartoon in collaboration with local artists or lecturers that work in arts department, right? You just have to say something and record it and then publish it online.

FT: So, we talked about the initiative behind that momentum for change with regards to research assessment. We also mentioned the framework and the tools in place to help with the design of alternatives and example of implementations. Now, I’d like to explore a point that came up a few times already and that is, is multi stakeholders enterprise, one in which various people, sectors etc. etc. need to be involved.

Noémie, you are currently involved in an EU-funded project on promoting excellent research. What have you found out about the role of the research environment in fostering responsible research and research integrity?

NA-B: Thanks, Florence, for this question. So, yeah, I'm part of the SOPs4RI project which means Standard Operating Procedures for Research Integrity. So, it's a very broad European project and we're trying to build a toolbox. Actually, we built a toolbox to help research performing organizations and research funding organizations promote better integrity in their organization. So, we provide some resources some existing guidelines to help them change some of their procedures to foster better research integrity. And one of the first steps of this project was to look at the kind of topics that were important to foster better research integrity. And we realized that to foster good research integrity, you don’t, you have to look way beyond research misconduct and questionable research practices because there's a whole array of other elements that come into play into research practices and that influence the way that researchers work. So, we actually found a whole broad array of different topics that are really important for all these organizations to consider and research environment is one of these core topics. It has a huge impact. So, for example, researchers when they learn about science, when they learn about success as well and about, what is successful in science, well they are in an environment that has a specific research culture and that shapes the way they see science and they perform science. So, for example, the way that they interact with their mentors and how their mentors teach them about success will have an impact on that. Also, how inclusive the environment is, do they feel safe? Do they feel included? Do they feel part of that environment that has a huge impact on the integrity of their work? It's the same for the level of competition. So, we know for example that quite often now there's precarity for early career researchers and that yields to, that leads to very high level of competition between young researchers because there's only 1 out of 10, or 2 out of 10 that will able to continue in academia. So, this has an impact on the way they conduct science. The same for the support that is available to them or the skills that they are able to build in their degree as young researchers but also as older researchers as researchers a bit later in their career. Do they have opportunities to build their skills? So, all of these things are elements that we captured in the toolbox of the SOPs4RI project and we provide guidance on how to foster better standard operating procedures in all of these areas. Why is this relevant for research assessment? Well more and more we're realizing that all these elements could be captured in some ways in research assessment at an institutional level. But they're also very important that we build these healthy research environments so that researchers perform better research and they feel that the environment is allowing for them to perform. We really have to put that together to make the environment supportive for the research, the assessment supportive for the research and then in that supportive environment, researchers will be able to conduct good science.

FT: Dasapta, did you want to add to this point on multi-stakeholders’ involvement in the reform of the research assessment landscape?

DEI: If I may say something for the, to the publisher community. I may suggest use a more divers way to promote their products. They can help me at least to give more perspective to our community. So, not only saying “look, our journal has journal impact factor, maybe above five”, not just saying that but also build the capacity. Because we know that Indonesian people they trust, they put more trust from people outside from Indonesia, right outside of their peers, then they're listening to fellow Indonesian. So yeah, I think that that's the role that a publisher can do in the future.

FT To conclude our discussion, what do you think are the next steps in the quest for a responsible research assessment landscape? Starting with you, Lizzie.

EG: Yeah, I mean, I think we're probably all going to have exactly the same answer to this question, which is that, you know, research is an international endeavour. And we therefore need to agree collectively and globally, what good research evaluation looks like. And bearing in mind that will be local differences. But I think for those, you know, researchers are highly mobile, and just getting some sense of agreement, perhaps around the core elements of, where expectations on researchers globally, so that we can perhaps design a kind of a skeleton standard narrative CV, or particularly a better way of assessing at the university level. So, we definitely need some kind of alternative to the rankings. And obviously, some of the work that I'm involved in with in the INORMS Research Evaluation Group, is to consider, how we can help institutions to provide a more narrative contextualised description, qualitative description of what they have to offer the world that goes beyond what the university rankings might seek to measure, called “More than our rank”, and that's something that we're working on at the moment and we're hoping to launch that in the autumn. But kind of taking that one step further, could we have some kind of narrative CV for universities? Could we agree what the core elements of a, or even a selection of different types of higher education institution might look like? So, teaching universities, or more technology-based universities, or research-intensive institutions. There's wide range. We think of our universities are largely the same, but there's huge variety. Can we develop some kind of templates against which those looking to seek to work at an institution or study at an institution or fund an institution can assess the office of those institutions in a kind of more qualitative, perhaps, or combined with quantitative data way? Obviously, I'm aware, fully aware that there are issues with peer review. We hold up peer review as the gold standard of research evaluation, but it's hugely problematic. There's a lot we don't understand about it. We know there are inherent biases. There are language issues with kind of narratives CVs, et cetera, et cetera. But there must be a better way I think, of assessing both researchers, research groups and institutions at both qualitative and quantitative ways. And could we develop some kind of agreement globally as to what those core elements might be? I think that'd be a huge, huge step forward. And then the other thing that I would say in this space. So, we're currently doing a revisit of The Metric Tide report, which was published in 2015, just after the REF 2014, to consider the question of the use of metrics in research assessment at institutional level. And one of the issues that came up in one of the roundtables yesterday, which I thought was a really pertinent, pertinent and important point is, you know, we know we want to move away from shortcuts like the H-index, like the journal impact factor, but we are seeking alternative shortcuts at the moment and where everyone's saying well qualitative indicators that’s too lengthy, qualitative approaches are too difficult to assess, are too time consuming, etc. But perhaps we have to accept as a planet but that good research evaluation does actually take time and give it the time and the resource that it needs in order to make these important decisions particularly where funding is involved, where reward or allocation of some description is involved. Perhaps we just need to accept that and give it the time that it needs.

FT: What about you Noémie? Any final comments?

NA-B: Well, I also agree with what has been said about the need for global agreement or at least a discussion that involves everyone that is not just us in Europe, that is not just America, that is not just the researchers or just research institution or just the funders, but really a discussion that involves all the voices, including the voice of early career researcher and I always make a point to put that in there. And I think we need to join forces to have something that puts us on the same line so that we know that when we make change it will be agreed on a global level. There's something that everybody is aware of, and that in that way, I think we will move towards change in a much more efficient and more sustainable way. I also liked the point that Lizzie raised about having concrete tools to help us do that. I think this is missing sometimes. We mentioned the SCOPE, we mentioned the SPACE, but these concrete tools are very important in shaping the implementation of more responsible research assessment, giving good example so that we can learn through mutual learning. So, I know, for example, DORA has the project, Tara, I think, that will collect some experiences from different research institutions, I think this is very valuable for implementing the change. And it's something that needs to happen right now. But I want to take a step back a little bit from this whole grand transformation reform, to also the researchers level, because as a researcher, I also see that I have a responsibility to change how I talk about success. And this is very small, it's very incremental, it's something that I do in all of my research practice. So, for example, when I talk about the papers that I publish, do I mention the journal name? Or do I actually mention what the paper is about? So, these types of little things that we do in our everyday life, they really are at the basis of the culture of research assessment, and I think we can make a change in that. So, we're, we created this culture of research assessment together as a community. We are part of the problem, and we are dealing with the problem, but we can also all be part of the solution. So as Lizzie mentioned, there is this tendency to measure for pleasure. Now, there's metrics on every little platform that I'm on, for example, ResearchGate, even Publons had metrics to compare my peer review with different researchers in my institution, we really like to measure for pleasure, but I think we need as researchers to also reflect why we measure how do we understand the metrics that we're using, and how we speak about success.

FT: Dasapta?

DEI: I don't see the change is coming in the coming years, because I can see that all the tools and resources are still mainly going to those metrics, but what I can see is the generation in the government is changing, right? So, another generation is, is born and going to take the responsibility as leaders in the Ministry of Education. So, I think I need to talk to those generation not the current generation that that is in taking the role, right. But in all webinars, all meetings held by the Ministry, if I have time, I come and I cannot say the regulation is wrong, right. But I can say that the regulation is too narrow to support the mission and the vision of the Ministry itself to going international. Because being international doesn’t mean English, right? It’s not, it doesn't mean Western Europe. It doesn't mean United States of America, right? Being Indonesian, spreading the science from Indonesian as Indonesian also is an act of international people, of international member, of international community. Let's be international, true international without unnecessary barriers. And one of them would be language. So, language is our only our way to communicate, communicate like this. We are doing right now. But language is not a criterion to assess research.

FT: Lizzie, you wanted to say one more thing?

EG: It was just making me reflect of a blog post written by Tal Yarkoni a few months ago called “No, it's not the incentives it's you”. And it was kind of pointing out that, you know, we can very easily blame the incentives in science to use steel others research and to publish in journals with questionable policies and do all sorts of activities that are just “well, it's just the incentives, isn't it? That's why I'm doing it”. But actually no, we do have a set of principles as researchers, we do have a set of responsibilities as researchers to engage in ethical and responsible ways and to make our own decisions. We do have autonomy. I'm not saying the incentives are not poor, because they are poor. And it does take a bit of a battle but certainly at the kind of grubbier end of poor practice, you know, not everybody is engaging in, you know, stealing first authorship of PhD students and some of these kinds of other very, very unhelpful and unethical practices. So, I think we do have to own our sense of responsibility and a sense of personal integrity, as we're doing, engaging in research, you know, that's a fundamental principle of being a researcher is to you know, have high standards for oneself, and to do the things that are the right things to do not always the easy things to do.

FT: Thank you all so much for your time and for this insightful conversation. I very much enjoyed it and it is very uplifting to hear about all the global initiatives currently in development to ensure a fairer and more inclusive way to evaluate research work, research environments and researchers.

Podcast Conclusion

FT: Thank you for listening to the second and final part of this podcast on research assessment. You could find a transcript of our conversation and more information about our guests on our website. This episode is part of Emerald’s recently renewed impact manifesto called “Are you in?”. You can find out more about this initiative by visiting the Emerald Publishing website. I would like to thank our guests, Lizzie Gadd, Noémie Aubert-Bonn and Dasapta Erwin Irawan for taking part in this conversation as well as Alex Jungius at This is Distorted.

End of Podcast – Part 2