Impact in education through embedding research into school-level practice podcast

Impact in education through embedding research into school-level practice – transcript

Daniel Ridge (DR): Welcome to the second episode of a brand-new collaborative podcast miniseries between emerald publishing and the Jacobs Foundation. In this episode, we hear from four experts about achieving impact in education through embedding research and evidence in school level practice. In the face of radical societal transformations, learning settings and educational systems throughout the world often failed to foster children's individual learning potential. Teaching methods and schools vary greatly, depending on such factors as the country's resources, and teacher training systems, among other factors. The schools see success when they generate and apply evidence and share best practices. Developing and fostering a global school system that facilitates the generation and dissemination of research and best practices strengthens knowledge exchange between schools within and across schools, networks and countries. And the use of evidence and decision making about the uptake of innovations of technology helps to fully harness variability in learning. To learn more about achieving impact through embedding research into school level practice, we're joined by four experts. Our first guest is Nora Marketos, who's currently the CEO lead of the learning schools portfolio at the Jacobs Foundation, where she oversees a program with the objective to strengthen the global school landscape through testing and scaling evidence informed practices. Prior to that, she managed multi stakeholder partnerships in rural Cote d'Ivoire, at the United Nations in New York City. And she's also worked in Uganda, Syria, and Switzerland. Our second guest is Dr. Christina Hinton. Christina is the founder and CEO of research schools international, and a research affiliate at the Harvard human flourishing program. Also joining us today is Dr. Bronwyn McGrath, who is a global program manager at the Aga Khan Foundation, where she leads the school's 2030 initiative, a participatory learning improvement program operated in government schools across 10 countries. And our fourth and final guest is Azad Oommen. Azad is the co-founder of Global School Leaders GSL, a nonprofit that develops effective school leadership to improve learning of students from marginalized communities around the world. So I'd like to begin by asking a fairly general question and perhaps Bronwyn you can begin, and that is do school leaders, including teachers that you work with tend to use research and data in their daily practice?

Bronwyn McGrath (BM): Thanks, Daniel. That's a great question. But I think first, we just need to think a bit about what we mean by data and to think more broadly about the kinds of evidence teachers and school leaders generate and use every day. So the art and science of teaching relies on teachers constantly gathering evidence on student learning levels, whether through like formal testing, or just observing their students. And in an ideal classroom, teachers are always adjusting their teaching methods and the content of their lessons based on this evidence. Obviously, this ideal is not necessarily the reality in all classrooms, and particularly in the low resource settings where schools 2030 program runs, teachers often are facing very overcrowded classrooms, they have very heavy curriculum to get through. So it makes it difficult to adjust and iterate teaching practices in light of evidence and data they might they may be bringing in. So schools 2030 is working to support teachers to use data and evidence more effectively through two key sort of activity streams. First is through our assessment work. We're supporting the generation of contextualized locally adapted learning assessment tools that can help teachers generate and analyze data on learning outcomes, and also the quality of the classroom environment. So this involves a lot of professional development support of teachers. So this is really important for us so that teachers can get gain skills and become better at gathering and analyzing classroom data. So one of our teachers said quite nicely before I was just scoring my students, and now I'm assessing their learning. So our vision is really for teachers and school leaders to be seen as actual researchers and to be valued as researchers who are gathering and acting upon school level evidence in order to improve learning outcomes in their schools. But the second way we're really supporting teachers and school leaders to use research and data is perhaps more closely aligned to some of the other work of people on this podcast today, we're working a lot with external researchers across many of our program countries to embed school researcher partnerships in the schools 2030 program. So these external researchers work with schools to support evidence generation on how to improve quality equitable education opportunities, and they're drawing on schools 2030 assessment data and tools, but they're also bringing in other qualitative and quantitative methods. So to get back to your original question on how schools are using research data in schools 2030, this kind of partnership with researchers, between teachers and school leaders and researchers is very, very new for our context, our program and our schools. Traditionally, a lot of these schools and teachers are seen as sites for data extraction, and not as equal stakeholders or actors in the research process. So for us, this research partnerships are very, very important part of the program because it's about strengthening teacher skills as researchers and ensuring they're part of the research process right from the design stage. And we really believe this will result in more relevant and powerful evidence on what works to raise learning outcomes for children and young people.

DR: Azad do you have a comment?

Azad Oommen (AO): Bronwyn mentioned the word or the words data extraction, and what we see is that often the data that's collected in school, whether it's at the classroom level, or at the school level, or tends to flow in one direction out of the school, and usually it's in, in response to demands from the education systems within which the place so whether the district level or the state level or the national level. And what we find is that school leaders spend an extraordinary amount of time on producing this data, which leaves the school and then never really returns to the school in any meaningful input to the school to improve their functioning. So oftentimes, this is administrative data around attendance, or it may be around infrastructure, things like this, that school leaders report into the education systems on, but the loop never gets completed because that information is never analyzed and processed, and then back into the school to help them improve the school. So I think this the words, data extraction, that Bronwyn used are very relevant to many of the contexts in which we find school leaders operating.

DR: So I believe, Nora, you have something to say about this as well.

Nora Marketos (NM): Having talked to a lot of stakeholders in the education system here in Switzerland, we actually had also quite some hesitancy from some of the teachers because they saw the data collection more as an accountability purpose, and not really an activity that serves their kind of improvement process around teaching and how they can actually continue with professional development. So in this sense, there was a lot of fear almost to dealing more with data because it was always only seen from that accountability and control perspective, and not really this enriching perspective as a professional.

DR: Well, it sounds like you're really trying to build relationships with teachers as an equal exchange. Can you talk a little bit about that?

NM: Exactly, I'm happy to do so. So what is really important also, that research and data provides value for the school stakeholders is that first of all, the teachers should actually feel and get an awareness that when using data, but then also evidence on what works from learning sciences, that they are aware of what is actually effective on the ground. So really not using the classical learning myth, such as the learning styles or the left, right and left brained person so that they really can make sure that they do no harm. So this is one thing, but then really for also the everyday decisions on how best to use their time and choosing a pedagogy or in making decision on how to also improve school culture, there it's really also a matter of using the data, but then also the research on what is actually most effective under certain circumstances. So to really use similar to what also in medicine is happening to really use those findings on an everyday basis but then also to going back to what Bronwyn and Azad mentioned before to really make sure that teachers and school actors can become much more an active also participant in research activities so broadening their engagement in the research process in the sense that they use everyday data that is generated in a scientific method gets to the skills around that, but then also have a broadened peer engagement with other schools or with their own colleagues as part of a professional development so that they really own the process and can also contribute and make sure that the research that’s created is actually relevant to them and to other colleagues.

DR: That's actually what I wanted to ask you about. I mean, obviously, research is very important to all of you, and you spend so much of your time and your work doing research. But what is the value of embedding research and using data for school level practice?

NM: So in the end, this is really a helpful tool to make better decisions, be it for school leaders, be it for teachers, so really getting acquainted and also have this as a tool for an everyday improvement for also really an evidence based exchange with other peers and to make sure that their own biases are actually not interfering too much with the teaching so in this sense supporting and providing the tools for the schools stakeholders to make the right decision to being as effective as possible in whichever sense they decide to do so.

DR: What is your experience with that then Azad?

AO: Yeah. So Daniel, I think one has to start with the fundamental assumption that schools today, well maybe it’s not an assumption, the fundamental thought that schools today are failing. So particularly in low and middle income countries where you see huge learning gaps among children, even as young as second and third grade. And then it becomes increasingly difficult for them to catch up. Now, if one starts with that assumption that the schools are failing, and we, we believe that schools need to improve, which is what all of us are doing, then one has to create a vision for the schools and the only way to really create an effective vision is to create the vision using data from the schools. And I think, from that perspective, data becomes so important for schools in order to create this vision for themselves. And so I think that as we think about how to create these data, this data, because one could collect data on all sorts of things but I think the data that's collected that really focuses in on student outcomes is what is really, really important, because then that places student outcomes at the center of all that we're doing. And so I think it's important to then think, in a multi-dimensional way about how to think about student outcomes. So we're not just talking about literacy, numeracy, but also social emotional learning, the whether students feel happy coming to school, whether parents are engaged in school, there's all sorts of indicators that go into the health of a school that can be centered on student outcomes. And so the importance of the data of embedding data systems within the school is that it really brings to life the vision for the school.

DR: I’d really like to hear some specific examples, I think that might flesh this out for us a little bit. So maybe Christina, you could begin by sharing a good practice example of a school whose leaders or teachers embedded research or data into practice.

Christina Hinton (CH): Sure, hi, this is Christina Hinton, I lead Research Schools International which is an organization that partners researchers from Harvard Business School with schools around the world to carry out research, professional development and dissemination of findings. And it's a really powerful model for a lot of the reasons that my colleagues were just speaking about, in particular, Bronwyn was mentioning the importance of the partnership between researchers and practitioners and that is exactly what we do. So, in contrast to kind of traditional research, where you look for gaps in theoretical literature in order to build the knowledge base, our research is really driven by the needs on the ground. So we asked the school partners, what do you need? What challenges are you facing? What innovative practices would you like to try out? These kinds of questions, and that really defines the research agenda. And so in that sense, we can be more confident that what we're doing is relevant to the schools themselves. And so I can give you an example of a project just to kind of illustrate how that plays out if that's helpful.

DR: Yes, definitely. That sounds interesting.

CH: So we've done lots of different projects with schools all around the world but just to think of one illustrative example with a school in the UK called Sevenoaks, we recently carried out a project on flourishing. And the school wanted to promote flourishing, which is holistic wellbeing and I’m also at the human flourishing program at Harvard so that's an area of research, a passion of mine, as well. And so we basically wanted to see, okay, we know all of this research on various activities that promote flourishing, how much are they happening at the school and where could they happen more? And so what we did is we developed an online survey and we worked in collaboration with the school on that to make sure it was sort of tailored to their interests and their needs and their context, developmentally appropriate and all that. And we have a team of teacher researcher fellows and student research fellows at each of our partner schools so they can give us feedback on things like surveys throughout the research process, and so we developed that survey we looked, and the surveys include like quantitative methods and qualitative or open-ended responses so we can really get a rich understanding of what's going on from those. So we analyzed the data we found out okay, here's some areas where we're doing beautifully and have some wonderful practices happening on the ground, concrete examples that we found and we like to share those back. So then we go to the school, we lead professional development and we start by sharing backwards positive examples. And that helps kind of spread good practice across the school. Because sometimes teachers are doing really interesting things in their classroom when no one else knows about it. And then the other thing we look for the surveys are, here's areas for growth. So here's an area where we know that this is really good to promote, due to wellbeing or flourishing and unfortunately, you know, not that much is happening actually on the ground, in the school, in that area, or in a way that's effective. And so then we identify those areas, and we say, okay, for example, you know, you could do more to support gratitude at your school, or engaging the kids in mindfulness or, you know, various things that we know can support flourishing. And so then we, we basically identify these areas for growth, and we invite the educators in the workshop to design something that they would like to do to target those areas for growth. And so one thing that's powerful about that we're creating something that they're excited about, they're getting to choose which which area they want to focus on, they're getting to create something that kind of winds with what they're doing in the classroom, and their belief system. And so but yet, at the same time, it's filling the research gap. So it is, you know, research driven and kind of bringing the school forward in the ways that they want to. So they create something that they want to do. And they kind of create that in the workshop and we do so we do the research in the first year, and then the workshop we do at the start of the next academic year so that was kind of excited to implement it. So then they go and put that into practice. And then at that point, what we do is we lead them through an action research course so they put their research-based intervention into practice and we guide them through a series of action research steps where they can kind of reflect on the impact of that. So they basically design a little study where they look at, you know, the impact of what they've created. And we guide them through each step of that so in that sense, we're also building their research school skills internally. Yeah, so the teachers go through that course and end up with some of their own research skills that they can apply to other topics as well. But in the end they sort of share what they found and celebrate the learning journey. So it's just a nice arc of really, truly working in partnership with schools, with researchers and practitioners hand in hand doing work that will actually have a real impact in the school. And so I think there elements of a model that people could, could use themselves as well, I think it's difficult as, as some colleagues were saying, when you partner with a kind of more traditional researcher, because they're not trained to do that type of work, like the, you know, they're interested in clicking the data and analyzing and publishing papers, whereas our program is very different because it's focused on, you know, what, what the needs of the schools are, and doing research to meet those needs. So it's, I think it's much more relevant in general, I think there's a place for more traditional academic research as well, because we need to continue building the field over time. But in terms of the kind of immediate value for the schools, I think our approach is very effective.

DR: Sounds like it's a lot of teacher development as well that you're helping the teachers develop by applying the research directly into the classroom. I'm wondering if Azad you'd like to add to that.

AO: Yeah, thanks, Daniel. So I wanted to share the example of this research and data work we've been doing around the role of school leaders in ensuring gender responsive and gender equitable schools. So we were as we were looking at the broader research around this issue, around how school leaders actually influence both boys and girls education that we were coming at it initially from looking at girls education. And we found that there's very curious lack of research on this broad topic of how what actions the school leaders take that can ensure gender responsive and gender equitable schools. And so we were rolling out a program in India and Kenya called Shakti that we will in which we were looking at our training school leaders on what we saw as ideas through which they could run gender equitable and responsive schools. And in the as a baseline study, and that we actually surveyed both our school leaders as well as parents and students around what they saw the issues relating to this topic were and what we found was that there was a big gap between how school leaders perceive the issue and how parents perceive the issue, so school leaders were seeing this issue as girls don't come to school because a lack of commitment from the parents and parents were seeing this as much more of an economic issue of what what is the economic return of sending their child to school. And given that we were seeing these disconnects between the various people who are involved in this equation, then we actually work with the school leaders in India to design a survey that they could implement in their own schools to really understand what the girls students in their schools saw as gender equity issues. And that was really interesting, because then that really started highlight both girls who are at risk of violence, of abuse, it also brought to light practices within the schools that could be implemented that would make the school a more welcoming environment from a girls perspective. So I think when we think about research and data, and I think this this conversation is so interesting, because the data that's collected in a school can often be very disconnected from sort of broader research questions. But I do think that when done properly you can find ways to, to frame or broad research questions that then the data which is collected within schools can feed in to.

DR: That's really fascinating. Does anybody have anything else they'd like to add?

NM: Thank you. I can also give you an example around the recent partnership with the IB, so International Baccalaureate school system, where we have partnered with two research institutes, Oxford, as well as ACR around how to measure in classroom creativity and curiosity and also finding promising practices, promising classroom practices. It was very interesting that it was really a bottom-up approach, even though conducted with top notch researchers, but really trying to get interested schools and school leaders to participate in the research in sharing their best practices. So that's the research is good actually, in a rigorous qualitative way, find common elements that define emerging best practice around how to strengthen curiosity and creativity in classroom. And what was interesting is that there was a large number of schools who wanted to participate, but then also continue using those findings as a kind of peer community. And maybe what we haven't addressed yet in this discussion is around, you know, incentives, positive and negative incentives, or why data and participation in evidence generation is actually still relatively limited in general, because in our example, we could see that just the ethics requirements were so high that in the end, the engagement between the peer schools was kind of limited, given that the identity of those participating in the research shouldn't be shared broadly. So in this sense, there is a lot of also discussion around how best to set up the frameworks of doing rigorous research, and making sure that school leaders are school actors who want to actually actively use but then also act upon the findings are not hampered by those regulations that were in the end set up for their protection, so that was quite an interesting finding, I found on first of all, active participation in the interests of the stakeholders at schools, but then also the certain limitations that were given due to the setup with the Ethics Committee.

DR: That's really fascinating. Anybody else?

BM: I can maybe come in with that an interesting example, Daniel, of some of some research partnerships as well, that we've had across our countries. But specifically, I'm thinking of some research partnerships in Kenya and Pakistan that have been working with schools and teachers and school leaders to understand the connections between the academic and non-academic or academic and holistic skill domains. I think that it sort of builds off what Nora was saying, as an example of making that clear value add to the teachers and the school leaders because in these contexts, there was sort of policy and curricular relevance placed on 21st century skills and and the sort of competency-based curricula, but there was not a lot of experience for teachers and school leaders on how to teach those in the school. So through these research partnerships, the teachers and school leaders were helping to define the questions that would, in the domains that they would focus on to understand how might we infuse things like curiosity and creativity and collaboration across our whole curriculum rather than thinking about them as discrete subjects. How do they reinforce academic learning? How can we think about academic and non-academic skills, sort of as an intertwined, whole child approach rather than as as discrete skills. So I think it was so valuable to have the perspectives of teachers right from the outset of the research process, really helping to define those questions and define the domains that were relevant for their own teaching practices and curricular requirements. But it was so important as well to have academic researchers who can support teachers on how to analyze that data. Because I think that's one of the the maybe gaps in teacher professional development in terms of really embedding research and data in school practice and in pedagogical practices.

DR: Yeah, it sounds like, you know, we've been talking a lot about the relationship between the researcher and the teacher, the leaders in the school and the exchange. And I'm wondering how can we encourage school leaders and teachers to be encouraged and supported to embed this research or data into their schools?

AO: I think, you know, when we go through our lived experience, as well as through research, we see that so much of the the use of data comes down to accountability for the use of the data. So I'll share two examples about what I'm talking about. One is that in 2020 two professors Karthik Muralidharan, and Ajit Singh, had done a research study on an experiment that was done in India, in a fairly large state where there was a lot of data that was being generated, and the around things like in order to do things like school rating, in order to look at how school leaders can create school improvement plans, and what they found was that while all this data was generated, and there was a relative fidelity to the amount of data that needed to be generated, it was not necessarily used very effectively and didn't really end up in creating any shifts in student outcomes. So I think from that example, at a systemic level, you would see that the school leader or the teacher, whoever the agent is who's collecting the data, has to be able to see that the data is being utilized and they have to be held accountable for producing the data and for then, taking whatever analysis is done and then to implement it in the schools. From our lived experience we had a program on during COVID, looking at remedial learning, we did this in partnership with a couple of other organizations. And there we found was happening there was that teachers were reaching out to students with scripted questions to encourage them their continued learning to STEM learning losses, we found there was that when school leaders were actually holding teachers accountable for doing the outreach for gathering the data for using the data to then further personalize the lessons that students were getting, then it was used effectively. But where school leaders were not there to hold the teachers accountable this did not happen as effectively. So so so I think there's this the accountability of the data that's being used, both in terms of sort of making sure that the data is actually collected on a regular basis, but also accountability in terms of the data actually being analyzed and fed back into the school is really critical for people are to see the value in investing in collecting data.

DR: That's really fantastic. You know, we've we've covered all of our questions that I had in mind, is there anything that you feel that we didn't discuss that you think is something that the listeners might find interesting?

BM: I might just pick up if I may for a moment on Azad's point around the analysis in the support and data analysis, because we have found particularly through the assessment work and as supporting teachers on learning outcome assessment, that that supportive analysis, not just analysis by, you know, maybe external researchers, and that's been brought back to the classroom, but supporting teachers themselves to analyze learning data so that they can work actively to support the learning needs of the students in their classroom, we've really found that to be a huge priority area and an area where there's a great potential for positive change. So that for just a small sort of illustrative example, in one of our schools in Uganda, this is about literacy, not, not the more holistic learning but very, very low literacy rates in a primary school that we're working in, in Uganda. And the teachers were used to testing for literacy and running some standardized tests as part of the curriculum, but there was not a lot of understanding of how to really look deeply at those tests to identify what are what is this data telling us about the specific areas that students are struggling with, rather than just saying you know, literacy levels are low, let's keep pushing through this textbook material. And we found through giving support and providing sort of external support to teachers on that data analysis, teachers were able to really hone in on specific areas that groups of students were struggling with so then they could target that, for example, blending consonant sounds, or perhaps it's at a higher level, it’s around the reading comprehension or something like that so that teachers can really understand what the data is telling them. And I wonder if others have had this experience as well but for us, that's been really the most important pinch point in terms of data use of school.

DR: Thank you for listening to today's episode. For transcripts and more information about our guests, please see our show notes on our website. I'd like to thank Sharon Parkinson and Annie Brookman-Byrne for the help with today's episode, as well as Alex Jungius of This is Distorted.