Ron Ragsdale talks to Hamish Chalmers, language education researcher and former international school teacher, about the power of randomised trials

An Oxford Professor once told the Gazette that you “came from an evidence-based family”. Can you enlighten us?

It’s true, funny as that sounds. My dad is a medical researcher and uses randomised trials to assess alternative approaches to healthcare, so I was brought up on the idea of fairness in decision making. For example, when deciding whether it was mine or my brother’s turn to do the washing up, we would both voice competing reasons why it shouldn’t be ours. To resolve this fairly, Dad would flip a coin.

Naturally, I would complain that it still wasn’t fair, especially if it was the second or third time in a row that it was my turn. So, Dad flipped the coin 100 times and demonstrated that over time we would end up doing the washing up the same number of times. As I grew up, I came to appreciate the power of randomisation.

How does this relate to your own research?

Since my days studying to become a teacher, I have wanted to understand the effects of what we do in the classroom.

If you are trying to assess whether a particular way of teaching is better than another, you have to use the two approaches, observe what happens, then compare observations. However, you must create groups of students that are as alike as possible.

Groups of people are messy. Some possess characteristics that influence how they learn in one way, and others possess different characteristics that will influence how they learn in a different way. The trouble is, when we create comparison groups we can see and measure some of these characteristics, but we can’t see or measure others. So, how do we account for differences that we can’t see or measure, so that we know our comparison groups are similar?

We randomise. Random allocation to groups spreads out the mess fairly. We then have confidence that any differences in average characteristics of those groups are chance differences and not systematic differences (or biases).

Isn’t this standard research methodology?

It should be, but I have been astounded to find that this most basic consideration has not been routinely adopted by researchers. It is common to find research that fails to take into account the potential for biases. We routinely see research that compares groups that are systematically different to one another. This makes it very difficult to determine whether the differences in outcomes are a result of the systematic difference between groups or a result of the different teaching approaches being compared.

We also often see research that does not even have a comparison group. Researchers assess students, teach them in a particular way, then assess them again. Any differences between before and after is attributed to the approach, when in fact the difference could be explained by any one of a number of things. I don’t think this is good enough

“Do you want trustworthy research that gives you an accurate answer (even 
when that is unexciting) or untrustworthy research that gives you a clear 
answer that is probably wrong?”

You are particularly interested in the use of L1 in the school classroom, yet many schools are known for insisting on English only. Does research give any insights on how this should change?

From an ideological perspective, international schools in particular must consider whether it is a priority for them to honour the linguistic and therefore cultural backgrounds of their students or whether they want to inculcate an ‘anglo-cultural’ milieu in the school.

Actually, in many international schools there is a legal requirement to teach the home languages to home students. Moreover, opportunities to use the L1 outside schools are plentiful.

The risk, therefore, to L1 maintenance and cultural identity is lower than it is in schools in the anglosphere with high proportions of students who speak non-English languages at home, such as government schools in Southern California, state schools in suburban London, or independent schools or private language schools in the UK with large proportions of students from overseas.

From an educational sense, international schools must consider whether there might be an academic advantage in incorporating students’ L1s into their teaching. Firstly, there is a self-evident advantage if both the L1 and English are used in academic contexts, as graduates will be better equipped than students who have developed these skills in English only.

There is also the idea that the L1 can support academic learning in English- medium international schools. If top IGCSE results are a goal for the school, for example, does using the L1 make that more or less likely? From the research that we have, it seems likely that using the L1 to explain abstract concepts helps students understand content and might even improve English language skills.

The Achilles heel is the purchasing power of parents. Because ‘English only’ tends to be assumed by parents (and many teachers) to be the most effective way of teaching, schools who don’t offer this risk losing out.

You have posited that there is a lack of evidence-based findings to substantiate what “use of L1” should look like in the L2 classroom. Do we need to have the results of more rigorous research in order to justify any particular stance at all?

Much of the research on the effects of using the L1 has either not adequately controlled for potential biases or has measured proxy outcomes for actual attainment.

For example, children who go to bilingual schools do as well as or better than children who go to monolingual English schools. However, parents have chosen which schools to send their children to. It is plausible that bilingual education is responsible for superior performance, but it is just as plausible that there is something about families who elect to elect to send their children to bilingual schools that predisposes them to perform better generally.

Another issue is the use of proxy outcomes. For example, we often see self- reported enjoyment, feelings of empowerment, notions of respect for multilingualism and so on as outcomes. These are all really important, but they are not the same as better test scores, or higher graduation rates, or whatever metric you believe is reflective of an effective education system. We must be careful about what we can legitimately claim on the basis of existing research.

This is even more problematic where lots of different L1s are spoken. There is no research that robustly examines the educative effects of using L1 where more than one L1 is represented. Taking the findings of research from bilingual schools where everyone speaks the same L1 and extrapolating it to monolingual schools with lots of different L1s is problematic, regardless of the quality of the research.

If we want to claim that using L1s is more effective, we need research conducted in those contexts. Despite an enormous number of international schools and local schools with significant numbers of language minority students, this either hasn’t been done or suffers from proxy outcomes and biased comparisons.

This is not some esoteric point about methodology, it has real world consequences: if we tell teachers they should move to a multilingual model, there will be costs, in terms of time, energy, money and opportunity. Schools should be confident that a switch to multilingual pedagogies will be an improvement before making these investments. This can only come via good quality research

Is it practically possible to do this? Is there a good reason that this research doesn’t exist?

It may be ignorance of the importance of bias reduction in educational research. It might be that researchers in our field are less well versed in randomised trial methodology. There is also a myth that randomised trials are difficult and expensive. The only thing that is unique about randomised trials is how participants are allocated to comparison groups. What can be trickier is getting the scale needed to produce trustworthy results. However, this is not unique to randomised trials.

The important element here is that we do as much as we can to reduce bias when we conduct research so that we can be more confident in our findings. That’s the power of randomised trials.

Another criticism of randomised trials is that they tend to report slim or no differences in effectiveness of alternative approaches. This is blaming the tool for the shortcomings of the worker. Biased studies can exaggerate differences and mislead us.

Remember, in this case we are comparing using the L1 with whatever teaching approaches are already being used, which may be very effective. Dramatic effects are rare, so it is likely that using the L1 will have a slim effect.

Do you want trustworthy research that gives you an accurate answer (even when that is unexciting) or untrustworthy research that gives you a clear answer that is probably wrong?

Randomised trials are not difficult. Fair tests are how we get closest to ‘the truth’. In education, far too many studies are unfair tests; wasting time, energy and resources and failing to adequately inform the very people that researchers are ultimately trying to help: teachers and students.

I want teachers to have confidence in using the results of randomised trials to inform their practice. I see nothing difficult or controversial about asking the research community to do better than it has tended to do so far.

Is there any unbiased research to suggest that using L1 in the classroom improves outcomes?

In my systematic review of research on using the L1, I found very little research (11 studies in total) that met my inclusion criteria. Of these, only four studies found that using the L1 helped improve L2 vocabulary acquisition. However, in all of these studies the L1 being used was the same for all pupils, a language that was shared with their teachers. In the UK it is far more common to find many different languages represented.

Technology could allow us to use ebooks set up so that when you click on a word, an L1 gloss of that word appears. Alternatively, an approach that does not rely on technology might be encouraging students who share the same L1 to discuss concepts in the classroom using that language before carrying out work around those concepts in English.

In both cases, these could be easily compared in a randomised trial, where classrooms are allocated to either use L1 in this way or not, then compare outcomes at the end.

You have been involved in a number of systematic reviews in language teaching. Can you tell us about your most recent review?

I was part of a team at Oxford that prepared a rapid evidence assessment focussed on foreign language instruction. We took a series of published reviews, assessed the quality of their methods, chose the most rigorous and updated them.

This will be published shortly by the Education Endowment Foundation. It brings together hundreds of studies and we believe that it fairly assesses the state of our knowledge about what we can claim about the effects of language teaching.

Hamish Chalmers is course director of the MSc in Applied Linguistics and Second Language Acquisition in the Department of Education at the University of Oxford. He is also Vice-Chair of NALDIC (The National Association for Language Development in the Curriculum). He was a primary school teacher prior to moving into academia, most recently working as Director of English as an Additional Language (EAL) at an international School in Thailand. His research interests are use of the L1 as a pedagogic tool in the mainstream education of EAL learners, fair tests for causal inferences, and evidence synthesis.