Tseng and colleagues searched the literature from 1995-2019 for studies on the effectiveness of study abroad programmes on language learning. They excluded studies that did not have clear measures of the linguistic outcome (including pre-test scores for comparison) or lacked comparison with a control group (ie, students studying in their home country), leaving 42 studies to take forward.
The collected studies included data from nearly 4,000 students learning English, Spanish, French, Japanese, Chinese, German, Russian and some mixed classes. Just over half the data came from students learning English and around a quarter from students learning Spanish.
There are pros and cons to including different language learners. On the one hand, it adds power to the study by increasing the sample size, but on the other hand, it also adds a lot of variation that is hard to account for, especially as some of the languages (Chinese, German, Russian and mixed classes) contributed less than 10% of the data each. The comparison of different language learning outcomes did, however, have useful implications regarding testing.
In the selected studies, the effects of study abroad were quantified as effect sizes (g). Having a significant difference in outcomes between studying abroad or at home may not be of much interest if the actual effect on those outcomes is very small. As the dataset increases in size, the likelihood of significant but rather small effects increases, and effect sizes that are both significant and larger are the focus of interest.
Altogether, 283 effect sizes were generated from these studies and entered into one, overall analysis – a meta-analysis. Because some of these effect sizes were related, being from the same data sets, this was accounted for by carrying out a three-level meta-analysis that could weight the data appropriately.
Tseng’s study is relatively rigorous in its analysis and by combining studies into a meta-analysis they had sufficient data to look at the effect of moderating factors on the overall effectiveness. For example, previous studies had not addressed how choice of outcome measure might influence how effective the programme appeared to be.
There is always some difficulty with meta-analyses in that different studies do not measure things or define categories in quite the same way and some factors are under-represented. Despite these challenges, 10 potential moderators and 11 outcome measures were identified for this meta-analysis, enabling a fuller comparison of study abroad compared to domestic programmes that goes beyond descriptive commentary, offering clear, quantitative results.
