AI Conversation Partners in Language Learning: What Randomized Studies Show

What g=0.484 Actually Means in Practice

In 2024, Wang and colleagues published a meta-analysis of 28 studies covering 70 measured effects on AI chatbots in language education (Review of Educational Research, DOI 10.3102/003465432412556). The average effect size was g = 0.484. On the SLA calibration scale by Plonsky & Oswald (2014), where d≈0.4 is small and d≈0.7 is medium, this sits at the lower end of medium. In plain terms: the average student who practiced with an AI chatbot scored better than 68% of students in traditional-instruction control groups.

For context, human one-on-one tutoring in VanLehn (2011) reaches g = 0.79, roughly 60% higher (DOI 10.1037/a0037123). That gap is real, but AI is available 24/7 at a fraction of the cost of a private tutor.

What Drives the Effect: Wang et al. 2024

The meta-analysis identified four moderators that shift the effect size significantly:

• Education level. University students benefit more than school-age learners, because adults use feedback more strategically.
• Proficiency level. Beginners gain less; the chatbot needs a vocabulary base to sustain a real conversation. B1+ learners show stronger gains.
• Interface type. Voice interfaces outperform text-only chatbots for speaking skills: you cannot practice pronunciation by typing.
• Interactivity. Adaptive systems that respond to what the student actually said outperform scripted bots with fixed reply trees.

The sweet spot: an adult learner at A2 or above, using a voice-based adaptive system.

Generative AI: A Step Up

Wang et al. (2024) focused largely on traditional chatbots. Lyu et al. (2024) analyzed 51 studies on generative AI across education (DOI 10.1007/s10791-026-10015-1) and found larger overall effects, especially in informal learning contexts, for productive skills (speaking, writing) over receptive ones, and for less common languages where live practice partners are scarce.

A field RCT confirms this: De Simone et al. (2025) ran a World Bank–funded randomized trial in Nigeria with 6 weeks of after-school sessions with Copilot/GPT-4 for English (hdl.handle.net/10986/43212). Results: +0.31 SD overall, +0.23 SD specifically for English (between-group). A moderate but real-world-replicable effect, not a lab artifact.

The Novelty Effect: An Honest Caveat

Wu & Yu (2023) analyzed 24 RCTs of AI chatbots (DOI 10.1111/bjet.13334) and found a consistent pattern: short interventions (4–6 weeks) produce higher effects than longer ones. The likely culprit is novelty: students engage more intensely with unfamiliar tools. As novelty fades, so does motivation, and the effect shrinks. Changing scenarios, goals, and conversation contexts is necessary to sustain engagement beyond 6 weeks.

The Dependency Trap

Bastani et al. (2025, PNAS) ran a school RCT in math. Students who used GPT-4 during practice improved task performance by +48%. But when tested without AI, they performed worse than the control group. They had offloaded cognitive work to the model rather than building their own knowledge structures.

The language-learning parallel: if a chatbot supplies words on demand or corrects every error instantly, the student never practices retrieval, the very effort that drives long-term retention. The design principle: provoke an attempt first, then support.

AI vs. Human Tutor: The Honest Numbers

VanLehn (2011) compared learning formats rigorously (DOI 10.1037/a0037123):

• Human one-on-one tutor: g = 0.79
• Step-based intelligent tutoring systems: g = 0.76
• Problem-level ITS (no step-by-step feedback): g = 0.40

Today's generative AI companions sit closer to the third category, though the best systems approach the second via adaptive hints. The human tutor advantage is real. But at €30–100/hour for a native-speaker tutor, AI enables 10–20× more practice hours for the same budget.

Pronunciation and Speech Recognition

Ngo et al. (2023) analyzed 15 studies of speech recognition in language learning (DOI 10.1017/s0958344023000113). Average effect: g = 0.69, medium-to-strong on the SLA scale. Breakdown: large effect for individual phonemes, small effect for intonation and rhythm. Partner-based dialogue practice outperformed solo exercises like reading aloud.

How Lexy Is Designed

Lexy is an AI language trainer across 49 languages, combining voice-first practice with multi-stage reinforcement. The starter course covers 16 lessons of 25 minutes each. Each lesson consists of seven stages: 1. Vocabulary (flashcards and matching quiz), 2. Warm-up (brief written chat), 3. Lesson (25-minute live voice dialogue with the avatar without typing), 4. Exercises (keyboard-based grammar tasks, listening, and tests for balanced cognitive pacing), 5. Dialogue (two-voice situational listening), 6. Story (reading story with sentence translation), and 7. Memory Flashcards (spaced repetition of the prior lesson from lesson 2 onward).

In lessons 1–15, Lexy deliberately does not correct errors during the live lesson: lowering anxiety keeps the affective filter down and lets comprehensible input work. In the core live dialogue (Stage 3), interaction is bidirectional: Lexy asks, student answers aloud, student asks, Lexy answers, activating the productive skills that Lyu et al. found most responsive to generative AI.

From A2 onward, 24 roleplay scenarios with Bingo goals constantly shift the communicative context, directly counteracting the novelty decay documented by Wu & Yu (2023).

Key Takeaways

1. The effect is real but moderate. g=0.484 (Wang et al., 2024) is reproducible and meaningful for the right learner in the right setup.
2. Conditions matter more than the tool itself. Voice interface + A2+ proficiency + adaptive system = effect. Remove one factor and gains shrink.
3. Dependency is a genuine risk. Bastani et al. (2025) documented a scenario where AI made outcomes worse. Design against it.
4. AI scales practice, not instruction quality. The tutor gap (0.79 vs 0.484) is real, but AI offers 10–20× more practice hours per euro spent, and that changes the equation.

🎯 90-Second Quiz 1 / 5

Test Your Knowledge: Article Quiz

90-second quiz: check whether you understood the key themes of this article. 5 quick questions. If you make a mistake, we explain it right away.

1. What is the primary psychological benefit of practicing with an AI avatar?

Zero fear of social embarrassment, judgment, or awkward pauses The AI enforces rigid academic tests The AI speaks exclusively in whispers