Restating views on Duolingo
Posted: Sat Aug 01, 2020 11:23 am
It’s a long time since I’ve given a detailed description of my beef with Duolingo, and as promised, in light of recent heated debate, here’s a run-down of it.
First up, I’ll say what Duolingo does right:
It gives of high-density practice with immediate feedback.
It personalises the learning track slightly.
…and that’s basically it. Don’t get me wrong, that alone is a pretty big thing, but that is really all it does.
And even that second one says “slightly”. This is where I start to get really annoyed with Duolingo: they promised us machine learning and personalised learning, and instead we have a tree system where in a course with 100, 150, 200 “skills”, we have at most a choice of 3 new skills to study from at any given time, and even then, that’s only a choice for the immediate term, because you’re going to have to do all of them before moving on anyway.
It could allow us to set our own goals. It could allow us to tell it what we’ve learned in classes, but the company doesn’t want to do that, because then they would be presenting Duolingo as a tool, not a course. If I’ve been studying a particular tense in a language and then try to practice it in Duolingo, I have to complete or test out of all skills that the Duolingo puts in the learning track before then. So if my teacher or textbook puts conditional before imperfect and Duolingo does it the other way, I’m thrown to the wolves.
This would be far less of an issue if Duolingo was more complete, but it was written by computer scientists who really didn’t understand teaching very well. They launched with “lesson notes” that weren’t much use. The problem with the lesson notes was that they were a continuous chunk of text covering all grammar points that arose in the lesson exercises, and that was just too much text: I literally couldn’t keep all of it in my head. What I did, then was read a bit, then start a lesson. But the bit I read often didn’t come up at all in the session, and even if it did, I’d have been so distracted by trying to recall answers for the first few tasks that I’d have forgotten what I’d read in the notes before I needed it. This is perhaps the most fundamental rule of teaching: give the student an immediate opportunity to practice what you’ve told them, or they’ll forget.
The assumption from the design team was that notes were useless, and they then spent years hiding them. What they should have done was quite simply teach. Describe a bit, practice, describe a bit more, practice that, practice both together, practice alongside previous material… etc etc.
They talk about big data, they talk about A-B testing, but when they discuss decisions, it all comes down to retention of users. They change one thing and retention gets worse, and they change it back. Their limited model sits in a “local optimum”, where people stay and get a reasonable number of answers right, but people don’t really learn unless they’re using something else. But with the size of their audience, they could have explored all the possibilities of language learning by now. They could have explored explicit instruction first vs explicit instruction after practice. They could have used the data from their message boards to find the point where the grammar-averse learner suddenly decides they need to know what’s going on, and inserted just-in-time instruction into the course. They could have explored every possible ordering of grammar rules and mapped out the most efficient path through learning.
They’ve been going nearly 9 years, they’ve had millions upon millions of hours of user data, they’ve got a celebrated computer genius sitting at the head of the company, and what have they got to show for it? A simple interactive quiz program with a few bells and whistles.
The gamification elements annoy me too – they simply don’t understand what XP is.
In a game, you might get 1XP for killing a rat and 10000XP for killing a dragon. Makes sense, right? The dragon’s a harder enemy than the rat. But every single task is worth the same number of XP in Duolingo. But there are two dimensions of complexity in Duolingo – there’s task difficulty and language difficulty. In Duolingo, you’ve got word and picture matches, you’ve got multiple choice, you’ve got fridge magnets and you’ve got full text typing; you’ve got L1->TL translation, you’ve got TL->L1 translation, you’ve got dictation; you’ve got single words like “orange” vs sentences with the potential for complexity like “He said he would have done it if I’d asked.” All of these reward the same. They hide the typed answers because the users don’t like them – well maybe they’d like them more if they received recognition for the extra work, because right now, they slow down your accrual of XP, where they should be speeding it up. XP here doesn’t reward positive behaviours – it just rewards grinding.
In a game, XP is exponential – the more you increase in levels, the more XP you’re going to get for winning a fight at your level. To keep you motivated to take on the hard fights, the XP you need for levelling up increases exponentially too. Duolingo doesn’t have exponential XP, but it does have exponential levelling. This means that the longer you play for, the less rewards you see. I suggest this is what has led to the proliferation of other reward mechanisms – trophies, owls, badges and leagues. They needed to add in various types of achievement to keep people feeling like they were progressing… but if they’d just implemented a proper XP system, this would not be an issue, as normal levelling up would occur frequently enough to motivate.
Then there’s Duolingo for Schools, launched in January 2015 to much fanfare, allegedly giving teachers control over their students’ learning, but as a teacher myself, I have to say it has never been usable for a language teacher. There are two options for assignments to set your students: a skill from the skill tree, or a set number of XP to achieve. The latter is utterly useless, because (as I said) the XP system rewards laziness and discourages working on challenging material. The former isn’t much cop either, as your students don’t get immediate access to the skill you’ve set them – they have to go through the whole skill tree, the same as any other user. In fact, in my test student account, I don’t even get any indication that I’m following a Duolingo for Schools curriculum, or about what my current “assignments” are or how close I am to completing them. If I wanted to use Duolingo for Schools as a language teacher, I’d have to change my teaching radically so that my lessons related to the Duolingo content, which might be easier if they supplied decent teachers’ notes, but the only thing you get is a list of words and phrases, and if the lesson isn’t named after a grammar point, the only way to find out what grammar it covers is to go through the lesson multiple times (because not every point appears first time round). So I could spend months of my life gathering data, months of my life building a syllabus around and then I’d have… a very mediocre language class. That whole thing about the lack of customisability I was on about earlier? That’s a million times worse when you push it towards teachers.
In essence, Duolingo for Schools is for schools that don’t have language teachers. It’s Duolingo, but with a school supervisor who just checks that the kids have done their allotted homework.
Overall, Duolingo is a massive missed opportunity. It’s the squandering of lots of potential by a couple of well-meaning techno-utopians who bit off far more than they could chew, and almost a decade on still have no clue about it.
And it’s not like you can just create a “better Duolingo”. With their price point of “free” and their massive press exposure, and little fish just disappear. Who remembers the site that was like Duolingo but was built around the idea of preparing you to listen to pop songs in your target language? It was better than Duolingo, but it disappeared because it was too similar. So we’re left with the likes of Babbel who set their stall on looking nothing like Duolingo, which means they end up missing out on the big advantage of online learning: rapid, high-frequency practice with immediate feedback. There’s fertile ground near Duolingo, but their tree’s so big it’s blocking the light from anything that tries to grow there.
First up, I’ll say what Duolingo does right:
It gives of high-density practice with immediate feedback.
It personalises the learning track slightly.
…and that’s basically it. Don’t get me wrong, that alone is a pretty big thing, but that is really all it does.
And even that second one says “slightly”. This is where I start to get really annoyed with Duolingo: they promised us machine learning and personalised learning, and instead we have a tree system where in a course with 100, 150, 200 “skills”, we have at most a choice of 3 new skills to study from at any given time, and even then, that’s only a choice for the immediate term, because you’re going to have to do all of them before moving on anyway.
It could allow us to set our own goals. It could allow us to tell it what we’ve learned in classes, but the company doesn’t want to do that, because then they would be presenting Duolingo as a tool, not a course. If I’ve been studying a particular tense in a language and then try to practice it in Duolingo, I have to complete or test out of all skills that the Duolingo puts in the learning track before then. So if my teacher or textbook puts conditional before imperfect and Duolingo does it the other way, I’m thrown to the wolves.
This would be far less of an issue if Duolingo was more complete, but it was written by computer scientists who really didn’t understand teaching very well. They launched with “lesson notes” that weren’t much use. The problem with the lesson notes was that they were a continuous chunk of text covering all grammar points that arose in the lesson exercises, and that was just too much text: I literally couldn’t keep all of it in my head. What I did, then was read a bit, then start a lesson. But the bit I read often didn’t come up at all in the session, and even if it did, I’d have been so distracted by trying to recall answers for the first few tasks that I’d have forgotten what I’d read in the notes before I needed it. This is perhaps the most fundamental rule of teaching: give the student an immediate opportunity to practice what you’ve told them, or they’ll forget.
The assumption from the design team was that notes were useless, and they then spent years hiding them. What they should have done was quite simply teach. Describe a bit, practice, describe a bit more, practice that, practice both together, practice alongside previous material… etc etc.
They talk about big data, they talk about A-B testing, but when they discuss decisions, it all comes down to retention of users. They change one thing and retention gets worse, and they change it back. Their limited model sits in a “local optimum”, where people stay and get a reasonable number of answers right, but people don’t really learn unless they’re using something else. But with the size of their audience, they could have explored all the possibilities of language learning by now. They could have explored explicit instruction first vs explicit instruction after practice. They could have used the data from their message boards to find the point where the grammar-averse learner suddenly decides they need to know what’s going on, and inserted just-in-time instruction into the course. They could have explored every possible ordering of grammar rules and mapped out the most efficient path through learning.
They’ve been going nearly 9 years, they’ve had millions upon millions of hours of user data, they’ve got a celebrated computer genius sitting at the head of the company, and what have they got to show for it? A simple interactive quiz program with a few bells and whistles.
The gamification elements annoy me too – they simply don’t understand what XP is.
In a game, you might get 1XP for killing a rat and 10000XP for killing a dragon. Makes sense, right? The dragon’s a harder enemy than the rat. But every single task is worth the same number of XP in Duolingo. But there are two dimensions of complexity in Duolingo – there’s task difficulty and language difficulty. In Duolingo, you’ve got word and picture matches, you’ve got multiple choice, you’ve got fridge magnets and you’ve got full text typing; you’ve got L1->TL translation, you’ve got TL->L1 translation, you’ve got dictation; you’ve got single words like “orange” vs sentences with the potential for complexity like “He said he would have done it if I’d asked.” All of these reward the same. They hide the typed answers because the users don’t like them – well maybe they’d like them more if they received recognition for the extra work, because right now, they slow down your accrual of XP, where they should be speeding it up. XP here doesn’t reward positive behaviours – it just rewards grinding.
In a game, XP is exponential – the more you increase in levels, the more XP you’re going to get for winning a fight at your level. To keep you motivated to take on the hard fights, the XP you need for levelling up increases exponentially too. Duolingo doesn’t have exponential XP, but it does have exponential levelling. This means that the longer you play for, the less rewards you see. I suggest this is what has led to the proliferation of other reward mechanisms – trophies, owls, badges and leagues. They needed to add in various types of achievement to keep people feeling like they were progressing… but if they’d just implemented a proper XP system, this would not be an issue, as normal levelling up would occur frequently enough to motivate.
Then there’s Duolingo for Schools, launched in January 2015 to much fanfare, allegedly giving teachers control over their students’ learning, but as a teacher myself, I have to say it has never been usable for a language teacher. There are two options for assignments to set your students: a skill from the skill tree, or a set number of XP to achieve. The latter is utterly useless, because (as I said) the XP system rewards laziness and discourages working on challenging material. The former isn’t much cop either, as your students don’t get immediate access to the skill you’ve set them – they have to go through the whole skill tree, the same as any other user. In fact, in my test student account, I don’t even get any indication that I’m following a Duolingo for Schools curriculum, or about what my current “assignments” are or how close I am to completing them. If I wanted to use Duolingo for Schools as a language teacher, I’d have to change my teaching radically so that my lessons related to the Duolingo content, which might be easier if they supplied decent teachers’ notes, but the only thing you get is a list of words and phrases, and if the lesson isn’t named after a grammar point, the only way to find out what grammar it covers is to go through the lesson multiple times (because not every point appears first time round). So I could spend months of my life gathering data, months of my life building a syllabus around and then I’d have… a very mediocre language class. That whole thing about the lack of customisability I was on about earlier? That’s a million times worse when you push it towards teachers.
In essence, Duolingo for Schools is for schools that don’t have language teachers. It’s Duolingo, but with a school supervisor who just checks that the kids have done their allotted homework.
Overall, Duolingo is a massive missed opportunity. It’s the squandering of lots of potential by a couple of well-meaning techno-utopians who bit off far more than they could chew, and almost a decade on still have no clue about it.
And it’s not like you can just create a “better Duolingo”. With their price point of “free” and their massive press exposure, and little fish just disappear. Who remembers the site that was like Duolingo but was built around the idea of preparing you to listen to pop songs in your target language? It was better than Duolingo, but it disappeared because it was too similar. So we’re left with the likes of Babbel who set their stall on looking nothing like Duolingo, which means they end up missing out on the big advantage of online learning: rapid, high-frequency practice with immediate feedback. There’s fertile ground near Duolingo, but their tree’s so big it’s blocking the light from anything that tries to grow there.