TRANSCRIPT
EPISODE 28: BURR SETTLES
Jim Freeze Hi! And welcome. I’m Jim Freeze, and this is The ConversAItion, a podcast airing viewpoints on the impact of artificial intelligence on business and society.
[UPBEAT MUSIC]
It’s hard to believe we’re kicking off our fifth season of The ConversAItion. A lot has changed since we launched back in 2019 but one thing has remained crystal clear: the role of AI in society is more pervasive now than ever. Our mission has always been to foster productive conversations around AI and better understand its impact—and I believe this mission is more important than ever, as more businesses, and everyday people, lean on AI in exciting and sometimes unforeseen ways.
To date, The ConversAItion has covered everything from autonomous vehicles, to AI-powered personal styling services, to Kentucky Derby predictions. This season, we’re thrilled to be back with an exciting lineup of experts from some of today’s biggest and most innovative brands, all pioneering AI applications that are increasingly prevalent in our everyday lives.
Together, we’ll explore AI’s evolving role in industries ranging from online dating, to language learning, to hiring. Each guest is making incredible advancements in their field, and we’re excited to share their stories with you. Whether their names sound familiar, or are new to you, we’re confident that you’ll walk away with a better understanding of how AI is changing the way we live and work, each and every day. Thanks for tuning in–and we’re thrilled to have you!
In today’s episode, we’re joined by Burr Settles, research director at Duolingo, a language learning app that uses AI to create customized educational programs for users. Burr joined Duolingo in 2013, one year after the company was founded to build and oversee all aspects of the app’s robust AI capabilities. We’ll dive into everything from Duolingo’s early days of pioneering the use of AI in education, to what drove the surge in app users at the outset of the pandemic. Burr, we’re thrilled to have you on the show. Welcome.
Burr Settles Thanks for having me, Jim. Pleasure to be here.
Jim Freeze Great, great. So Burr, you joined Duolingo very early on, only after the app had just been launched basically. Prior to that, you were a post-doc at Carnegie Mellon after earning a PhD from the University of Wisconsin, Madison. So very impressive academic background, and I’m always happy to have fellow big 10 folks on The ConversAItion. So we’re thrilled about that. What drew you to Duolingo, and what was it like working there in the early stages?
Burr Settles Well, my path to Duolingo is… Well, I don’t know actually, if any of us in the early days had a conventional path, but my background was in machine learning and artificial intelligence, natural language processing. And after I finished my PhD in Wisconsin, I did a thesis in something called Active Learning, which is machine learning algorithms that rather than just passively consuming data and learning how to parrot it back, they engage in the learning process. So it’s kind of a dialogue between–in the case of a lot of natural language applications–a human expert who is trying to teach the machine how to solve this problem automatically. So that’s what I did my PhD in, which led me to a postdoc at Carnegie Mellon, where I got to know Luis von Ahn, who was co-founder and CEO for Duolingo. I didn’t work on Duolingo at CMU. Although interestingly, I did co-draft the NSF grant in 2009 that funded Duolingo as a research project.
Jim Freeze Oh, interesting.
Burr Settles Yeah. There were just two projects on that grant, and I was on the other one at the time. But when my post-doc was ending in 2012 and I was on the job search, that was about the time that Duolingo spun out. And I interviewed there along with a bunch of other opportunities and was interested in taking the risk. I also studied computer science in school. I didn’t really know that natural language processing, applying computers and AI to language tasks was a thing until I got to grad school. And so I fell in love with it. And then Duolingo seemed like a no-brainer exciting opportunity at the time.
Jim Freeze Yes. It seems like a very natural fit for your background, especially given the grant you just talked about. So when you joined Duolingo, applying AI in education was a fairly new concept. What opportunity back then did you see in this space?
Burr Settles I don’t know how new of a concept it was. There had been lots of research and intelligent tutoring systems, and psychometrics and things, mostly for assessments, but also for some instruction. But most of that work was done in the math domain, where answers are kind of right or wrong, and there’s a lot more objectivity, and the concepts that you teach and the concept space is much smaller. With language, it was really kind of wide open. If somebody made a mistake, there are all kinds of nuances as to why. And the concept space, like the number of words you have to learn and grammatical concepts is much, much larger. So that was the challenge that we faced. And at Duolingo, the way we think about AI and instruction is to sort of scale the one-on-one private tutor experience to as many people as possible, not to replace great teachers, but most people in the world just don’t have access to a great teacher or a great language tutor.
And so we think AI is a way to scale that kind of experience to as many people as possible. And if you think about great tutors, I claim they’ve got three properties. One, they know the content really well. Two, they know how to make that content exciting and engaging. And three, maybe most importantly, they know how to get inside your head and figure out the things that you know, or you don’t know, the things that you struggle with that you’ve likely forgotten by now and need to go back and practice. And so we’ve turned those three properties into research programs, basically. So we’ve got AI for content development, for engaging learner experiences and for personalization.
Jim Freeze That last point was a really interesting one. Your research spans, as you mentioned, content development, kind of immersive experiences, knowledge modeling and personalization, which I think is a really interesting concept given how personal learning can be. Can you walk us through some of your projects that you’ve spearheaded, and their specific impact on today’s app and in particular, the role that AI plays throughout? You were just kind of touching on that.
Burr Settles Yeah. We can walk through a few different projects in that last group, like that personalization thing. So one of my first projects was to create a placement test that was computer-adaptive and very efficient. So when I first joined the company, it didn’t matter if you were a newcomer to French, or if you had taken four years of high school French, or if you’d lived abroad, you still had to start with lesson one of basics one at the beginning of the curriculum. And so, one of my first projects was to create a computer adaptive placement test that actually borrowed a lot from my background in active learning, in machine learning. So there, there’s a collaboration between the machine and the human to find the right decision boundary for some kind of classification.
In this case, the classification is whether or not the Duolingo user can solve this task or not. And so we would start with a medium difficulty question. And if you got that correct, then we could give you something slightly harder. Or if you got that wrong, then we could back off and give you something easier. And we can quickly zero in on where in the curriculum you belong. And if you’re not familiar with Duolingo, it’s organized sort of in this progression, a mastery learning kind of thing, where all the lessons are organized into skills, which are thematic. And then as you complete those skills it kind of unlocks the next set of skills. And so that way the placement test would unlock you to, I don’t know, like the 10th row in the curriculum, if that’s where you belonged or the 20th row.
Jim Freeze That’s very interesting. It sounds like the technology does a great job of accounting for highly nuanced personal processes for individuals along with knowledge recall. I think about that because I did four years of high school Spanish, but I remember so little of it. I don’t know if somehow I could easily recall that by using the app.
Burr Settles Well, that’s also another aspect of the personalization models. So one of the second projects that I actually worked on after joining was a space repetition model. And so the idea behind space repetition is that you learn something better by practicing it over longer intervals rather than cramming. And there’s an associated phenomenon called the Lag Effect, which is, each time you practice it, if you wait longer than you did the last time to practice it the next time, then it kind of reinforces the connections in your brain and your memory. And so we had millions of users who were doing, at the time, hundreds of thousands of exercises every day were orders of magnitude larger than that now.
But at the time we had all these data that we use to fit, we created some novel nonlinear regression models. We call half-life progression, and it can predict how likely it is that a user has forgotten a particular word or a particular grammatical concept, given their history with it. The number of times they’ve seen it in the past, the number of times they got it right or wrong, how long it’s been since they last saw it within the app. And then we can use that to prioritize the things that we put into practice sessions.
Jim Freeze Wow. Talk about personalization. That’s terrific. So I recently read an article in TechCrunch that said that you saw your fastest growth period in March of 2020. And I think we all know what happened in March of 2020. Can you share a little bit about how the pandemic impacted the app and the language learning space more broadly, and any trends that you saw?
Burr Settles As different countries went into lockdown, you could look at our usage, and there seemed to be a two or three day lag. We would see a spike for a particular country and the number of people who are downloading and starting to use the app. And that was usually about two or three days later after broad lockdowns were sort of instituted. And so I don’t know the exact numbers, but the Duolingo learning app grew significantly during that time. And it sort of plateaued as things started to open back up. Now we’re kind of back to our pre-pandemic growth levels, just at a higher baseline than we were at before.
Jim Freeze It is interesting to see how much the pandemic changed behaviors and impacted businesses. I’d love to double-click, so to speak, on one specific AI application within the Duolingo app, which is the English proficiency test. Can you share a little bit more about how AI powers the entire end-to-end experience and how the pandemic impacted the specific usage of the English proficiency test?
Burr Settles Sure. Most people are familiar with the Duolingo learning app, which is kind of our flagship project product. But several years ago, five years ago, actually, we launched the Duolingo English test, which is an online English proficiency test. So let’s say you were from India or China or some other non-English speaking country, and you want to study in the United States or go to university in an English medium school. In addition to taking things like the SAT or the ACT, you would have to take a test to prove your English proficiency level. And most of those tests historically have required testing centers. They’re very expensive. These testing centers are not in every city. They’re generally just in capital cities, and many countries don’t even have one. And so our idea was to have a computer adaptive English language proficiency test. And being computer adaptive, it could be much more efficient.
So rather than taking four hours, it could be one hour. And you could take it anywhere online as long as you had internet access. So that was the basic idea, the constraints that we were trying to satisfy. And so we leaned on our strengths, our core competencies in AI and natural language processing to generate the test content or pseudo automatically do that. There are still subject matter experts involved in the creation. And we also use AI to automatically grade those items, automatically administer them during the test to make it as efficient as possible. And then also to work along with proctors to ensure the security and integrity of the test scores before they’re released. There’s AI involved in every step of the process. You asked about the effects-
Jim Freeze The pandemic, yes.
Burr Settles … of the pandemic, right. Because it was such a new test, and it’s such a new paradigm, a new way of reaching students, universities and academic institutions can move sometimes at a glacial speed, the adoption was largely on fringe populations, people who otherwise wouldn’t be able to get to a test center or were low-income, high achieving students, which was great we were able to start serving those populations that were not being met by the status quo. But then as soon as everything locked down, we were suddenly the only English test that was a viable option for anybody who wanted to study in an English-speaking university. And so it was kind of this forcing function where universities who had spent several years testing the waters with it for things like student athletes, who typically, because of the competition and practice schedules, they’re unable to do conventional English testing.
So those were the primary people taking the test before. And then all of a sudden, we grew from a few hundred universities and institutions to more than 3,000 during the pandemic. And the test-taking volume grew by 2000%.
Jim Freeze It’s pretty amazing growth.
Burr Settles And it stayed there and continues to grow.
Jim Freeze Oh, that’s fantastic. So as a final question, I’d love to have you look into your crystal ball and share where you see an expanded role for AI in education. I mean, what do you think is possible within the next five or 10 years?
Burr Settles Well, going back to what I said before, using AI for creating content, creating engaging experiences and personalizing those, those are the core things that we’re focused on. And the technology that enables that I think will only get better. And so we’ll be able to reach more and more students who previously otherwise wouldn’t be able to be reached. And then also using the data and these analytics platforms that kind of fall out of natural usage of students. We can also create tools through platforms like Duolingo for schools to enable classroom teachers to have more insights into what their individual students are doing, what they know well, what they’re struggling with and help them be even more effective in the classroom. So I see it as a win-win both in terms of scaling a great learning experience out to as many people as possible. And then also leveling up people who are in more traditional classroom settings.
Jim Freeze What we’ve talked about on this episode is exactly what we try to do with conversation, which is to educate people about the impact of AI on business in particular on top has had a big impact on your business, but also just in society in general, and kind of the value that Duolingo is bringing. It’s been fascinating. I really appreciate you taking the time to join us on today’s episode. Thanks a lot, Burr.
Burr Settles Thank you very much for having me. We’re just getting started.
Jim Freeze It sounds like it. Thank you very much.
That’s all for this episode of The ConversAItion. Join us next time for an episode featuring Frida Polli, Co-founder and CEO of Pymetrics, an AI-powered talent-matching company serving millions of job seekers, and employers, every year.
We’ll discuss everything from how Pymetrics maps consumers to jobs based on behavioral indicators to developing technology that strips away human biases.
This episode of The ConversAItion podcast was produced by Interactions, a Boston-area conversational AI company. I’m Jim Freeze, and we’ll see you next time.
[UPBEAT MUSIC]
