I could spend a career on that, I thought.
Five years later, I published my first peer-reviewed research paper in Artificial Intelligence in Education, the culmination of years of work during my PhD. The topic? Applying deep-learning techniques to predict how likely students are to pass an exam before they take it. In my mind, it was a contribution to Khan’s vision of education – using AI to help students and teachers know when students were ready to take their exams.
It was also a complete and total flop.
Teaching is already “good enough”
So, why did my paper fail? I was solving a problem that no one actually had – students pretty much know how they’re going to do on an exam, and so do their teachers. Even though it sounds magical for AI to be able to predict a students test score before they touch an exam, the underlying educational problem just isn’t very big. This is, in fact, emblematic of a larger category of “small problems” in education that researchers love to study. They could be summed up with the following assertion:
Teachers are good enough at teaching, and students are good enough at learning.
Seems heretical, right? It’s also exactly the opposite of what much of the current buzz about AI says: “AI will create better content.” “AI will help students learn to program faster.” “AI will help you choose a major.”
To be clear, I think all of these things are true. However, we need to ask ourselves: how big could the effects of these types of tools be? If we solve these problems, will education be radically different, or 2% better?
Better teaching creates minimal gains
Luckily, researchers have taken the time to answer this question for us. In educational research, impact is measured with “effect size”, a statistically normalized way of measuring how strong the impact of a study is. Although it depends on the exact study, an effect size of 0.1 translates to about a 1% increase in an average student's grade.
So how big are the effect sizes in studies where we try to help teachers teach better or students learn better? On average, about 0.1. Even at the top end, the best studies have effect sizes of at most 0.5. This means that the greatest, groundbreaking education techniques and technology tends to raise students' grades by about 5%, or about one partial letter grade (e.g., from a C+ average to a B- average).
That’s still pretty great, and we should absolutely pursue those projects. Let’s squeeze every last percentage point we can out of this new technology! However, we should also ask ourselves: Is a 5% increase in grades (and the increase in learning that represents) the ceiling of what AI can do for us?
No, it’s not. But to break this ceiling, we need to be creative in how we apply AI to education. If we just tinker with the old way of doing things — but with AI — the benefits will simply plateau as they have in the past.
We already know how to revolutionize teaching, but we don’t
We already know how to revolutionize teaching. In fact, it’s been known about for well over half a century! It’s a type of instruction that has regularly been shown to increase a student’s grade by 20% - enough to turn failing students into average students, and average students into star performers.
The method? One-on-one tutoring.
So, why aren’t we doing it then? The answer is simple: scalability. In most educational settings, it’s not possible to have one-on-one tutoring for everyone due to cost, time, and having enough instructors. This conundrum has been ignominiously dubbed the two sigma problem.
At this point, you may already be saying “Oh, we could use AI systems as tutors!” However, there is one important question to consider before we get there: why is one-on-one attention so much better than any other type of teaching?
AI tutors under the current model is more of what’s not working
Although academics come up with lots of complex frameworks to discuss it, understanding why one-on-one tutoring works isn’t terribly complicated. Here’s a personal example of why it’s successful.
Despite being fairly successful academically, I had a moderately bumpy road in high school. My sophomore year I ended up failing out of my pre-calculus class. And not by a little, by a lot. My final score for the semester was around 30%. I don’t have the kindest memories of my teacher, but the truth is that although with a better teacher I may have scored a bit higher, there’s no way I would have passed.
Ultimately, the teacher wasn’t the problem. I was socially isolated in a new school, I had too many extracurriculars, my sleep schedule was bad, and due to stress and sleep debt I was sleeping through my first period math class.
With all these headwinds, neither ChatGPT nor a better teacher would have saved me. I still would have failed. This illustrates why even the best teacher and learning interventions can only do so much. What happened after also illustrates why one-on-one tutoring is effective.
Metacognition, not cognition, is the ingredient to success
Finally, I couldn’t keep my problem a secret anymore. My parents found out. They forced me to trim down my extracurricular activities; they took away my phone so I’d go to bed on time; they tracked my homework to ensure I didn’t dig a hole for myself again; eventually I ended up switching to a different school where I had more friends.
The one-on-one time I had with my parents had little to do with pre-calculus. To be honest, I doubt either of my parents remembered pre-calc enough to be particularly helpful. But what I lacked was not the cognitive ability needed to succeed, it was the metacognitive ability.
Metacognition is just a fancy word academics use as a catch-all for the internal dialogue we use to organize ourselves. Things like goal setting, prioritization, emotional regulation, and accountability. Time and time again education research in a number of fields has shown that the reason people don’t succeed has much more to do with metacognition than cognition, and that it is the benefits of another person who aids you in these difficult metacognitive tasks that gives one-on-one tutoring its power.
Conclusion: Don’t just build faster horses with AI and education
A quote often attributed to Henry Ford says: “If I’d have asked people what they wanted, they’d have said faster horses.” Though the quote is likely apocryphal, the moral is a useful one when building disruptive technology: we sometimes have to think beyond what the customers articulate as solutions, to determine if there is a better way to solve the underlying problem.
For a century, the best way educators have been able to help their students is by getting a little better at teaching here and a little better at learning there. AI is going to accelerate this process even further, unlocking new pedagogies and teacher techniques not previously possible.
But we shouldn’t just build a ‘faster horse,’ we should begin to look beyond – to what today still feels unimaginable. Whether you believe we’re headed towards an AI singularity or towards continual marginal improvement on the existing AI systems, one thing is certain – this AI wave is in its infancy. GPT-4 is the equivalent of the Model T, or the IBM Mainframes of the 1970’s.
Imagine a world where AI costs orders of magnitude less than it does today (The CEO of OpenAI thinks it might go to zero) and the GPT-12 model is fine tuned to your data. Imagine a world where a 10th grader struggling with his sleep schedule has an AI buddy who helps him set better goals, empathizes with the problem’s he’s having with his friends, and get him back on track before things get ugly.
Imagine AI as a real tutor. Click below to learn more about AI.
Additional resources:
Securing educational institutions in the digital age
How to Use ChatGPT for learning
