80% Solutions
There was a time not so long ago when I was an assistant principal of a middle school, and as part of my role I oversaw data analysis and personalized learning. Of course we didn’t call it personalized learning then - or we did, but nobody understood what that meant. One of the biggest areas that got personalized was the area that we were required by law to personalize: Individual Education Plans (IEPs) for students who had diagnosed disabilities.
I don’t want to get too into the weeds on how IEPs get developed or implemented - if you’ve been there you know that it’s a pretty grueling process - but when you come out the other end there are certain interventions that need to be carried out with fidelity in order to support students. At a conceptual level, I feel like every learner should have an IEP, and in theory I’m glad that there’s a mechanism to help kids get the support they need. In practice, I often found that I was navigating the Wild West of the early days of intervention software. Everything had these strict guidelines for what it would take to use the software with fidelity, and those strict guidelines were nearly always at odds with the reality of what happens in an actual school day-to-day. Stuff like, “the student needs to spend one hour, three times a week over a period of no less than 120 days” which on its face doesn’t seem that crazy, but it has to fit in with all kinds of other interventions, and then there are weeks when there are holidays or state tests or whatever else they can throw at you.
This set off my BS meter, which I think is pretty well honed, and I started pulling the efficacy reports for a lot of these programs (and because this is so long ago, and because I hold out hope that some things have changed, I’m not going to name names). Let me tell you: none of them had any sort of replicated or externally validated measures; all of them were reporting p-value that would never have been suitable for publication (like p < .15, which basically means that it’s useless once in every six cases). And that’s when I realized that all of this fidelity stuff was well-intentioned, but ultimately none of these interventions had been developed with the larger context of actual learners in actual learning environments in mind.
Let me remind you again: product in education is about ecosystems, and even if you’re building a specific tool to be used for a limited purpose it has to fit within the ecosystem. It’s not good enough for it to be bug free or feature perfect if it’s incompatible with the broader context of its use.
In the case of those early interventions (and, sadly, many things still), the fix for that was to try and constrain what the rest of the ecosystem could look like and try to mandate a very restricted context of use. I am not here to say that won’t work at all, but I think there’s something like a rule of thumb that says that the more you restrict the context, the more you restrict the amount of time your product can be effectively used. If it needs laboratory conditions, you’ve got 30 minutes at best.
If, on the other hand, you intentionally create products that acknowledge the agency of the user and the diversity of the environment, then you’ve got something special. Within an education institution, I think of this as the 80% solution - the idea that you shouldn’t try and create 100% of the solution to the problem, because in education you don’t have a user but rather a participant that should be actively involved in creating the solution alongside the product. There is no way to standardize the people who will use the product, and for a lot of products we also should be cautious about atomizing the people and assuming that products need to be used by individuals to be effective.
People working on technology products may object that they have to create 100% solutions because their solutions do actually need to just work. I would probably want to differentiate at this point between products that support learning and products that support operations. The former allows the ecosystem to thrive in its purpose, while the latter makes sure the ecosystem just functions. That latter group often does require 100% solutions - if it’s tracking student lunch payment accounts or storing student records, then it does need to just work. At the same time, those kinds of products need to also hold on to an awareness that they aren’t the reason the ecosystem exists, and so they shouldn’t excessively constrain the purpose of the ecosystem.
For products that support learning, though, learners need to bring something of themselves to the product as well, and there need to be novel or unexpected ways that the product can be used. Think about a game like Sim City, possibly and inadvertently the greatest piece of educational software of all-time. It’s mechanics as a game work, and it’s code runs efficiently and effectively. Also, there’s no way that the designers were able to anticipate all of the different ways that people would build cities, the different things they might optimize for, the often random layouts people come up with.
*I spent way too long looking through Sim City pictures. Source*
It has to work, but it also has to work within the broader ecosystem, and the ecosystem is more than the product and services that comprise it - those all exist in service of something else, the learner. As Richard Culatta put it, “Nothing works anywhere, because nothing works everywhere.” (That article, by the way, should be required reading).
Other required reading this week:
Camera Above the Classroom - on an education product that supports operations in the creepiest manner possible. Ahhhh, the delicious potential of AI...
Silicon Valley Came To This Kansas School. That Started a Rebellion - on an education product that supports learning, and maybe a prime example of that Culatta adage above. I know quite a few people in the Summit world, and I strongly recommend visiting their campuses to anyone looking into who is doing interesting work in the Bay Area. I suspect there’s some truth to what Tavenner says in this article about resistance to change, but I also suspect there’s some undervaluing of the importance of getting the whole ecosystem right.
