Last time, we talked about Harry Potter‘s popularity: I mentioned Ozy’s theory that Harry Potter is popular mostly because it got lucky, and I contrasted that with my explanation that 8+ factors had all come together to make Harry Potter the sensation it was.
I want to move on to the meta-level question: How, in general should we decide between a simple theory like Ozy’s (it all came down to one factor, plus luck that snowballed!) to a more complicated explanation (it was eight or more factors)?
Two Failure Modes
I’m going to argue that we’re at risk of making two opposite errors every time we think of a question like “Why is Harry Potter so popular?” That is, we can fall into two different “failure modes.”
First, there’s the failure mode of oversimplify our model, the error I’m accusing Ozy of. We might build a model that doesn’t take into account the complexities of the situation. In practice, this usually ends up with the same problem Ozy demonstrated—shoving a lot of what we’re trying to explain into “random chance.”
On the other hand, there’s the failure mode of telling “just-so stories” about the thing we’re trying to explain: we find some feature of the phenomenon that we’re trying to explain and build a story around how that feature played a causal role.FN 1 This is the error that my last post might have made. A critic could argue that I’ve just identified some facts about Harry Potter and then come up with post-hoc just-so stories about how those facts led to Harry Potter‘s popularity.
I want to use my previous post as a (potential) example of this failure mode. I was already planning to do so in this follow-up post, and I’m going to use some of the (excellent) comments from the last post to help make the point.
Arguments That The Last Post Told A Just-So Story
In my last post, I argued that it wasn’t a coincidence that Harry Potter became so popular—that there were many features of Harry Potter that set it apart from competitor series. I made this list with the benefit of hindsight, and there are at least two ways someone could attack it.
First, they could argue that some of the “differences” I identified aren’t differences at all. They could say, as NN did, that Animorphs had good worldbuilding too, and thus worldbuilding couldn’t have been what set Harry Potter apart:
Speaking as someone who did read the entire Animorphs series, I think that it actually does have a decent amount of this sort of worldbuilding. Scattered throughout the books are all sorts of references to alien species we never meet (or only meet once, like the various alien monsters that the main villain shapeshifts into during fight scenes), planets we never visit, events that happen offscreen, alien words we’re never told the meaning to, etc.
Second, a critic could agree that the differences I pointed to are differences, but argue that they didn’t play any role in making Harry Potter successful. (As an example, if I’d picked some obviously irrelevant difference—”Harry Potter was successful because it had a red train in it”—then this criticism would be entirely valid.)
One particular form this attack could take is claiming that I could argue the same “difference” either way. For example, I claim Harry Potter was more popular than Animorphs because it was longer. But if Animorphs had been the more popular series, could I have argued equally plausibly that Animorphs‘ brevity explained its (hypothetical) greater popularity?
In this hypothetical universe, I might say that Animorphs was more popular than Harry Potter because the Animorphs books were much shorter—that kids would never have the attention span for a longer book, that publishing books every month or so maintained readers’ interest, and that short books require less of a time-investment, which makes it easier for popularity to spread by word of mouth.
To the extent that “Harry Potter became more popular because it is longer” and “Animorphs became more popular because it was shorter” both sound equally plausible, this suggests that my focus on the difference in length between Harry Potter and Animorphs might be a just-so story. That is, I might be grabbing onto a random difference and ascribing it causal significance—without any real evidence that that causal significance exists.
Both of these lines of attack are getting at the same general issue: did I really figure out the causes of Harry Potter‘s popularity, or did I just come along after the fact and point to some things that look like causes? As droid put it in a comment to the last post, the real question is whether “these attributes can only be identified reliably in retrospect” or whether they can “make better predictions about which [books will be popular in the future.]” To the extent that the supposed causes don’t have any predictive power, they may also not have any explanatory power. They might just be just-so stories.
Striking the Balance Between Oversimplification and Just-So Stories
This is the part of the post where I’d like to offer a brilliant solution for threading the needle between the twin failure modes of oversimplification versus telling just-so stories. Unfortunately, I don’t think there is a knockdown solution.
If we were building a formal, mathematical model of something, telling just-so stories would result in “overfitting“—the process of building a model that accounts for our noise so well that it loses the signal. In a statistical model, there are mathematical tools to avoid this. This may provide some reason to prefer formal models, if you have the choice—a formal model gives you an extra line of defense against just-so stories.
But of course we’re not building a mathematical model—we’re talking about Harry Potter. When we’re talking about something informally, I think the best solution is to be cognizant of the two potential failure modes. That is: constant vigilance!
Every time we try to explain something—the popularity of Harry Potter, why the crime rate dropped, an election result, why a friend treated us a certain way, whatever—we should guard against both failure modes. We should try our hardest not to have too simple a model that doesn’t include the relevant causes; we should also be aware of the danger of having too complex a model that includes a lot of just-so stories.
And remembering that almost any model risks erring in one of those ways, should help us maintain some humility as we develop explanations—something that the world could definitely use more of.