Contra Yudkowsky on Quidditch—and a meta point

This post is a bit inside baseball for the rationalist community, but I promise to provide enough context for everyone else to follow the argument.

Eliezer Yudkowsky is wrong about Quidditch. He’s wrong on an object level. He’s wrong on a meta level. And the way he’s wrong on the meta level perfectly encapsulates a frequent rationalist failure mode.

Let’s back up and take in some context: Eliezer Yudkowsky is one of the most prominent members of the rationalist community and did quite a bit of blogging/writing at Less Wrong on the ways to become more rational. Among many other posts, he wrote rationality is systematized winning, which argued that “if the ‘irrational’ agent is outcompeting you on a systematic and predictable basis, then it is time to reconsider what you think is ‘rational'”.

By this standard, what Yudkowsky has to say about Quidditch isn’t rational—that is, Yudkowsky uses a flawed system, and the flaws in that system resulted in bad predictions—and will continue to lead to the same error in future predictions.

So, what did Yudkowsky say about Quidditch as depicted in the Harry Potter books? In writing Harry Potter and the Methods of Rationality, Yudkowsky repeatedly returns to the “irrationality” of Quidditch as it is portrayed in the cannon Harry Potter books.[FN: Note that, in this post, I am attributing views expressed by a character in HPMoR to the author of HPMoR. I don’t endorse that as a general practice. In general, the author of a work of fiction need not, and likely does not, endorse the views of characters in that work. However, based on the totality of Yudkowsky’s writings, I strongly believe that he does endorse Harry Potter’s views of Quidditch. If anyone disagrees, I’m happy to debate the issue in the comments.]

Yudkowsky has Harry express his views early on. Harry has the following exchange with Ron:

“Catching the Snitch is worth one hundred and fifty points? ”

“Yeah -”

“How many ten-point goals does one side usually score not counting the Snitch?”

“Um, maybe fifteen or twenty in professional games -”

“That’s just wrong. That violates every possible rule of game design. Look, the rest of this game sounds like it might make sense, sort of, for a sport I mean, but you’re basically saying that catching the Snitch overwhelms almost any ordinary point spread. The two Seekers are up there flying around looking for the Snitch and usually not interacting with anyone else, spotting the Snitch first is going to be mostly luck -“

“It’s not luck!” protested Ron. “You’ve got to keep your eyes moving in the right pattern -”

“That’s not interactive, there’s no back-and-forth with the other player and how much fun is it to watch someone incredibly good at moving their eyes? And then whichever Seeker gets lucky swoops in and grabs the Snitch and makes everyone else’s work moot. It’s like someone took a real game and grafted on this pointless extra position so that you could be the Most Important Player without needing to really get involved or learn the rest of it. Who was the first Seeker, the King’s idiot son who wanted to play Quidditch but couldn’t understand the rules?” Actually, now that Harry thought about it, that seemed like a surprisingly good hypothesis. Put him on a broomstick and tell him to catch the shiny thing…

Ron’s face pulled into a scowl. “If you don’t like Quidditch, you don’t have to make fun of it!”

“If you can’t criticise, you can’t optimise. I’m suggesting how to improve the game. And it’s very simple. Get rid of the Snitch.”

So, within seconds of learning the rules, Harry has diagnosed a key flaw in Quidditch. And this isn’t a passing observation; he returns to this issue over and over again. He even uses it as a sort of shorthand for rational thinking as a whole; for example, Harry praises another character by saying, “he’s the only other person I know who notices stuff like the Snitch ruining Quidditch.”

But Harry is wrong.

The Snitch doesn’t ruin Quidditch—it makes it great

Let’s back up. What makes a sport great? Well, let me propose two hypothetical awful sports by comparison. First, imagine a coin-flipping contest, where two contestants flip fair coins and whoever gets the most tails wins. “So exciting! You never know who might win.”

No. It would be deadly dull: There’s no skill at all involved (the coins are fair) and seeing who happens to get lucky doesn’t make a great sport.

Ok, let’s go to the opposite extreme: our next sport is going to be a height contest. We line all the contestants up, and whoever is the tallest is crowned the champion. No luck involved at all. Surely this is a thrilling sport!

Also no. Here there’s too little luck—there’s no point in a “sport” where the outcome can be determined in advance because there is no (or basically no) room for luck.

As I hope these examples show, there’s a continuum: On one side, you can have a contest that is all luck and no skill; on the other, all skill and no luck. Neither extreme is satisfying in the least. Instead, all existing popular sports fall somewhere in the middle area of the continuum: where skill plays a big role, but luck can be the decisive factor. People love American football because you never know which team might win on any given Sunday—and the same could be said of basically all other popular sports.

We want skill to matter, but for luck to still play a role.

* * *

With this framework in place, we can return to Harry/Yudkowsky’s critique of Quidditch. Harry’s point is essentially that Quidditch is too much like our fair-coin-flipping sport: The winner of the match is too random, since finding the snitch is largely driven by luck (or skills that aren’t visible to the audience, like the seekers’ skill at spotting the snitch). Harry points out that a team could do well on all the “skill” aspects of the game, but still lose the match when the other team catches the snitch by “luck”. And all of this is true enough.

But it misses a key point—Quidditch is not about winning matches. We see one major Quidditch contest, the Hogwarts Quidditch Cup, and it notably is not a tournament. Instead, it’s a contest to see which team can score the most points over the whole season. And winning matches might be random/depending on the seeker, but racking up points over the whole season certainly isn’t. The Harry Potter Lexicon articulates this well:

On the face of it, Quidditch scoring is unfair. In fact, it’s so unfair that you can barely call it sportsmanlike. Since catching the Snitch gains one side the equivalent of fifteen goals and ends the game so the other team can’t counter it, Quidditch is essentially a match between the two Seekers and nothing else. So what makes it so popular? Do witches and wizards just watch it for the violence and fancy broom tricks?

Not at all. Quidditch is always played in a series. Unless you’re playing an informal game in the apple orchard, every Quidditch match is part of a larger series of matches, and accumulated points are what count toward ultimate victory. The Quidditch Cup at Hogwarts goes to the team with the most total points, not the one who has won the most matches. The standings we see in the Daily Prophet for the British and Irish Quidditch League (DP1, DP2, DP3, DP4) list the teams in order of how many points they have in total, from the Tutshill Tornados with 750 points down to the lowly Chudley Cannons with only 230. Nowhere in the standings does it note how many matches each team won. Although we don’t see evidence of it, there must be a similar system for the World Cup, which would imply that Bulgaria and Ireland were the top scorers in the world that year.

As a result, Quidditch is actually a well-balanced game. The seekers give it that “any given Sunday” feeling where you never know who might win an individual match (and never know if the match might end, keeping each minute exciting). On the other hand, winning matches doesn’t really matter in the long run, and the seeker’s contributions will largely cancel out. So the highest-ranked teams will be the ones that have the most overall skill in all positions. This scoring system does a great job of striking a balance between coin flips and height measuring; it hits that sweet spot of enough randomness to be interesting and enough skill to be compelling.

Harry James Potter-Evans-Verres was wrong.

* * *

So what? A fictional character was wrong. What makes that worth blogging about?

This is worth blogging about because Harry’s error—Yudkowsky’s error—is the achilles heel of rationalists. Harry had a logical insight, came up with an argument for why it was correct, never heard a good argument against his position, and concluded that the popular position was entirely wrong.

What Harry failed to account for was that the popular position could be entirely right even if no one he met could make a good argument for it. Harry should have, but didn’t, have the following thought

Thousands of seemingly intelligent people have conformed to a system with the current rules of Quidditch. This provides strong evidence that the current rules have something going for them. Even if no one can explain what this advantage is, I should have a strong prior that the rules have advantages—and thus require very strong evidence before concluding that they don’t.

(Yes, Harry can think with hyperlinks. That’s just the sort of guy he is.)

Note that this is not just Chesterton’s Fence.

If you’re not familiar with Chesterton’s Fence, I’ll let Scott Alexander explain:

G.K. Chesterton gave the example of a fence in the middle of nowhere. A traveller comes across it, thinks “I can’t think of any reason to have a fence out here, it sure was dumb to build one” and so takes it down. She is then gored by an angry bull who was being kept on the other side of the fence.

Chesterton’s point is that “I can’t think of any reason to have a fence out here” is the worst reason to remove a fence. Someone had a reason to put a fence up here, and if you can’t even imagine what it was, it probably means there’s something you’re missing about the situation and that you’re meddling in things you don’t understand. None of this precludes the traveller who knows that this was historically a cattle farming area but is now abandoned – ie the traveller who understands what’s going on – from taking down the fence.

As with fences, so with arguments. If you have no clue how someone could believe something, and so you decide it’s stupid, you are much like Chesterton’s traveler dismissing the fence (and philosophers, like travelers, are at high risk of stumbling across bull.)

Chesterton’s Fence warns against tearing down a system when you don’t understand why it came about. But Harry didn’t do that—he had a very good theory for how Quidditch came to have a seeker: that it was a good role for some noble’s kid. This explanation seemed plausible, and even (within the HPMoR universe) turned out to be correct.[FN: And, stepping outside the fictional universe, Yudkowsky presumably had a similar theory for why Harry Potter was written with seekers in a starring role: so that Harry would have something dramatic to do in Quidditch matches.] So the point isn’t just “don’t tear down fences if you don’t understand why they were built.”

Instead, the point is this: If there’s a fence that a lot of people have chosen not to tear down, then you should be very reluctant to tear it down. You should be reluctant even if the fence looks dumb to you, and even if you can think of historical reasons for people to have built a fence, and those historical reasons don’t change your mind about the fence being dumb right now. Sure, you might be the lone genius who sees through the conventional dogma. But you also might be missing something. Even smart people can forget that Quidditch isn’t about winning matches.

In general, one of the greatest virtues of the rationalist community—and of Yudkowsky in particular—is a skill at Taking Ideas Seriously. That is, rationalists will follow through on ideas and take them to their logical conclusion. They’ll act on the consequences of ideas, instead of just dismissing those ideas as “interesting,” but taking no action. (Or, even worse, intellectually accepting an idea as correct but still taking no action.)

Here’s one way of articulating this rationalist virtue:

One of my favorite results from the Less Wrong Survey, which I’ve written about again and again, shows that people who sign up for cryonics are less likely to believe it will work than demographically similar people who don’t sign up (yes, you read that right)—and the average person signed up for cryonics only estimated a 12% chance it would work. The active ingredient in cryonics support is not unusual certainty it will work, but unusual methods for dealing with moral and epistemological questions – an attitude of “This only has like a 10% chance of working, but a 10% chance of immortality for a couple of dollars a month is an amazing deal and you would be an idiot to turn it down” instead of “this sounds weird, screw it”.

This is rationalists’ greatest strength.

It’s also their[FN: Our?] greatest weakness. Sometimes “this sounds weird, screw it” is exactly the right answer. Sometimes—not all the times, but sometimes—there’s a reason that no one else has implemented the solution that’s just occurred to you. Sometimes you’ve seen to the heart of the matter, and everyone else is blinded by convention. But sometimes, you’ve missed that Quidditch isn’t about winning matches.

Rationalists should balance their skill at taking ideas seriously with a habit of epistemic humility. Epistemic humility means thinking that, if you disagree with a bunch of people, there’s a real chance the problem is on your end. You should therefore require extreme proof before going against such a strong consensus—much stronger proof that you would naively feel like you need, from the inside view. [FN:This is similar to the concept of “epistemic learned helplessness” except that epistemic learned helplessness is appropriate only in the 99% of the time when you aren’t the smartest person around. I’m saying that even in the 1% of the time when you are the smartest person around, taking ideas (too) seriously might be a bad idea.] Rationalists can work to cultivate epistemic humility, and many of the best do. My criticism of Harry and Yudkowsky is that they haven’t.

Or, because a picture is worth 2,494 words, I think Yudkowsky might be this guy:

This happens in geek circles every so often. The 'Hey, this is just a system I can figure out easily!' is also a problem among engineers first diving into the stock market.

Related Posts

12 thoughts on “Contra Yudkowsky on Quidditch—and a meta point”

  1. One problem I have with your analysis here is that the point about epistemic humility– about not assuming that “thousands of seemingly intelligent people” are obviously wrong about something– doesn’t really work if those thousands of people are all made-up fictional characters (or worse: offscreen fictional characters!). I always interpreted the Quidditch exchange (and most of HPMOR in general) as Yudkowsky [affectionately] critiquing Rowling’s worldbuilding more than as an internally-consistent narrative.

    Rowling can hardly respond to Yudkowsky’s criticism by just appealing to how many thousands or millions of fake people she can invent who all agree that she is right and he is wrong. She could make the game-versus-season argument that you make in this post (which is a good argument), but notice how that argument has nothing to do with Chesterton’s fences or epistemic humility; if there’s a “fence” here– even if it’s a good fence!– it wasn’t built and maintained by generations of a community who know better than you. It was just built by a single British YA author.

    Yudkowsky was constrained by the established Potter setting. If he wanted to write his fanfic to critique the Harry Potter universe, then he was going to have to either have his Harry Sue disagree with a vast number of people… or write in a legion of new characters, all of whom also agreed with him. Surely the latter would have been an even more ham-fisted way to prove his point, yes?

    1. That’s a good point. I think my post doesn’t do enough to distinguish between criticisms of Yudkowsky and of (Yudkowsky’s) Harry. (In slight defense of the point I’m making, both Yudkowsky and Yudkowsky’s Harry are frequently invoked examples of rationality done correctly, so criticisms of Harry aren’t entirely beside the point when arguing that rationalists should try harder to adopt epistemic humility.)

      But you are absolutely correct: Harry is failing to demonstrate epistemic humility because he disagrees with thousands of in-universe characters, but that criticism doesn’t apply to Yudkowsky.

      Nevertheless, I still think that Yudkowsky is failing to show epistemic humility in a subtly different way, but still in the way I’m talking about. I think that his real/out-of-universe belief is that Rowling wrote a very flawed sport just to have a hero vehicle for (her) Harry, and that the large number of fans who love Quidditch just didn’t notice how flawed it is. Thus, he’s not thinking that he noticed a flaw that thousands of fictional people didn’t notice, but he is claiming to have noticed a flaw that Rowling, her editors, and millions of fans didn’t notice (or noticed and chose to ignore).

      However, the evidence suggests that he’s mistaken about that. (We don’t have great evidence, but the Daily Profit Quidditch standings that HP Lexicon cites provide at least some evidence that Rowling thought of the issue and had the same solution that I propose.) So Yudkowsky is still incorrectly concluding that he figured something out that escaped large numbers of people. This still seems like a failure of epistemic humility, though a different one.

  2. “Eliezer Yudkowsky is wrong about Quidditch. He’s wrong on an object level. He’s wrong on a meta level.”
    It’s called fractal wrongness! When someone is wrong on all possible levels. And it even has a definition on the rationalwiki, I did not expect that since I saw it on a forum sig.

    Also a point that goes in your sense is the Quidditch match at the beginning of HP 4 where the team winning the game (and the World Cup) is not the team grabbing the snitch.

    1. I’m not sure I’d go so far as saying that he’s wrong on every level. I’m pretty sure I can think of at least one or two levels on which he’s right!

      (And besides, when I hear “fractally wrong”, I think of something that’s wrong on many, many levels. Does two really qualify?)

  3. However, based on the totality of Yudkowsky’s writings, I strongly believe that he does endorse Harry Potter’s views of Quidditch. If anyone disagrees, I’m happy to debate the issue in the comments.

    I’m going to call you on that. In the very chapter you linked, a character makes basically the argument you suggested. Not in as much depth but it’s there in outline:

    She was one hundred percent on the side of Harry Potter that it was time for Hogwarts to give up on those gibbering slowpokes and just change the rules, starting here and now. But not by eliminating the Snitch, that was going all the way back to eleventh century Kwidditch. It didn’t matter if Headmistress Hufflepuff had first introduced the innovation because one of her students had wanted to play the game but not been suited to the usual roles. Snitches had caught on internationally because it was more exciting when the game could always end in the next minute.

    Maybe Yudkowsky does agree with Harry overall, I don’t know. But it’s not because he hasn’t considered the “snitch makes the game more exciting” position.

  4. “As I hope these examples show, there’s a continuum: On one side, you can have a contest that is all luck and no skill; on the other, all skill and no luck.”

    No, they don’t. A height contest is a test only of innate ability, or “talent”, but skill is a combination of talent and experience. A better comparison would be chess: no randomness (other than who plays first, I guess, but that’s pretty small), all skill. And most people appear to feel that chess isn’t a particularly thrilling game to watch, but some people enjoy it.

    And I don’t buy the point-based argument. It’s still vulnerable to exactly the same problem: a team with a fantastic seeker that is otherwise middling can rack up massive points over the course of a season. Indeed, it appears that Bulgaria did just that in the lead-up to the World Cup, unless they were just having a REALLY bad day when Ireland spanked them in the non-seeking roles.

    Also, if we believe the word of the author herself (which seems reasonable), then we know exactly why the snitch is a part of Quidditch: to infuriate men, because in her experience it seems absurd to men that the snitch is worth so many points. Source:

    So while Harry is guilty of failing to exhibit epistemic humility, I would say that Yudkowsky is correct. I disagree with your argument that 150 points for a snitch is good game design, and the snitch was even explicitly DESIGNED to be bad game design! Yudkowsky isn’t disagreeing with Rowling, he’s agreeing that it’s well designed to be poorly designed.

  5. Even if only the total points are important, two teams can cooperate to raise the scores arbitrarily high by simply ignoring the Snitch and scoring points with the Quaffle over and over. The Slytherin and Ravenclaw teams arrange to tie the House Cup during the HPMOR finale by using this method.

  6. I’ve noticed myself making this key error a great many times, especially recently, as I’ve begun to crack a few social quandaries.

    Small talk, for example. No one could give me a good explanation of why small talk is a good idea. I took this, my natural disinterest in it, and my inability to immediately see any use for it as proof that it was more likely than not a useless practice.

    It was worse because when I asked people why it was worthwhile, they didn’t just fail to give me a good explanation, they INVENTED BAD EXPLANATIONS on the spot. When I finished tearing those apart, I was all the more confident that my analysis was correct.

    In the end, I’ve figured out that small talk is useful after all, but it has not escaped my notice that if I had “gone with the crowd” about small talk, I would probably have figured out why small talk was useful more quickly, since I’d have interacted with it so much more. I also would have benefited to some degree from what small talk provides in the meanwhile, and gotten better at it in the process.

    This point has been driven home to me over and over again in areas where most people don’t know why what they’re doing works, and don’t have the discipline to admit that. My improved heuristic is to take someone’s inability to give good reasons, and their presentation of several bad reasons for something as much weaker evidence either for or against the thing in question.

    In short, I ask for good answers if they have them, but if they don’t, I don’t update very much at all against the thing in question. I have a weak expectation of strong evidence in favor, and a strong expectation of weak evidence against. Most people don’t understand what they’re doing, so their poor presentation skills are only weak evidence of the actual uselessness of any given thing.

    As for Quidditch itself, I think your analysis is fine. I do note that, to my knowledge, not every final match of Quidditch was delayed in an attempt to score extra points, and indeed, not every match consisted of the team with better players trying to rack up points while simply interfering with the other team’s Seeker (far easier to stop them from catching the Snitch by interfering with them than by catching it one’s self). In the end, Rowling wrote a world with some inconsistencies, and Yudkowsky could try to clarify them in one direction or the other, but couldn’t do so flawlessly either way without restructuring some major history of the story.

  7. I agree with the general point, but I don’t agree with everything in this post.

    “Instead, the point is this: If there’s a fence that a lot of people have chosen not to tear down, then you should be very reluctant to tear it down.”

    As someone pointed out above, the people who decided not to tear down the fence (or in this case, the snitch) were fictional.

    However, lots of real people have decided to tear down the fence. In real life muggle quidditch, the snitch is usually only worth 30 points, not 150.

    Also, the vast majority of real sports have nothing even roughly equivalent to the snitch, which seems like something of a Chesterton’s Fence in itself.

    So wouldn’t the more humble thing to do be to defer to the judgment of real people, and assume that standard sports rules are significantly better than Quidditch rules?

    (By the way, I wrote a few hundred words about why, contra contra Yudkowsky, I do personally think Quidditch might be improved by removing the snitch, but I decided to delete it and spare you all that display of tedious nerdery.)

Leave a Reply

Your email address will not be published. Required fields are marked *