The system by which the Hugo nominations are converted into the final ballot, known as E Pluribus Hugo, is coming up for renewal at this year’s business meeting, and I thought it would be interesting to see what the actual impact on the last five years of ballot papers has been. (I was involved with the administration of the votes for all but one of those years.) Mike Glyer has written a passionate defence of EPH, which I hope you will all read; but I thought I would add some empirical analysis.
(By the way, I keep an archive of all known Hugo statistics here.)
For each of the last five years, including three when there were also Retro Hugos, I have looked at the published results and compared the real final ballot with the hypothetical final ballot that would have resulted from taking the same nominees and votes, but simply ranking them in the order of total votes received, as would have been done under the old system (hereafter, “the old system”). I have assumed that all disqualifications and withdrawals were the same as in real life, and I have also assumed that nominees who missed the ballot because of EPH were in fact eligible, which they may not have been.
Apologies for posting this as an image; I generally love WordPress, but it is not good at tables. I have also noted below the final ballot ranking of those finalists who got onto the ballot because of EPH, in brackets after each entry.
It’s interesting to note that while the intended effect of EPH was to disadvantage nominees whose support is too similar to other popular nominees (and thus to boost nominees with more localised support), another important effect has been to split ties. It is not impossible to have a tied result in EPH, producing a ballot with more than six finalists, but it is much less likely to happen than under the old system. For voters and administrators, this must be counted as an advantage. With at least 19 categories on the ballot, it is already quite long enough without being extended further by a quirk of the numbers.
The overall effect on Hugo ballots is that 19 finalists in 18 categories who would not have qualified under the old system did qualify for the final ballot under EPH, and 29 finalists in 26 categories who would have qualified under the old system were kept off the ballot paper by EPH. There are also two cases (not otherwise noted here – you can check them out for yourself) of nominees who were disqualified by the administrators under EPH, but would not have had enough votes to even be disqualified under the old system.
There is a surprising skew in terms of diversity. (My apologies in advance to anyone whose identifications I have got wrong, even by implication; I’ll try and fix any mistakes brought to my attention.) Of the 29 potential finalists excluded by EPH, 12 (counting Bill and Ted Face the Music, but not Camestros Felapton) are white men or by white men. Of the 19 finalists who owed their places to EPH, 12 (counting Tenet) are men or by men (all white, I think, apart from John Picacio). Basically the potential finalists excluded by EPH were 41%-45% white men, and the finalists who got onto the ballot thanks to EPH were 58% white men.
These are small numbers of course. There are only four cases in three categories where women who would have qualified under the old system were replaced by men thanks to EPH (all in 2018), and one should balance that against two cases where men were replaced by women thanks to EPH (one in 2019, one in 2021). So the case that EPH systematically replaces one gender with another is weak. The stronger effect seems to be that in the process of splitting ties, the nominees excluded are more often women than men. Is this simply because there are more women nominees in general these days?
I actually pointed out back in 2016 that the effect of EPH on recent ballots then would also have been to make them more male. It is interesting that this effect seems to have continued. I’m generally in favour of EPH for its deterrent effect on slates and also (we’ll get to this) its impact on the Retro Hugos, but we must also be clear about the empirical evidence of its effect on the ballot. (In, admittedly, only about 10% of all categories.)
I have also recorded the final ballot ranking of each of the 19 finalists who were got their place because of EPH. One of them actually went on to win their category (Gardner Dozois, Best Editor Short Form, 2019) and two won second place. One took third place and another took fourth place; five ended in fifth place, six in sixth place and three finished in seventh place, below No Award (puppy candidates in 2017). So the majority of finalists brought onto the ballot by EPH are not especially popular with voters deciding the winner (as you would expect from those at the lower end of the nominations curve), but the exceptions are exceptional.
Retro Hugos and EPH
Three years of Retro Hugos have been subject to EPH, 1943 (awarded in 2018), 1944 (awarded in 2019) and 1945 (awarded in 2020). The effect here has been much more drastic. It is of course possible that different administrators might have made different calls about which categories to allow on the ballot, but I’m proceeding here on the counterfactual basis that all the same categories would have been run with all the same votes cast.
Because of the lower overall numbers participating in Retro Hugo nominations, the likelihood of ties for the last place on the ballot is much greater, and EPH has come into play as a tie-breaker much more frequently. In those three years, no fewer than 19 potential finalists who would have qualified for the ballot under the old system were knocked off it by EPH, and only 4 were added to the ballot by EPH. All (as far as I can tell) were white men or by white men, so there is no diversity angle to consider.
Of the four lucky stories (they are all stories) added to the ballot by EPH, two finished in last place in the final ranking, and one in fourth, but one, “I, Rocket”, by Ray Bradbury, won.
There’s an obvious fix to make the Retro Hugos less hassle for everyone, which is not to run them in the first place. However, if the decision has been made to run them, I think EPH helps rather than hinders the process, and as an administrator I’d much rather have it on my side than not.
But the Business Meeting gets to decide these things, and I don’t.
I’ve added links to my (incomplete) analysis of the 2017 Best Fancast category in the comments on Mike Glyer’s article (including the spreadsheet I used to analyze the published tables on EPH that year).
EPH’s view of diversity comes down to “diversity of support” rather than demographic diversity, as you note: “to boost nominees with more localised support.” However, I think it has had an indirect effect on demographic diversity by changing the voting pool, due to the decisions of those who had previously been trying to game the system. It was not intended to chase anyone away – indeed, in the discussions leading to EPH we assumed that at least some Puppies were arguing in good faith and we tried to accommodate their stated concerns about explicit or implicit slating by anyone, regardless of political affiliation. But the fact that a bunch of folks seem to have responded to the adoption of anti-slating measures by giving up on the Hugos does seem to have increased the demographic diversity of nominees from those who have remained.