A blog for better streets and public spaces in Portland, Maine.

Wednesday, November 2, 2011


Yesterday, the Maine People's Resource Center released a poll (detailed results here) to try to preview the upcoming mayoral election. The poll itself projected a victory for Mike Brennan, and that's been the conventional interpretation coming from the press as well (witness analysis from Mike Tipping, one of the poll's sponsors, and the Press Herald).

However, as I'll illustrate below, the projected Brennan victory isn't statistically supported by the poll's data. The most we can say is that Brennan is a front-runner for 1st-choice votes. That will get him far, but the dynamics of the ranked-choice ballot are too complicated for a poll like this to predict the actual winner with any confidence.

In spite of its shortcomings, this is the only poll we're going to get before election day, so here are some observations:
  • The poll decided to eliminate six lower-tier candidates (Bragdon, Lapchick, Dodge, Haadoow, Vail) from poll questions. This group of six received 11.1% of the poll's 1st choice preferences, which means that each candidate from this group received, on average, 1.85% of the vote (some probably got more, some less). Ralph Carmona, who WAS included in the poll, received only 1.4%. So the decision about whether or not to include a candidate in the poll seems to have been somewhat arbitrary.

  • But let's talk statistics. The poll surveyed roughly 500 people (actually 477, but let's use a nice round figure to make the math easier). That, they say, gives the results a 4.44% margin of error. I'm assuming that they're assuming a normal probability function distribution (i.e., your typical bell curve) in calculating their margin of error.

    A poll with this sample size would work fine with a regular, one-round election of two or three candidates. But it's much less reliable with nine candidates (plus a big, vague "other" category) in the mix. And it's WAY more problematic when instant runoff dynamics come into play (more on that later).

  • To illustrate: suppose you're blindfolded and instructed to throw darts at a wall that's painted partly red, and partly black. You have a judge to tell you what color each dart hits, and you're supposed to infer, from what he tells you, how much of the dartboard is colored black, and how much is red.

    If you have 500 darts, and 400 of them hit a red section, you might reasonably conclude that 80% of the wall is red. And, based on the normal probability distribution, there's a 95% chance that you'd be within 4.4% of the correct answer.

    But what if there are 15 different colors, and you still only have 500 darts? Your targets would be much smaller, which means that there's more likelihood for error, and therefore you'd be much less confident in your inferences. There could be a sizeable part of the wall that you don't hit once with any of your 500 darts, for instance, and there might be another tiny section that you accidentally hit fifty times.

    The proportion of random error to the to the size of each target is much higher in this case. And that's what's happening in this poll. The poll report would have been a lot more honest if it had included error bars for each candidate, like this (the black bars show where each candidate's bar would be if the estimate were increased or decreased by 4.4%):

  • In this view, it's clear that Jed Rathband, Dave Marshall, and Markos Miller (near the center of the graph) might actually be ahead of putative "front runner" Nick Mavodones in terms of 1st-choice preferences: all of their error bars overlap in the 9%-11% range.

    So, grain of salt number one: with fifteen candidates in the mix, the margin for error in this poll is very large in proportion to the putative results.

  • These error margins become exponentially more problematic when the same poll tries to extrapolate the results of a series of instant runoff reallocations. The pollsters seem to reason thusly: "Carmona gets eliminated in the 7th instant runoff round, and 30% of Carmona's voters picked Rathband as their second choice, therefore, Rathband should get 30% of Carmona's ballots to be boosted from 8% to 8.6% of the voters' top choices."

    OK, but let's recall that there were only seven people of the 500 polled who picked Carmona as their first choice. That's way too small a sample size from which any statistician worth her salt would draw any conclusions. If you see seven Canadians at Old Orchard Beach and three of them are smoking, you can't conclude that 43% of all Canadians smoke. Similarly, inferring that 30% of Carmona voters will choose Rathband as their backup is statistically spurious.

    But that's exactly what this poll is inferring for all of the lower-tier candidates as the pollsters goes through the motions of a ranked-choice election. In each round of instant runoff possibilities, the pollsters are building on, and multiplying, their statistical errors.

    By the tenth instant runoff round, they're reallocating Markos Miller's votes based on a sample size of 35 poll respondents - still way too small, and by then they've more or less arbitrarily re-allocated about a third of the ballots to other candidates.

    These conclusions are built on a logical house of cards - and the flimsy logic gets geometrically flimsier as it goes.

    So, grain of salt number 2 for this poll: ignore this poll's instant runoff projections. They're worthless and frankly they make the pollsters look silly.
OK, so flaws aside, there are some good things about this poll (and as I said, it's the only poll we're going to get pre-election, so might as well make the most of it). So here are some interesting things that we CAN conclude from this poll, in spite of its flaws:

  • Brennan and Stimling are clear front-runners for 1st-preference votes.

  • But both of the front runners will get, at most, about a third of the ballots in the first round. That means that they'll need to accumulate 15% to 20% of lower-choice votes from other candidates as those lower-tier candidates get eliminated in the instant runoff process.

  • Because they're front-runners, it's unlikely to matter whom voters pick as 2nd choices behind Strimling and Brennan, since those guys are less likely to get eliminated in the instant runoff rounds. That's too bad for Nick Mavodones, Markos Miller, Jill Duson, and Jed Rathband, all of whom get marked as choice #2, to no avail, on a lot of Brennan and Strimling ballots.

  • The lowest nine candidates, together, will get about 19% of the 1st-choice votes. Even if all the second-choice votes on those ballots are marked for the front-runner (which is unlikely), it won't be enough to put him over the 50% threshold.

  • Duson looks like a lower-tier candidate with only 2.9% of 1st choice votes. Her 2nd choice numbers look good though: she gets 8.6% of everyone's 2nd-choice preferences (more than anyone save Brennan, Strimling, and Mavodones). That said, most of Duson's 2nd-choice votes seem to come from people who pick Strimling, Mavodones, Brennan, and Marshall as their 1st choice. None of those candidates are likely to be eliminated before Duson, which means that those 2nd-choice votes won't do her any good.

  • Strimling does surprisingly well with 1st-choice preferences, but he drops way behind among everyone's 2nd and 3rd choices. As runoff rounds progress, it looks likely that Brennan will get more and more follow-up choices from lower-tier candidates, and widen his initial lead over Strimling.

    [an aside: so apparently Stimling's pessimistic and divisive, "we-should've-let-my-cronies-build-a-hotel-on-the-Maine-State-Pier" campaign style is turning people off. Good. Let this be a lesson to future candidates and a ringing endorsement of ranked choice voting.]

  • Middle-tier candidates Miller, Rathband, and Marshall need lots of lower-tier candidates' second-choice votes to survive elimination in the later instant runoff rounds. Again, the sample sizes for those lower-tier candidates are too low to make any real inferences from this poll about whether that's possible.

  • Accepting that either Miller and Rathband are likely to get eliminated in a later round (which is a disappointment for anyone who cares about the issues I write about in this blog), their voters could deliver another candidate a vital block of 2nd-choice ballots if they chose to coordinate their voters in the last days of the campaign to mark someone like Brennan as their 2nd choice.

    If those 2nd choice votes put the Mike Brennan over the 50% threshold, then the new mayor would owe a lot to Markos and/or Jed. It could be a way for those guys to see their policy ideas prioritized, without actually needing to attend to the daily business of mayoring.
Six more days 'till the election. We shall see!


Anonymous said...

The precinct distribution is also totally out of whack, over-sampling the islands and other precincts while under-sampling others.

Lisa at Energy Circle said...

Christian, I love your dorky mathy way!

Lisa F.

Corey Templeton said...

Great analysis, thanks for crunching those numbers. The final results should be pretty interesting.

Mike said...

Lots of problems with this. Here are three quick points:

1. The category is other/none and it's likely that many or most of the respondents were actually undecided.

2. Your description of how MoE works is incorrect. Here's a good primer: http://abcnews.go.com/images/PollingUnit/MOEFranklin.pdf

MoE is different for the whole poll (the number usually reported, at a 50% result) than it is for individual results. The farther a result is from 50%, the lower the MoE. They also skew slightly away from the extremes - results are only right in the center of the error distribution when they're close to 50%.

There's a very legitimate case to be made for including error bars in poll results, but yours are incorrect.

3. No margin of error was calculated or expressed for the IRV extrapolation. As the release makes clear, the stated MoE applies only to reported results for 1st, 2nd ad 3rd choice.

For more discussion of methodology, see this follow-up post: http://www.downeast.com/the-tipping-point/2011/november/huge-response-down-east-mprc-poll

Wm. A. Everitt said...

With the actual election behind us, it's amazing the degree to which Mike Tipping's poll was accurate. He seems to know his way around constructing a poll.