Canadian Politics Today

Beyond Politics => General Discussion => Topic started by: Michael Hardner on May 30, 2018, 10:51:42 am


Title: Polling isn't getting worse
Post by: Michael Hardner on May 30, 2018, 10:51:42 am
https://fivethirtyeight.com/features/the-polls-are-all-right/

Quote
You read that right. Polls of the November 2016 presidential election were about as accurate as polls of presidential elections have been on average since 1972. And polls of gubernatorial and congressional elections in 2016 were about as accurate, on average, as polls of those races since 1998. Furthermore, polls of elections since 2016 — meaning, the 2017 gubernatorial elections and the various special elections to Congress this year and last year — have been slightly more accurate than average.

Anyone here surprised ?

I liked how vitriolic fivethirtyeight was when - after giving a 30% for Trump to win or something - people told them YOU SAID TRUMP WOULD LOSE !

:D
Title: Re: Polling isn't getting worse
Post by: cybercoma on May 31, 2018, 11:34:18 am
The polls are accurate. The polls, however, do not measure public opinion in the same way elections are held. They would have to sample every single riding to give a more accurate picture of the election outcome. That would be prohibitively expensive, so they make inferences from popular opinion and extrapolate based on the characteristics of the populations in each riding. Most of the time they don't take that second step and just report on popular opinion, sometimes adjusted for voting intention and past voting behavior in relation to demographic characteristics. Another challenge with polling is that you're dealing with human beings. We are rational creatures, which means the publication of results from polls leads people to think about and react to them, thereby changing the results almost immediately upon publication. Nevertheless, the polls that we have a far better predictors than drawing names out of a hat or flipping a coin. The Trump and Hillary polls is also a perfect example that the electoral system does not work based on the will of the voters. The polls were very accurate. In fact, Hillary Clinton did beat Donal Trump in the election, based on the number of votes each received. That's, however, not how you get elected in the United States. That's not how a premier is elected in Ontario either. You only need a plurality of votes across various sub-divisions of the population. But at the end of the day when you aggregate the vote, ignoring ridings and "electoral college" the polls are very close.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on May 31, 2018, 02:19:09 pm
They do have seat projections from Ontario this election.  Let's see how it turns out.
Title: Re: Polling isn't getting worse
Post by: cybercoma on May 31, 2018, 07:03:00 pm
They do have seat projections from Ontario this election.  Let's see how it turns out.
"Projections" being the operative word. Those are still based on popular opinion, then translated through algorithms based on demographic characteristics in the ridings. They're not riding by riding opinion polls, which would be far more accurate and FAR more expensive.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on May 31, 2018, 07:21:02 pm
"Projections" being the operative word. Those are still based on popular opinion, then translated through algorithms based on demographic characteristics in the ridings. They're not riding by riding opinion polls, which would be far more accurate and FAR more expensive.


Wow.  That sounds right.  How'd you know that ?  I am impressed...
Title: Re: Polling isn't getting worse
Post by: cybercoma on June 02, 2018, 09:23:27 am
I follow these things closely.  :P
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on June 02, 2018, 09:30:35 am
I follow these things closely.  :P

You nudged me into looking into all the 'projections' and I'm starting to think we are in a fool's paradise when it comes to projections.

Everyone seems to be posting TWO poll trackers, neither of which seem 100% clear to me.  For example, why does the CBC tracker put the chance of an NDP majority high above that of a minority ?  And the other tracker is not done by a statistician.

I am starting to suspect we will see a BIG SURPRISE on election day.

PS: This may be the rebirth of intelligent 'public' discussion I have been waiting for, for 15-20 years or so.  I never suspected that the re-emergence of intelligent public discussion would start with error-prone lone guns and build on that but it makes sense.
Title: Re: Polling isn't getting worse
Post by: Squidward von Squidderson on June 03, 2018, 09:38:14 am
Quote
For example, why does the CBC tracker put the chance of an NDP majority high above that of a minority ?

For the NDP to get a minority the Liberals would need to win seats that would have gone to the PCs.  That doesn’t look at all realistic. Hence, a minority NDP gov’t is very unlikely.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on June 03, 2018, 09:52:49 am
For the NDP to get a minority the Liberals would need to win seats that would have gone to the PCs.  That doesn’t look at all realistic. Hence, a minority NDP gov’t is very unlikely.

I don't follow this.  Why can't the NDP win seats that would have gone to the PCs ?  That is what happened in 1990.  When I read the methodology of the other tracker it seemed that they were basing results on past results which seems experimental to me.
Title: Re: Polling isn't getting worse
Post by: Squidward von Squidderson on June 03, 2018, 10:09:31 am
I don't follow this.  Why can't the NDP win seats that would have gone to the PCs ?

Yes, that would be a more likely scenario....   but then it would be a majority NDP gov’t.   

What you don’t seem to be getting is that for an NDP minority, the Libs would have to win seats unexpectedly.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on June 03, 2018, 10:23:24 am
Yes, that would be a more likely scenario....   but then it would be a majority NDP gov’t.   

What you don’t seem to be getting is that for an NDP minority, the Libs would have to win seats unexpectedly.

'fewer' but ok...

Look the Liberals are already in 3rd.  If the NDP win X seats they will have a minority.  If they win Y seats, where Y > X, then it's a majority.  Why is it is easier to win way more seats than just a few more.
Title: Re: Polling isn't getting worse
Post by: cybercoma on June 03, 2018, 11:19:55 am
You nudged me into looking into all the 'projections' and I'm starting to think we are in a fool's paradise when it comes to projections.

Everyone seems to be posting TWO poll trackers, neither of which seem 100% clear to me.  For example, why does the CBC tracker put the chance of an NDP majority high above that of a minority ?  And the other tracker is not done by a statistician.

I am starting to suspect we will see a BIG SURPRISE on election day.

PS: This may be the rebirth of intelligent 'public' discussion I have been waiting for, for 15-20 years or so.  I never suspected that the re-emergence of intelligent public discussion would start with error-prone lone guns and build on that but it makes sense.
The problem with these private firms is that their specific methods and algorithms are proprietary. If you believe in the free market then competition between them means that they all want to have that one magic formula that is the most accurate. They tend to be so.

In any case, all statistical projections are just that. The formula is accurate, it’s just that the formula includes a measure of unaccounted variance. We can never know how big that’s going to be. And as I said earlier, it’s a moving target. Polls are publish and people react to them, which then changes the results as soon as it’s announced. Each poll taken is simply a snapshot in time using the best prediction method available. They try to translate it into riding projections but it’s still fundamentally based on popular vote. Seats are won by popular vote, except on a riding by riding basis, whilst the polling typically contacts about 1200 people across different riding. Why’re pretty good at inferring ridings from that most of the time, but it’s obviously imperfect. Incumbents tend to be sticky, so they account for that. Start power also helps candidates, so they try to account for that. Voter fatigue can play a role, so they also try to account for that. Then there’s demographic composition of the riding and how peopemwith those characteristics usually vote. So there’s a lot they consider.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on June 03, 2018, 12:04:15 pm
The problem with these private firms is that their specific methods and algorithms are proprietary. If you believe in the free market then competition between them means that they all want to have that one magic formula that is the most accurate. They tend to be so.

The CBC Tracker strategy is owned by a private firm ?

Quote
In any case, all statistical projections are just that. The formula is accurate, it’s just that the formula includes a measure of unaccounted variance. We can never know how big that’s going to be. And as I said earlier, it’s a moving target. Polls are publish and people react to them, which then changes the results as soon as it’s announced. Each poll taken is simply a snapshot in time using the best prediction method available. They try to translate it into riding projections but it’s still fundamentally based on popular vote. Seats are won by popular vote, except on a riding by riding basis, whilst the polling typically contacts about 1200 people across different riding. Why’re pretty good at inferring ridings from that most of the time, but it’s obviously imperfect. Incumbents tend to be sticky, so they account for that. Start power also helps candidates, so they try to account for that. Voter fatigue can play a role, so they also try to account for that. Then there’s demographic composition of the riding and how peopemwith those characteristics usually vote. So there’s a lot they consider.

Sure, but perfect is the opposite of good.  That is to say: just because predictions can't be perfect doesn't mean we should not talk about accuracy.  Such discussions are part of fomenting a reasonable public IMO, since these tools are used to rationalize policy.
Title: Re: Polling isn't getting worse
Post by: cybercoma on June 03, 2018, 07:46:04 pm
I’m not sure who CBC uses these days, but they usually contract these things out to private firms.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on December 26, 2020, 03:29:47 pm
Old thread but I think it's a propos.

The polls AREN'T all right.

This is the listen of the year for me, and changed my mind.

https://fivethirtyeight.com/features/politics-podcast-how-to-make-polls-better/
Title: Re: Polling isn't getting worse
Post by: Queefer Sutherland on December 26, 2020, 04:48:10 pm
Old thread but I think it's a propos.

The polls AREN'T all right.

This is the listen of the year for me, and changed my mind.

https://fivethirtyeight.com/features/politics-podcast-how-to-make-polls-better/

I'm not going to tell you I told you so.  But I just did lol  ;)
Title: Re: Polling isn't getting worse
Post by: Squidward von Squidderson on December 26, 2020, 05:20:02 pm
I'm not going to tell you I told you so.  But I just did lol  ;)

Even broken conspiracy-clocks are right twice a day.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on December 26, 2020, 06:14:39 pm
I'm not going to tell you I told you so.  But I just did lol  ;)

I don't remember discussing this with you but no matter.  My views on this have changed 180 degrees.  I was partially swayed by the type of person Nate Silver maps to, ie. liberal rationalist and as such I am no better than some Iowan fooled by Trump. In my defense I have now changed.  In his defense, he hasn't come through with a final analysis yet.  Let's see if he changes his tune on what his model really provides.

I enjoy being wrong.  The world becomes very interesting when you have a bunch of things to learn and review.
Title: Re: Polling isn't getting worse
Post by: guest78 on December 29, 2020, 12:32:41 pm
The problem is that people are less willing to pollsters their voting intentions.  Thus, polls have become increasingly inaccurate.  Some polls were off by 10+ points in the recent presidential election.  Some Republican senators didn't lead a single poll up to election day ended up winning by 8+ points.  Some polls had Trump down 5-8 points in Wisconsin, Michigan, Pennsylvania etc.  He ended up losing by 0.5%.  When polls are off by that much, in one direction, they're tantamount to voter suppression.  But that kind of voter suppression the Libs don't care about.  Because it works in their favour.
Title: Re: Polling isn't getting worse
Post by: wilber on December 29, 2020, 12:51:04 pm
The problem is that people are less willing to pollsters their voting intentions.  Thus, polls have become increasingly inaccurate.  Some polls were off by 10+ points in the recent presidential election.  Some Republican senators didn't lead a single poll up to election day ended up winning by 8+ points.  Some polls had Trump down 5-8 points in Wisconsin, Michigan, Pennsylvania etc.  He ended up losing by 0.5%.  When polls are off by that much, in one direction, they're tantamount to voter suppression.  But that kind of voter suppression the Libs don't care about.  Because it works in their favour.

In these cases they suppressed the vote to the Republican's advantage. They make Democrat supporters less likely to feel the need to vote if polls show they have a big lead.
Title: Re: Polling isn't getting worse
Post by: guest78 on December 29, 2020, 12:53:01 pm
In these cases they suppressed the vote to the Republican's advantage. They make Democrat supporters less likely to feel the need to vote if polls show they have a big lead.
Not necessarily.  If you think you're candidate is going to lose, you're less likely to be motivated to vote. 
Title: Re: Polling isn't getting worse
Post by: wilber on December 29, 2020, 01:00:53 pm
Not necessarily.  If you think you're candidate is going to lose, you're less likely to be motivated to vote.

That would make it a wash.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on December 29, 2020, 01:13:33 pm
1. The problem is that people are less willing to pollsters their voting intentions.  Thus, polls have become increasingly inaccurate. 

2. Some polls were off by 10+ points in the recent presidential election.  Some Republican senators didn't lead a single poll up to election day ended up winning by 8+ points.  Some polls had Trump down 5-8 points in Wisconsin, Michigan, Pennsylvania etc.  He ended up losing by 0.5%. 

3. When polls are off by that much, in one direction, they're tantamount to voter suppression.  But that kind of voter suppression the Libs don't care about.  Because it works in their favour.

1. Yes, but inaccuracy doesn't necessarily follow.
2. Republicans over-performed from what they were polled at.
3. It doesn't seem to have worked in their favour though.  It seems odd for us to say "Republicans ignore polls" and yet say they're supressed.  But the main conclusion is we need to take a hard look at polls and figure out how to fix them.  We do need polling and some people got it right.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on February 15, 2021, 04:09:41 pm
As per my note to Graham above, the final assessment from the five thirty eight came out of their performance.

They concluded, and I concur, that Trump voters are reluctant to talk to pollsters.  This isn't the same as the shy Trump voter theory, but it does result in being unable to predict Trump's final numbers.

Some polls seem to get accuracy from an oversampling of landlines, but that doesn't seem to have occurred on purpose.

The five thirty eight works ok in other contexts, but it's not a poll.  It's an idea by one guy only.  Not as accurate as a well designed poll.
Title: Re: Polling isn't getting worse
Post by: cybercoma on February 16, 2021, 09:52:48 am
There's so much **** that goes into this that it would be exhausting to go through it all. The polls are mostly sound but the largest problem is the proprietary methods companies have for weighted distributions and outcome predictions. They poll nationally for locally-contested polls that work like FPTP, especially when it comes to the electoral college. There are different proprietary models that these pollsters use to take a sample and project the results on a national scale. There will ALWAYS be an element of error to this. They're not and never have been shooting for pin-point accuracy. Predictive models like these are always shooting for a range. The results of the polls have always been well within tolerances of that, for most reputable pollsters anyway.

The trouble with hotly contested elections is that the margins of error overlap, so it makes the prediction a lot more difficult. If someone only has a ~17% chance of winning, that's still literally Russian Roulette for the results. 17% is literally a 1:6 chance of winning, so load up that gun, spin the chamber, and pull the trigger. Chances are you survive, but that's still not a game any rational person would look at and say, "there's NO chance of dying."

The problem with polls are not the polls themselves, but rather most people's understanding of what they actually say. The news reports on sensationalist, nonsense absolutes. People are uncomfortable with uncertainty. Statistics is literally the field of math where you need to live in the area of uncertainty.
Title: Re: Polling isn't getting worse
Post by: Michael Hardner on February 16, 2021, 10:16:12 am
There's so much **** that goes into this that it would be exhausting to go through it all. The polls are mostly sound but the largest problem is the proprietary methods companies have for weighted distributions and outcome predictions. They poll nationally for locally-contested polls that work like FPTP, especially when it comes to the electoral college. There are different proprietary models that these pollsters use to take a sample and project the results on a national scale. There will ALWAYS be an element of error to this. They're not and never have been shooting for pin-point accuracy. Predictive models like these are always shooting for a range. The results of the polls have always been well within tolerances of that, for most reputable pollsters anyway.

Right, but what you are missing (I think, anyway) is why polls have started experiencing issues post-2000.  It's mobile phones and random sampling, which do not go together.  If polls are within tolerances (and I don't think they always are) there would still be a question why Trump support always skews the same way.

It's not "shy Trump voters" but it's "paranoid, "we hate the system", "refuse to talk to pollsters" Trump voters that come out of their holes like groundhogs ONLY TO VOTE FOR TRUMP that are the latest iteration here.


Quote
The trouble with hotly contested elections is that the margins of error overlap, so it makes the prediction a lot more difficult. If someone only has a ~17% chance of winning, that's still literally Russian Roulette for the results. 17% is literally a 1:6 chance of winning, so load up that gun, spin the chamber, and pull the trigger. Chances are you survive, but that's still not a game any rational person would look at and say, "there's NO chance of dying."

Ok but Silver's model is NOT a poll it's a subjectively weighted aggregate of polls.  He decides, for example, what are 'good' polls and weights them according to.... [throws a dart at dartboard] quality factors.

Quote
The problem with polls are not the polls themselves, but rather most people's understanding of what they actually say.

Also true, and I was saying this after 2016 also.  But when the 2020 results skewed the same way, something indeed seemed odd.  Of course, you can flip a coin and get heads 50 times in a row but this seemed different.  The conclusion Five Thirty Eight reached is that there likely is a phantom Trump voter.

Quote
The news reports on sensationalist, nonsense absolutes. People are uncomfortable with uncertainty. Statistics is literally the field of math where you need to live in the area of uncertainty.

Sure but good random sampling is the lifeblood of quality polls, and The Five Thirty Eight have their own oddities that have to be absorbed in such a conversation.

That said, I'm still happy with having them around.   The Georgia polls showed up accurately, meaning that absence of Trump is a good thing ... in yet another new and wonderful way.
Title: Re: Polling isn't getting worse
Post by: cybercoma on February 17, 2021, 10:53:05 am
OR how about maybe, just maybe, the possibility of Republican fraud?

For all the politicians in Georgia who push back against him. How many went along with his coup attempt?

Food for thought.