Mathematics & Polling

Mathematics & Polling Script DD: You don't have a democracy without public opinion, folks. So when you, when you get that call, do you take some time, do the survey. This is how people know what Americans think, you know, and how the public has a voice in a democracy. (intro music starts) SH: Hello everyone, I am Sam Hansen SW: And I’m Sadie Witkowski. SH: And you are listening to Carry the Two, a podcast from the Institute for Mathematical and Statistical Innovation aka IMSI. SW: The podcast where Sam and I talk about the real world applications of mathematical and statistical research. (intro music ends) SH: Welcome to sixth, and final, episode of our season all about the intersection between mathematics and democracy and politics.And as the opening clip foreshadowed we will be wrapping up this season with a deep dive into the world of surveys and polls SW: Excellent! And very timely as my news feed has been inundated with very little else as we creep ever closer to the election SH: Mine too (sighs), which is why I reached out to NORC at the University of Chicago, one of the leading opinion research institutes in the world, for a guest and they provided DD: I am David Dutwin, Senior Vice President at NORC and Chief Scientist of Amerispeak, our major probability panel that we use for, in part, for election polling and lots of other research. SW: And what did you talk to David about? SH: Well first I wanted to make something clear, and that is just what surveys and polls are. A topic that turns out to be more challenging than I first thought, mostly because people, myself included, often use the two words interchangeably DD: There's two to three key distinctions. Number one surveys , most importantly, survey a known population SH: Which is not something that is even possible in election polling DD: In election polling, the biggest challenge in the world is we're surveying a population that doesn't yet exist, right? It will only exist once all the voting is done, meaning we interview people, we say, are you going to vote? And they say yes, and it turns out that between 80 or 90 percent are telling the truth in both directions. People who say no, a lot of them end up, you know, 10, 10 ish percent of them shop. SW: Ooooo, that is a very big difference. We can’t count the “I voted” stickers until after they vote! And David mentioned a second difference too? DD: Secondarily surveys have a big range of fit for purpose. SH: Some are going to be used to answer important social questions DD: When you want to know crime statistics, health statistics, official statistics that the government produces, you want to be able to conduct surveys at the highest quality possible. You may even go door to door. You may provide incentives, whatnot. SH: While the answers sought by other surveys may be less pressing DD: How much do you like Kim Kardashian today? Like, well, if you're 5% off, you know, the world will go on. It's okay. Right. SW: You better watch out Sam, there are some people for whom there are not questions more important than that SH: I know, and I want all the Kardashian fans listening to know that all I am saying is that having high quality and trustworthy public health information has more of a bearing on governmental policy than Kim's current Q rating SW: I don't think you’re really helping yourself there, so let me help you instead. If surveys can have this wide range of purposes and how accurate they need to be, how can surveys be made to fit those needs? SH: Well that comes down to what’s known as survey design, and the first, and potentially most important part of survey design is how does the survey get its participants DD: There's a small list of ways that one can field a survey. The first distinction is whether you use probability sampling or non-probability sampling. SH: And the purpose of a survey definitely plays a role in which one David would choose DD: The number one synonym for good science is randomization. Same thing with surveys. If we're able to mix every address or every telephone number or whatever in the world in a big bowl and then take out a spoonful of that bowl, right, when you take that spoon out, nobody questions that that spoon is a super accurate representation of the rest of the bowl. SW: Man, this is right up my academic alley from my old psychology days! So going with that analogy the goal when using probability sampling would be to mix up the populations you want to survey and then by asking the survey questions from the people who show up in the spoonful. That should get you a good sense of what the whole population thinks, right? DD: Think about this, that there's about 300 million adults in the United States and we'll do a survey of a thousand people and come up with the right answer time after time after time. SH: While probability sampling is very effective it does come with a rather major problem DD: The biggest fish in the tank is non-response error. We may randomly sample and ask 10,000 people to participate in our survey. The sad truth in this day and age is that you'll be lucky to get a thousand out of that ten thousand SW: Considering how many phone surveys I ignore, I see why getting a 10% response rate is lucky. SH: Yeah, it’s hard out there to conduct surveys. And if you help administer surveys like David does, all those non-responses bring up a very important question DD: The question is is the nine thousand you didn't interview somehow different than the thousand you did. If the answer is yes, it doesn't matter if you interviewed, you know, 100 out of 10,000, as long as those two groups, the people who responded and the people who didn't, are the same across every conceivable metric. SW:And this where the idea of a representative sample comes in, right? SH: Exactly, you need the sample you survey to be representative of the population you want to understand. But this isn't exactly easy to determine DD: It's hard to know, you know, exactly whether the people who responded are different, whether, you know, maybe people have settled to the bottom. In this election and in past elections, one of the big concerns has been that people of lower educational status, people who are not trustworthy of science or institutions like the mass media, they have percolated to the bottom. They certainly seem to not want to respond to surveys. SW: David said that probability sampling yields highly accurate results, but non-response error already makes it seems like surveys should be faulty and I know there are other errors to come SH: You’re right there are other errors to come, but it is important to remember that error is not the same thing as bias DD: 9 out of 10 people chose to not respond to your survey. Well, that's an error, but it doesn't mean your survey is biased. SW: Right! That totally makes sense, error doesn't mean a survey is wrong just that it has the chance to be so SH: Exactly. DD: You know just because you're not getting a certain cohort of the American public to do your survey as much as other cohorts doesn't necessarily mean your survey is biased. It depends on whether that cohort that's being underrepresented answers questions differently than the cohorts who are participating. SH: And as long as we are talking about error I want to quickly mention another that has to do with sampling, specifically how the people running a survey reach out to the people they want to respond to it DD: There's coverage error where when you sample a population with a technology, that doesn't actually cover the whole population. SW: Going back to my earlier admission… This is why we shouldn’t try to survey only using landline phone numbers in 2024? SH: Not unless your population tends toward my father's age, he still has a wired rotary phone on the kitchen wall SW: Ooooo, vintage! But let’s talk about surveys done using non-probability sampling, they also suffer from these errors right? SH: They do, but we should probably let David explain what non-probability sampling is before we start talking about errors DD: Non-probability sampling is different. It's getting people conveniently. The goal is to be able to do surveys as inexpensively as possible. The problem is you've let go of that random selection. There are things on the backend that you try and do scientifically to try and make sure it's still representative. And sometimes that works and sometimes it doesn't SW: Ah, so those pop ups or generic emails asking me to answer a few questions might be for a non-probability based survey? SH: They very well might be, same as a person with a clipboard asking questions outside a grocery store. These would all be examples of what’s called convenience sampling, and given how rarely I click on those emailed survey links or talk to those people on the sidewalk I can promise they have non-response error SW: Just like a convenience store meal might be easy, but not great for you, a sample of convenience might not work out. I mean, it is almost like giving a person a clipboard makes them invisible SH: Or at the very least clipboards generate a version of the Someone Else's Problem field from Hitchhiker's Guide to the Galaxy SW: At least I now know what prop I need next time I want to be left alone SH: Yeah, one way you may have noticed non-probability surveys trying to combat this is by suggesting we send a link to the survey on to your network. Which is a method called snowball sampling SW: Hahaha, just rolling those survey snowballs together until you have a Frosty the Snowman of survey results? SH: Pretty much. And while that might not be as jolly as the holiday character, I would definitely be more likely to fill out a survey sent by a friend, I am sure there are still plenty of non-response issues though. SW: What about coverage error? SH: Non-probability sampling runs into the same issues. Say you want to survey university students on their feelings about mathematics but the only email lists you have access to are for faculty SW: This happened to you didn't it…? SH: I don't want to talk about it. SW: [laughs] Alright, but with all of these issues around sampling alone how do any surveys actually get done. SH: Well that is where groups like NORC come in DD: AmeriSpeak is our probability panel. SH: Or in other words a group of people that NORC has identified as being willing to take part in their surveys. And that identification is hard work DD: Amerispeak is different than any other probability panel in that about half of our panelists we actually do get door to door. There's nothing like going door to door to get people to participate in research. If you mail a letter, if you make a phone call, yeah, about four or 5% will participate. If you show up at somebody's door about 35 to 45% of people will participate. SW: Oh wow, that is a lot of effort and I am guessing quite expensive and slow SH: For sure, but since they are recruiting for a panel and not a single poll the cost can be spread over many surveys SW: And does the recruiting work pay off in surveys? SH: According to David, it definitely does DD: AmeriSpeak is still able to get true response rates in the 10 to 15% range, recruitment rates in the 35% range. We're just the highest quality probability panel in the United States because we're the only one who does that effort. SH: Oh, and if you are looking for do a survey David has important information for you DD: Anyone can use the panel. They want to do some research, they want to do a poll or a survey, we can use our panelists to field that survey. SW: Ok, so we have spent a lot of time already on just getting people to take a survey. I know you have to have notes up your sleeve on survey design SH: Well, there is a vast and deep literature out there about what types of questions to use, from open ended to multiple choice to scales, and how to write those questions, but to fit our purposes as a math and stats podcast I am going to need to condense it a bit SW: How much? SH: What? SW: How much are you going to condense this vast and deep literature? SH: Ehhh, to a single error DD: Measurement error is when that question doesn't really accurately represent the concept SW: And how can this error, that you are using to stand in for an entire discipline, be managed? SH: One way is finding an expert on the discipline of developing survey questions and that way would be your best bet. Beyond that making sure you are using the language and meanings of the population you are trying to measure and defining any technical terms and jargon is a great place to start SW: And don’t forget prototyping! But basically, it’s following good communication practices? SH: Basically, yeah SW: Ok, so we have our sample and we have our questions. What happens after the sample answers those questions? SH: Well in a lot of cases, the people running the survey will look at how close to their target population the demographics of the people who responded are and if the people who responded do not match the goal population they will often weight the results so they more closely line up DD: And you do that commonly through what's called raking, which is iterative proportional fitting. SW: (starts to talk) Ok, I know this raking isn’t what overzealous lawn keepers do in the fall SH: (interrupts) I know, I know. That needs an explanation DD: You correct for gender, you apply that weight, then you look at your distribution on, say, age. You fix age, you look at that combined age-gender weight, you apply it, you fix for race-ethnicity. And then you're at the bottom and you look back up at gender and you realize, well, gender is now off again. So you rake it back. And so you're raking and raking. You're iterating. Right. And maybe after 10, 15 iterations, you have a situation where you've got convergence, meaning every parameter you're trying to match a benchmark from the census matches up within a tolerable range and you're good. SW: And these corrections are essentially taking demographics that are underrepresented in the responses and giving them more value? More oopmf SH: There is a lot more going on technically, but that is my basic understanding too SW: But futzing with data like this is always a risky prospect. What impact does this have on the survey's error? SH: Quite a bit DD: The more you weight, the more you're introducing variance into your estimates and variance means larger margins of error. And at a certain point, you've blown up your variance so big to try and chase down bias that now you have other problems. So it's always a balancing act between variance and bias. You know, what kind of variables do you enter into that weighting? Do you interact those variables? So, you know, gender by education, instead of having them as sort of main effects. There's also, I mean, there's a huge literature on weighting. SW: Of course there is. But let’s say weighting went perfectly. Then they analyze and report the findings? SH: Or otherwise use the results, yes. That is if they are a survey. SW: As opposed to what? SH: To election polls of course SW: That's right, I forgot we were just talking about surveys to get to polls SH: Well not just to get to polls, as surveys do play an important role in a democracy. A point David made more than once DD: We are making better decisions because of those data and surveys are a key part of those data that we use to make well-informed decisions. SW: And anything that can help a democracy make more well-informed decisions get a thumbs up for me SH: And from me. You hear that surveys Carry the Two is giving you Two thumbs up! (SW laughs) That said I can't help but feel a bit less positive about polls, if only because of the fatigue I get from the sheer number of them SW: Ugh, I get that, there are way too many to stay on top of SH: Something which is a recent phenomenon it turns out DD: There are a lot of polls out there. It's funny to look back to the 80s and 90s and think about like, wow, you know, back then, literally there were like three or four entities doing one or two or three polls per cycle. I mean, you literally had like two dozen polls throughout the whole election cycle. Right now, we have two dozen polls happening in basically like three days at this point. SW: Two dozen or so polls during an entire election? That is almost unbelievable today SH: Right? But if we are going to transition to polls, we should probably start at the same place we did for surveys SW: The fit for purpose? SH: The fit for purpose! DD: A very high quality survey, you might want to be in the field one or two months. You can't do that in an election, right? But yet the fit for purpose is to be, you know, unbelievably accurate. So you have this disjunction between the fact that the goal is to have an official statistic, essentially, but you're spending money more like the Kim Kardashian example a little bit, maybe not as much, but closer there than official. SW: And I am sure that is just exacerbated by the tight money situation in the media, since so many of these polls are sponsored by news organization SH: Ugh yeag, but thankfully we are not an econ podcast so I don't have to dive that deep into that. Instead let's talk about what polls have to do differently than surveys from a math and stats perspective SW: So the ways they work around not knowing what population they should be sampling from? SH: Exactly, a problem pollster mitigate with something called likely voter models DD: Likely voter models can be incredibly simplistic, like, do you plan to vote? Yes. Okay, you're in. That's your likely vote. Or they can be very complex, asking a range of questions. How enthusiastic are you? Did you vote last election cycle? Did you vote in the midterm election? How much attention do you pay to the media? And throwing these things all in, back in the old days, you know, it'd be usually some sort of regression equation. More currently, maybe people use tree-based models to spit out a prediction. SW: And what do they do with the prediction? SH: It depends on the model. DD: The prediction says they're 60% likely to vote. Do I put them in or not? Well some models will have that cut point or some models will use that prediction as a weight. So, that person gets, is 60% of a person, right, essentially. That's kind of the, I think the, more elegant way to do it SW: And by using those weights the pollsters get a way of estimating their final population. That seems reasonable. Though, like before, I am sure it increases the error SH: The models won't be perfect so of course they will introduce some more possibility of the poll being inaccurate. But as David says DD: You know, it's a constant art and science election polling. We take every election, we pick it apart, we try to put it back together in a way that's better for the next cycle. SW: Sure, sure, but can we talk a bit more about polling errors? SH: Of course, do you have something specific in mind? SW: In fact I do, did you talk to David at all about the Margins of Error that I always see next to polls when they are reported? SH: Ah yes that, that did come up DD: So, importantly, that margin of error that's based on 3% is only measuring sampling error. It does not account for the possible errors that might have been introduced from non-response, or coverage, or measurement, or other things. SW: And to clarify - all of the errors we discussed earlier aren't even included in that error? SH: Sadly no, those are all part of what is known as design effect. Once you add the design effect error is added to the sampling error the potential for inaccuracy goes up a bit DD: In reality, you know, the true error range is probably something more like 5 or 6%, but that's never talked about. SW: A bit, potentially doubling the margin of error feels like more than a bit. So why isn't it talked about? DD: Why? Because nobody wants to explain that to lay people like, "Oh, there's this thing called design effect, and what the hell is that? SW: Ok, fair SH: This also explains why it may seem that pollsters are more laid back when their polls don’t match election results DD: When the reality of an election is outside of that 3%, they're like, oh my God, the polls lied to us, essentially. When people technically in the field, statisticians, methodologists, et cetera, fully understand that that 3% is not an accurate portrayal of the error range of that given survey. SH: Though it could also be related to that not really happening that often DD: When you look at the level of error that has occurred, it's typically less than 3%. It's quite good. SW: Being within the 3% margin is quite impressive given everything I have heard about polls so far, but there are a couple of common complaints that I think now is the perfect time to address SH: Shoot SW: Well let’s talk about the idea that polls are tilted toward one party or another. That is a complaint I hear all the time SH: And it’s a complaint that doesn't really hold up DD: If you look at the bias that has occurred in every presidential and midterm election since 1950, We're close to 50-50 in terms of being biased to Democrat versus being biased to Republican. SW: And, how about the idea that as the US population has become harder to sample and non-response rates have gone up, thus making polls become more biased? SH: Those things have definitely introduced more potential for error, but pollsters thankfully are really good at their jobs and those potential errors haven't turned into biased results DD: The good news is probability sampling is very robust. It can deal with a lot of non-response error and still not have a great deal of bias. And you see that in these bias graphs over time for elections, that they really are flat lines generally. SH: There are a couple of problems though that David did bring up, one of which ties directly into something we discussed earlier this season DD: You do a national poll, it in some ways doesn't mean anything because of the Electoral College. SW: The white whale of the electoral college again? What can't it ruin? That said, there are state level polls right? SH: Sure, but state polls are usually done with even less money and support than the national polling DD: We all know at the end of the day, there are eight, 10 states that really are going to decide this election. And the polling in those states may or may not, will be of varying quality right now. SH: And then there can also be momentum effects where either a candidate builds up a head of steam or suffers a sudden fall in support, something polls may not be able to capture DD: These are things that poles have a very hard time picking up because it takes three to four days of field work to do it, even a halfway decent poll. And in those three or four days, things might be moving and shaking underneath. And so it's hard for those poles to get a real accurate portrayal. SW: In other words, watch out for a late October Surprise! SH: Definitely watch out for one but more importantly do not be surprised if a surprise doesn't register in the polls SW: That is an important distinction SH: Thank you SW: You're welcome? (pause) So, is that it. Have we covered everything there is to know about polls? SH: Not even close, but before I share even more with you let's take a quick break so our audience can learn about a different cool podcast from the University of Chicago Podcast network they can listen to (Ad music) SW: Sounds good SH: If you're enjoying the discussions we're having on this show, there's another University of Chicago podcast network show you should check out. It's called Big Brains. Big Brains brings you the stories behind the pivotal scientific breakthroughs and research that are reshaping our world. Change how you see the world and keep up with the latest academic thinking with Big brains. Part of the University of Chicago Podcast Network. (Ad music stops) SW: Alright, what's next? SH: Well it just so happens that there is a type of poll that we haven't even talked about yet SW: What? What type of poll is that? SH: Exit polls SW: Of course, how could I forget SH: I almost did too, until David brought up a partnership that NORC has with the Associated Press called VoteCast DD: Its calling card is the election exit poll. SW: And exit polls are exactly what they sound like right? A survey of people as they exit polling stations? SH: I do believe that is how they used to be conducted, but things have changed over the years with an increase of voting by mail and early voting SW: How does VoteCast handle those changes? SH: As with gather a probability sample, a lot of work DD: In that election day and certainly the last three days before election, we are doing about 100,000 nationwide interviews to provide an exit poll. SW: And these exit polls are then used for what? To help call the states? SH: VoteCast will certainly be part of how AP, as well as Fox News and some other news organizations report the election results. But David wanted to make it clear that exit polls have a much larger purpose in democracies DD: And exit polls are important because in a democracy, we don't think about this much maybe in the United States, but in a lot of other countries, that exit poll is sort of that corresponding truth metric, right? To make sure that the election itself was not corrupt. If you saw incredibly different answers between the election outcome and the exit poll outcome, you would worry that your election was corrupt in some way. SW: David is right, that is not something I had really thought about SH: Me either, and while all of the evidence makes it very clear that US elections are incredibly trustworthy, knowing that NORC and AP VoteCast are out there doing this work does provide me a sense of comfort SW: For sure SH: So there you have it. That is how polls and surveys are designed, the errors they can have, and some of the math and stats behind it all SW: Then how come I still feel like there is something missing SH: Because something is missing SW: Aaaannnddd SH: (silence) SW: Aaaannnddd SH: (silence) SW: Oh come on, what is it that is missing? SH: (laughs) Only the most important part, at least as far as most people are concerned: actually hearing about the results SW: Wow, my brain got so far in the weeds. How did I miss that we hadn't talked about how polls are reported on? SH: That’s because all that other stuff was so interesting. But if we move to the reporting topic, I guess really what I want to know from you is where do you go to get your polls news fix? SW: Often I see it reported in your classic TV coverage like CNBC or ABC or CNN. All those alphabet soup news shows., though if you asked me for poll specific reporting I tend to think of 538 SH: Then I think you are going to like our next guest NR: My name is Nathaniel Rakich. I am the Senior Editor and Senior Elections Analyst at FiveThirtyEight. SW: Oh wow, you went straight to the horse's mouth I see SH: It felt like the right thing to do SW: So what did you ask Nathaniel about first SH: My first question was about what they do when a new poll comes their way NR: When a new poll comes across the transom, we will enter it into our big old polling database. SH: A task for which they need some crucial information NR: We need certain information in order to input it, so like the dates and the sample size and the population that was polled. SH: That is for existing pollsters, new pollsters get deeper vetting NR: We will contact the pollster and kind of schedule an interview and we'll ask them some questions about their poll, some questions about the methodology that they use, some kind of questions about the science of polling in general. We may ask to see their data and we kind of evaluate from there. You can tell pretty quickly whether somebody is actually conducting a poll and whether they know what they're doing or if they, you know, they are just kind of blustering. SW: Oooh, quality check! It is really helpful that they are doing that evaluation step for us SH: I know, because there is no way that I am qualified to do it myself SW: Me neither. So once a poll is in the database what happens? SH: Once the poll is in the database, then it starts to get incorporated into the work 538 produces NR: We have polling averages of the race between Harris and Trump. We have a polling average of President Biden's approval rating, and the poll will get entered into that. SW: What about the election forecast? SH: Instead of the single polls playing a role there, it is the polling average as a whole NR: The polling average then gets fed into the forecast along with a bunch of other data on, say, fundraising or incumbency status and sometimes like race ratings from like the likes of Cook Political Report, which do kind of qualitative election predictions. SH: And then in the end, the 538 forecast combines all of that into a prediction. NR: It makes a prediction sort of about the election, but really it makes like 10,000 predictions because you're simulating the election 10,000 times. But the real value in the model isn't kind of looking at what the most likely scenario is, you know, X electoral votes to Y electoral votes. It's looking at the probabilities and what percentage of the time Donald Trump wins versus what percentage of the time Kamala Harris wins. And that's what you see on our forecast page. And if you ever see a 538 analyst talking about the odds of somebody winning the election, that's where they are drawing that data from. SH: All of that said, Nathaniel wants to make one thing very clear NR: We're not trying to quote unquote call the election. SW: If they aren't trying to call the election, how do they view their work? SH: Really it comes down to what most journalists are trying to do, inform the public NR: I don't want people to think that what we're trying to do is call elections or make predictions kind of for fun to increase our street cred as seers and oracles of the election. We are actually just trying to give people the right sense of how certain or uncertain they should be about the election. SH: Something Nathaniel often uses metaphors to accomplish NR: I try to describe it to people instead of using the precise numbers as these kind of metaphors. Oh, this election is basically a coin flip, you know, or in another event, you know, if like somebody had a 70 percent chance of winning versus a 30 percent chance of winning, I would say, well, this candidate is favored, but they could still lose and think about it this way. a really good hitter in baseball is a 300 hitter. And that the odds that the underdog could win this election are the odds of a good baseball player, Juan Soto, getting a hit. SW: I hadn't heard the baseball hitter one before, that one’s pretty great SH: It is almost like Nathaniel gets paid to do this for a living! SW: Sarcasm noted! SH: (laughs) But seriously, communicating polls and other numbers to the public in an understandable way is something Nathaniel spends a lot of his time thinking about NR: I'm spending a lot of time thinking about how the public is going to, a public that is not all that well-versed in math and probabilities and statistics, and how we can communicate this to them without misleading them and kind of meeting them where they are. And so that is, yeah, that's something that we think a lot about at FiveThirtyEight and all of our coverage is really trying to bring this admittedly wonky, but I think necessary part of analyzing elections and politics to a broader audience and helping to improve kind of the numeracy of the public so that they understand how their fellow Americans think about the world and so that they have the right idea about who is going to win an election going into it, if we even can know that, which in this case, I don't think we can, but couching the uncertainty in previous years, I think was very important. SW: I wish a lot more journalists thought this deeply about how they communicate quantitative information and what uncertainty really means SH: Me too, even if that would put us out fo a job, which is why I also asked Nathaniel for some tips about how we should approach reading and analyzing news stories about polls SW: Oooh, exciting! SH: The first tip has to do with stories that are based on single polls NR: We have basically a rule at FiveThirtyEight, which is never write a story off of a single poll. And that's because a single poll is subject to normal amounts of error, right? SH: That said, there can be some cases where writing about a single poll makes sense NR: I guess sometimes there'll be a poll that gets a lot of discussion, you know, in the media or something like that. And sometimes we'll need to write a debunking and we'll say, oh, this poll's an outlier, just throw it in the average, but we'll never kind of write a story that characterizes the race or characterizes how Americans feel about an issue based on one poll. SH: For example NR: So when we write a story about education, we make sure that we are looking at the polls holistically, and maybe they agree and maybe they disagree. And when they disagree, we'll write about that, will say, yeah, this one poll says this thing. This other poll says another thing. And maybe this is due to different samples. Maybe it's due to different question wording. It can be all due to all sorts of things. SH: Which is why NR: A lot of the time with a 538 story, we don't end up reaching a conclusion because it's just kind of an exploration of the data. And sometimes at the end, you just have to throw your hands up and say, we don't really know who's going to win this election or how Americans really feel about this issue. But wasn't it kind of an interesting journey through the polls to to find that out. SW: (laughs) That sounds like a research graduate student experience. But basically, it comes down to be careful when a new story is using a single poll to report on US opinions on a topic. That makes sense, but then what should I look for when I want a current snapshot of something like an election? SH: Both Nathaniel and David suggested that polling averages are better sources for election numbers. While we have already mentioned 538's polling average, and election forecast, there are also ones available from Real Clear Polling, the New York Times, and many other places SW: Look for averages instead of single polls, got it. What else? SH: Be careful when the polling numbers are really close NR: Trump leads Harris by one point, and you usually can't say that. We can't say that that's a lead because polls have a margin of error. SH: In other words NR: Basically anytime a poll is within one point, that's like a virtual tie. And I think should be couched as such. SW: Margins of error mean close polls are the same as coin flips, keep them coming! SH: Next up, pay attention to methodologies and question wording before comparing two polls NR: let's say that there was a poll a year ago asking do you support or oppose obamacare right and it had only 40 support and then there's a poll this year that says do you support the affordable care act which guaranteed health care you know make sure that all americans had health care and you let people stay on their parents' health plans until they were 26 and cover people for existing conditions, etc., etc., etc. And that has 60 percent support. SH: Which would make it very tempting to say ACA support went up by 20% over the course of the year, but NR: That's not necessarily true because those two polls use different methodologies, use different question wording. And when you say Obamacare, people think about Barack Obama and they think about all our other complicated feelings about Barack Obama. Whereas if you use the Affordable Care Act, they might be more favorable toward it. And certainly if you describe certain provisions of the bill that are popular, that is going to increase support. And so even if they use the exact same question wording, if they were from two different pollsters, because some pollsters use phones and some use online surveys, and those are going to reach different kind of types of people, you can't make that comparison. I recommend only using polls from the same pollster kind of as an apples to apples comparison when you're kind of drawing trend lines like that. SW: Only compare polls with the same methodology and questions, which in most cases means only comparing polls from the same pollster. Are there any more? SH: One more, this time having to do with when a campaign itself releases some numbers NR: It's not unusual for campaigns to conduct many polls, just to get a sense of where they are to develop their campaign strategy. Maybe you'll see as the public, you'll see one of those polls over the course of the campaign. Chances are it's going to be the best one, right? They're going to release the best one, the one that puts them in the best light. SW: Campaigns are trying to make themselves look good? Hmmp, noted! And Sam? SH: Yeah? SW: Please make sure to thank Nathaniel for me because these are really going to make it easier to parse polling news, even outside a presidential election! SH: Happy to, hopefully it will help him get through the next few days before the election is done and the results posted NR: I definitely, I definitely dream about politics and work. During this time of the year, when we are two, three weeks away from Election Day, it is pretty all-consuming. SW: Dreams about politics and reporting on polls? Somebody deserves a long vacation very soon SH: 100%, and before our show goes on its own mini-vacation would it be ok with you to give both of our guests a final word? SW: Of course DD: People in survey research work every day to make sure that surveys are as accurate and as state-of-the-art as possible, and that's a never-ending endeavor. NR: I'm going to be a little bit of a broken record but i think the most important thing that i wish people understood about polls is that they're not perfect, but they are the best tool we have for predicting elections and certainly for measuring public opinion and learning how the rest of the country feels. And talking to three voters in a diner is not a substitute for taking a scientifically rigorous survey of 500 people in a swing state. (outro music starts) SH: Don’t forget to check out our show notes in the podcast description for more David and Nathaniel, including links to their work we discussed on this episode SW: And if you like the show, give us a review on apple podcast or spotify or wherever you listen. By rating and reviewing the show, you really help us spread the word about Carry the Two so that other listeners can discover us. SH: And for more on the math research being shared at IMSI, be sure to check us out online at our homepage: IMSI dot institute. We’re also on twitter at IMSI underscore institute, as well as instagram at IMSI dot institute! That’s IMSI, spelled I M S I. SW: And do you have a burning math question? Maybe you have an idea for a story on how mathematics and statistics connect with the world around us. Send us an email with your idea! SH: You can send your feedback, ideas, and more to Sam AT IMSI dot institute. That’s S A M at I M S I dot institute. SW: We’d also like to thank Blue Dot Sessions for the music we use in Carry the Two. SH: Lastly, Carry the Two is made possible by the Institute for Mathematical and Statistical Innovation, located on the gorgeous campus of the University of Chicago. We are supported by the National Science Foundation and the University of Chicago. SH: Is it having high quality and trustworthic… Trustworthic? Uh.. (laughs) SW: The perfect…(gibberish) SH: (laughs) That really sounded like I was reading from a script. Hmpph SW: (Scottish accent) Perfectly. I've been watching Shrek. SH: (laughs) I'm the one who's really close to Scotland right now. SW: (laughs) I know. SW: All right, Ira Glass. SW: Communicate quali... SH: (laughs) SW:It's not qualitative. is definitively not qualitative.