Political Numbers & Statistics

Political Numbers and Statistics Script (Door slams) DS: Sorry, just got to keep the dog out. SH: Oh, of course. Not a problem at all. DS: If she spots me on a Zoom call, she wants to join in. SH: Laughs (Intro Music Plays) SH: Hello everyone, I am Sam Hansen SW: And I’m Sadie Witkowski. SH: And you are listening to Carry the Two, a podcast from the Institute for Mathematical and Statistical Innovation aka IMSI. SW: The podcast where Sam and I talk about the real world applications of mathematical and statistical research. SH: Welcome to episode five of our season all about the intersection between mathematics and democracy and politics. In this episode we are going to be ... SW: (excited) Sorry to interrupt, but are we going to be talking about the mathematics of dogs? How they are statistically just, the best? SH: (sighs) I am sorry Sadie but, a couple of things. One, cats are the best, and two we will not be talking about dogs, instead we are going to be talkings about something much, much, much more interesting, to me at least, political numbers and statistics SW: I’m really sorry for you that somehow numbers and statistics are more interesting to you than dogs, but go ahead I guess SH: I mean, again, if it was a kitty or even maybe a bunny maybe there would be an argument to be had but I am just not a big dog person SW: And you are a big number person, I get it. I mean you are wrong, but I get it SH: Thank you for understanding, I guess. So, just moving on, to help us dive into the deep end of the pool that is political numbers and statistics I got in touch with one of my favorite people DS: OK, my full name is Professor Sir David Spiegelter FRS OBE blah blah blah etc etc for ages. But David is just fine. SW: From that introduction alone I am starting to get why David is one of your favorite people. He’s so charming SH: Yes he is, and while it is a plus his personality is not the ONLY reason I invited him on Carry the Two DS: I'm an emeritus professor of statistics at the University of Cambridge. Previously, I was Professor for the Public Understanding of Risk at the University of Cambridge, the first and only post because it doesn't exist anymore. So that was it. That was me. And I was also Chair of the Winton Centre for Risk and Evidence Communication. SH: A role David was very happy to have filled DS: In 2007 I became a communicator and I was paid to do public engagement and that's what I've been working in and I just feel really fortunate I had a chance to change my career. SH: But most importantly for us today DS: I'm a non-executive director of the UK Statistics Authority, which is the body in the UK that supervises the national statistics system and the government statistical service and all of that. SW: The national statistics authority? SH: Right? In the UK they have done something really cool and consolidated all the number and statistics production under one roof. In fact it is more than just the production under that roof DS: It's got two wings underneath the Statistics Authority. There's the Office for National Statistics, which actually is the producer of national statistics for the country, runs the census, GDP, all that stuff. But it also contains the Office for Statistics Regulation, which is the statistics watchdog, which is actually also watching the Office for National Statistics and tells them off if they don't think they've done their job very well. So the stats authority is both supervises the production, also supervises the criticism of the production. SW: Ok, ok, I need a second to digest this. Not only does the UK have a single authority responsible for producing statistics related to the country but they also have a watchdog that oversees the criticism of those stats? SH: I promise you I was just as incredulous as you are when I first heard about this. This feels like a pipe dream as a US citizen where whichever party is not in power seems to regularly argue that the stats being put out by the government are flawed in some way. SW: A pipe dream for sure, but a beautiful one. To have political numbers and statistics being produced by a governmental agency that has a built in accountability group sounds wonderful. SH: And one that could be very useful in a democracy, as this questions I asked David shows SH: What sort of thoughts have you had as this, you know, statistical communicator to the importance of communication of things like statistics for a populace and, you know, just for an informed electorate? DS: You cannot overstate it. You know, it's just, I did, obviously, when I was working as just as a statistician, I did sort of think it was important that people understood a bit about statistics, but it's become overwhelmingly important, just absolutely crucial. SW: I can't say I disagree SH: Neither can I, and David feels so strongly about it that he has dedicated himself to this work DS: And so being able to essentially critique those claims, I think, is an essential part of being a modern citizen. And I'd like to think that what I'm trying to do is to empower people to be able to judge better the trustworthiness of the claims they hear, which are based on statistical evidence. And that's it. That's my job. That's purely what I want to do. SH: Work that he knows takes a whole village to accomplish DS: And the whole point is that you need a whole ecosystem of organizations and individuals who are really dedicated to, to use the standard phrase, to empower people, to improve. And this is an awful phrase, but we've never been able to think of a better one, the quantitative literacy in the population. SH: Now David knows that this is not a problem only of the so-called quantitative literacy of the population, that their are deep questions of identity and belonging and beliefs that are central to how people evaluate numbers and stats, but he still feels that there needs to be a certain baseline DS: However, the education is essential, is absolutely essential. It's not the only thing, but it is a necessary condition, I think. SW: Sure, but as mathematical communicators we both know that there can be quite a bit of hesitancy when it comes to engaging with this material. SH: (sighs) You can say that again, but David says that there is a recent, and I would argue still current example, that shows if done the right way statistical and numerical communication can be very welcome DS: But during COVID was a real demonstration of, I think, in a crisis, in a crisis in which there are so many competing claims based on statistical evidence, how welcome it was by the media and by the public to have somebody just to explain it to them. And I became very popular. You know, I still get people go, oh, my, usually say my mum loves, thinks you're great. But I still get people say, oh, during COVID, you were a trusted voice on the radio. SW: I am betting though it was not just because he was explaining the stats SH: Well, not exactly DS: Another little quote I use, there's no point being trustworthy if you're dull, because nobody asks you back. And so you have to be quite engaging. So you get asked back because in the end, you are subject to whether people ask you. You can write blogs as much as you like, but they don't get anywhere. What you really want is people to ask you onto their programs. And that means you are essentially a slightly passive recipient of invitations. SH: And while it was David's charm that kept him getting invited back, he made it clear to me that it was his trustworthiness that really won over the audiences DS: Audiences liked having someone who wanted to explain things without trying to beat them over the head with who was to blame and what's going to happen and whose fault it was and all this stuff and what the policy should be. SW: But how did he avoid getting dragged beyond the science and into those policy questions? SH: By clearly and explicitly saying no DS: You have to learn how to refuse to answer a question in a nice funny work pleasant genial way. Not oh no comment you know not pompous not dismissive not negative but just to say sorry that's not my job SH: In fact David thinks it is very important for us all to remember just what our jobs are DS: I got very cross with many scientists during Covid who I felt went way outside what was their professional domain. And were using their academic credentials in order to argue for particular policies. SW: What was the issue that he has with academics advocating for specific policies? SH: It is not so much that they were advocating for policies, but that they were not making it clear when they were being scientists and when they were being advocates DS: And I've become actually quite outspoken about what I feel are the biases that many academics can slip into who have strong beliefs and agendas about what they can do and whether it's about COVID or whether it's about climate change and so I actually I think that it is that you really if you're talking to the media you need to be absolutely clear which hat you're wearing are you doing this as an academic scientific expert explaining the evidence or are you a policy advocate and you can be both quite tricky but you can do both if you're clear which one you're being SH: David sees this as what is known as white hat bias SW: Ooooh, very cowboy of him. I like that phrase. But, you know what, let's let David explain it instead of me DS: I saw I see so much white hat bias. So white hat bias is when people you know for the best of reasons, feel that, you know, because they want to argue a particular point of view, start taking shortcuts on the evidence. Start not being quite open and balanced, not being properly scientific about it, because they're the goodies. They're on the side of righteousness and justice. So they want to persuade people, not inform them, they want to persuade them. And so they take liberties with the science. And you see so much of that around it. SH: And really, white hat bias is really easy to fall into DS: No, no, nobody thinks they're the baddies. Everyone thinks they're the goodies. I think I'm the goodies. But I mean, I know that my personal biases influence what I work on and how I talk about it. And so nobody can be completely objective. There's no such thing as a totally objective scientist. It'd be ridiculous. Everyone's got their own views. SW: And who ever doesn't agree with your views are the baddies SH: That does seem to be the overwhelming viewpoint these days. Which is why it is so important that the numbers and statistics that are being produced and used are trustworthy, something David tries to make sure happens in the UK through his role at the UK Statistics Authority SW: Oh yeah, I really want to hear more about this authority (ad music plays) SH: If you're enjoying the discussions we're having on this show, there's another University of Chicago podcast network show you should check out. It's called The Chicago Booth Review Podcast. What’s the best way to deliver negative feedback? How can you use AI to improve your business strategy? And why is achieving a soft landing so hard? The Chicago Booth Review Podcast addresses the big questions in business, policy, and markets with insights from the world’s leading academic researchers. It’s groundbreaking research — in a clear and straightforward way. Find the Chicago Booth Review Podcast wherever you get your podcasts. (ad music ends) SH: As David mentioned earlier there are two branches, the office of national statistics, the ONS, which produces statistics, and the one that I want to focus on the Office for Statistics Regulation, or the OSR SW: And that is the oversight group, right? SH: You are correct. And according to their website the OSR "provides independent regulation of all official statistics produced in the UK, and aims to enhance public confidence in the trustworthiness, quality, and value of statistics produced by government." SW: Am I right that this sort of oversight is uncommon? SH: You could say that DS: Because it is quite strange in a sense for the government to set up a watchdog of itself. SH: Strange or not, they have definitely found a role to play DS: Basically, if people, and it could be anybody, me or a member of the public, don't like the way that official statistics have been used by a minister or anybody, they write to the office OSR and complain. SW: And do they actually get complaints? SH: Yes, yes they do. For example when Boris Johnson was prime minister he made a claim about post-acute Covid employment statistics that turned out to out to not be true DS: So there was a lot of effort to go behind the scenes, talk to his political advisers to say, look, what he says is untrue. You know, the work initially, can we not, can he not do this? And he kept on saying it. So he got a public letter, which got a lot of coverage, criticizing him. SW: Just a second, an official governmental office wrote a letter criticizing how the prime minister was using a statistic? [laughs] This just keeps on getting less believable SH: If you find that unbelievable, wait until you hear about what happened during the UK's recent election SW: Ooook SH: Their were dubious numbers being shared by both Labor and the Conservatives about the tax costs of the other side’s policies and those claimes came to OSR's attention, and one of them kept on using them and suffered the consequences DS: And particularly the Conservatives made some claim that this was a Treasury number, which it wasn't. It was a Conservative central office number. And, you know, they were told behind the scenes that this was not appropriate. They carried on using it. And so they got a public, well, colloquial phrase, a public bollocking is what we would say in English, with Rishi Sunak being personally named as quoting these inappropriate numbers. SW: (disbelieving) They publicly told off one of the parties running for election? SH: They did SW: And do they get to do this often? DS: And that's only done really quite sparingly. If this was done all the time, it would not be appropriate, I think. But it's done quite sparingly. SH: But because it is done sparingly it has an impact DS: And it is quite feared by people, we know that, that people don't like getting letters and they don't like being told off, put on the naughty, the statistical naughty step SW: I bet they don't, but how does the OSR avoid getting pulled into the partisan political fray? SH: By being very clear about what they stand for DS: it's got three pillars trustworthiness, quality and value. So trustworthiness is the number one pillar for official statistics in this country. SW: Those sound like great pillars to have, but having pillars and maintaining their integrity are two different things SH: That is very true, but by applying them to all of the people whom they regulate, regardless of viewpoint, helps them stand tall. For example DS: Because one of the things is that if a politician makes a claim, quotes any number at all, the source of that number must be publicly available. People have to be able to check the working of that, where it came from. And they don't like that. They really don't like that. SW: Wooooow, wow, wow, wow. I can't imagine how busy the OSR would be after a session of congress or a state of the union SH: Maybe if they existed then the claims wouldn't be as outlandish? SW: I like that optimism Sam SH: It is easy to be optimistic when I have David Spiegelhalter talking with me about another part of the work that the Office for Statistics Regulation has engaged in, the creation of a code of practice for statistics SW: And is this for all statistics or just governmental ones SH: Officially it is for the governmental statistics, but they do suggest that other producers of statistics and data voluntarily commit to following the code SW: And what is in the code? SH: Well it is based around those three pillars of trustworthiness, quality, and value that David mentioned earlier, but it also has a bit focus on communication and transparency SW: Oh, that sounds like it is right up our alley SH: Indeed it does. While talking about those parts of the code David talked to me about the four parts of philosopher Onora O'Neil's concept of intelligent transparency. They are that information should be accessible, so people should be able to find it, comprehensible, people should be able to understand it, useable, it should be fit for purpose, and, the most important one to David, it should be assessable DS: Someone should be able to assess, check the working, should be able to assess the claim. If they want, most people won't no they wanted the skills or the interest to do it but somebody and there's a lot of groups around fact-checking organizations and so on who want to do it should be able to do that so it's the assessability of a statistical claim that is i think the number is absolutely vital. And that's it if it's assessable then yeah go ahead and claim SW: And does it seem like the OSR has been working? SH: Yes it does, at least according to David DS: The indicators are very positive about it, for really quite, as I said, really quite a small budget, that it has a positive...But I've got my bias. Talk about bias. I've got my conflict. I've massaged my conflict of interest, I must declare. I'm on the board. I would say that, wouldn't I? SH: I mean, true. Even though it is a small budget, it is still a budget I assume you want to maintain. DS: Well, increased preferably but. SH: Jokes about bias aside, the work OSR engages in does truly stand out DS: I'm actually very proud of the body and really a lot of people say, "Wow, this is amazing. Can we have one of those please?" SW: Of course they do, and do you know how I know? SH: Because you want one too? SW: Because I want one too! SH: Sadly I don't think there is going to be a Secretary of Statistics position in the US cabinet any time soon, and definitely no agency that has oversight over the claims made by our politicians. Though it is important to recognize the amazing, and quite trustworthy, data and statistics work being done by so many US agencies SW: For sure, without the information provided by the Census, the federal reserve, CDC, NOAA, and so many other governmental, but not partisan, agencies so much research and science wouldn't be close to possible SH: Exactly, and they are reporting out observations and not making claims SW: So, without an oversight agency like the OSR what are we supposed to do with all of those people who are making claims SH: Well David has some advice DS: Well, I could recommend an excellent book called The Art of Statistics, which is full up with guidance on how to interpret the claims in the news, available from all good booksellers. SH: Well I’ce heard that the same author who wrote the Art of Statistics also wrote a new book out called The Art of Uncertainty that is on sale in the UK and for pre-order in the US DS: But anyway, I wouldn't stoop so low as to promote my own publications. That would be unethical, I'm sure. SH: It isn't unethical for us to promote his publications though, and I promise you they are worth your time. But if audio is you preferred format SW: And why wouldn't it be? SH: Then David suggest listening to the BBC program More or Less presented by Tim Harford for examples of how to critically analyze numbers and statistics DS: I've learned so much from him and that program about how to take apart a number. SH: And one of the biggest things he has learned from Tim, may not be what you would expect DS: And and one of the things I've learned from Tim Harford is that before actually trying to look at a number and where does it come from and things like that you have to start you have start a few steps back and here's the first thing he says how does the number make me feel. I love that, how does it make me feel what do my emotion what's my emotional response to this number does it frighten me does it reassure me do I distrust it because that is really important to know about your own feelings because it actually gives you some guidance as to what the person who's telling you the number is trying to make you feel SW: How does the number make me feel? Oh, I love that so much. It is always so tempting to default to being analytical, like Spock, when exposed to numbers and forget the emotions they can surface SH: And those emotions are im…por…tant! DS: You know, if it's a pure national statistics body releasing the monthly GDP figures, well, actually, they haven't got it. They're not trying to make you feel until someone starts making some claims. But a lot of the numbers in the news are released by campaigning organizations, by charities, by bodies with a particular axe to grind. And they want you to be frightened, reassured, et cetera, et cetera. Think the way they want you to think. SW: Once you know how the number makes you feel, what’s the next step SH: Well you should of course look at what claims are being made but for the numbers themselves, make sure you can figure out what the people using the number actually mean by it DS: So often, if you look at a number, you look at what they define, you realize, well, that's not what I understand by a violent crime or something like that. It doesn't seem to be counting what I actually think it's counting. What does the number actually mean? SH: The next question should be about scale DS: Is it a big number? Because a lot of numbers can sound big, and you realize that's not very big at all. So it's the context. It's the context. It's absolutely vital. SH: Then the next step is a bit harder, but maybe the most important DS: The most difficult thing actually is what am I not being told?That's really difficult to do, really, because you're not taking part of what you're seeing. You're trying to think what I'm not seeing. That, I think, is the biggest skill of all. SH: And finally the technical bits DS: And then you actually get down to some of the more statistical stuff. Is this number biased? Are there margins of error? If you're hearing about risks, am I hearing about the absolute risks as well as the relative risks? Blah, blah, blah, all that kind of slightly more technical stuff. SW: So how does the number make you feel, what are the claims, what do the people using the number mean by it, is the number big, SH: Or really small SW: Right. What am I not being told, and the technical stats stuff. That’s kind of a lot to remember SH: It is, but once you start using these questions to start analyzing political numbers and statistics they will quickly become second nature, not that David thinks it should be on individuals to figure out this stuff on their own DS: Now, all these things, I think, can be taught. They can be codified to an extent, and they can be taught in schools. And it's one of my big beliefs that this should be an integral part of the education of everybody in the world. SW: I mean if everyone was taught these questions in school around the world we would all have much more informed electorates! SH: And, potentially a new olympic discipline! SW: (laughs) What!?!?! DS: You know I think pulling apart statistical claims could be a new olympic sport. You know they could, you know, so you give everyone a crap medical paper and you get you know you spot 10 errors of this and it's a race to do it. I think it'd be great but it might not be visually very exciting SH: Oh I would, watch it I feel like there'd be a lot of like hair pulling and like pen twirling and things like that. DS: Yeah, yeah, exactly, yeah SW: Please please LA Olympic committee, get this on the schedule! SH: Yeah, Sadie and I will buy front row tickets. Hell we will be your announcers if you need them! SW: (announcer voice) Oh, that was a close one. Jill almost missed that incorrect p value SH: (announcer voice) She may have seen it, but was she too late? Naveen has spotted that p value over a minute ago and has just jotted down the misuse of chi squared too! SW: (laughs) So those are the questions someone who is seeing or hearing political numbers and statistics should ask. Does David have any guidance for the people sharing them? SH: Funny you should ask DS: The other thing that we've emphasized as elements of trustworthy communication are I do think incredibly important. Five points. Okay, here's my five, my manifesto. So here's my manifesto SW: A manifesto? I can't wait SH: Then let's go (countdown music) SH: (Announcer Voice) Number 1 DS: First of all, that when people are making these claims, they should be genuinely trying to inform people rather than persuade them. It's not propaganda. You're trying to empower people to understand things better and make better decisions. SH: (Announcer Voice) Number 2 DS: Second, that it is, is balanced. None of this white hat stuff. You actually do give the positives and benefits, the winners and losers, the harms, etc etc. So you don't say vaccines are safe and effective. You say they're safe enough and effective enough to use in some people in some circumstances. SH: (Announcer Voice) Number 3 DS: You're up front with my uncertainty, preferably quantified. If you can put an interval around something, it's great. It doesn't reduce trust. We know that from audiences that you can be uncertain. It does not reduce trust. SH: (Announcer Voice) Number 4 DS: That you should be clear about how good the evidence is behind your claims. And this isn't just a measure of the statistical uncertainty. This is much more about, well actually you know have we got a good model for this do we really understand the science. And that so often you don't that this is really quite tentative. I never believe any confidence interval credibility but any of these statistical intervals that I spent my entire career calculating. I don't believe any of them because they're all based on assuming the model is true we know the model isn't true. So, how can you quote something that you know is based on things you know aren't true. You know, they may not be that bad, but they're not true. SH: (Announcer Voice) Number 5 DS: I do think that it's part of the responsibility when making these claims is to try to preempt the misunderstandings, to say what the number doesn't mean, to try to head off the misinformation at source. So you're not in the back foot. And this means actually having an evil mind of being able to work out what malicious people will do with your number and, and to try to head it off in the nicest possible way. (countdown music stops) DS: So that's it. That's my manifesto. Just those five points. Pretty easy, isn't it? Come on. Just sign up to that. We're all tickety-boo. SW: Where do I sign? SH: Umm, Sadie SW: Yeah? SH: I don't think he was being literal SW: I know, but I still want to sign up! SH: OK fair, me too, as I am guess will many of our listeners SW: Of course, because our listeners are awesome SH: Yes they are! They are very very awesome. Though not everyone would be on board with those five points and I think David is realistic about whether or not politicians themselves would want to sign on to his manifesto DS: I mean you can't stop them making claims and they always want to impress people. They're always grossly overconfident. They always say, we're certain about this. They always have to be. They never admit uncertainty. Worse, they never admit provisionality. They never say, oh, well, we're going to try this and see if it works. And if it doesn't, we'll change. SW: I never thought about it that way, but he is right, it is that unwillingness to admit uncertainty or change that can be the most frustrating thing about how politicians use numbers and statistics SH: Which is why groups like OSR and the media are so important DS: But that's why you need a whole ecology of high profile fact checking, strong and critical media, preferably something like the Office of Stats Regulation, and a really active group of people who critique claims and can make a fuss and can get people with followers who can get themselves on the media. Because actually, because the media love it if they can catch somebody, a politician, out saying something stupid. And so actually, you can get an audience for this. SW: I couldn't agree with that more, we LOVE a good ‘gotcha story’ SH: Same, especially because, as the old saying goes, numbers really are all around us. And that is as true in politics and the media as it is anywhere else. DS: Because a lot of people use numbers because they sound like they, right, and make all sorts of ridiculous claims all the time. Most of the numbers in newspapers are rubbish, because they're always just numbers that sound big. You know, most of the claims, they just don't stand up to any scrutiny at all. And when I, you know, when I do sort of More or Lessy kind of things, get sent a list to look at, I just can't be bothered. I just say they're all wrong. They're all nonsense. They're just to sound impressive. And most of them don't matter. People don't care. Except when people do care and it starts influencing people's feelings about something. SH: See wasn't this more interesting than talking about dogs? SW: mmmm…No comment …but yes SH: (Laughs) (outro music) SH: Don’t forget to check out our show notes in the podcast description for more David, including links to his work we discussed on this episode SW: And if you like the show, give us a review on apple podcast or spotify or wherever you listen. By rating and reviewing the show, you really help us spread the word about Carry the Two so that other listeners can discover us. SH: And for more on the math research being shared at IMSI, be sure to check us out online at our homepage: IMSI dot institute. We’re also on twitter at IMSI underscore institute, as well as instagram at IMSI dot institute! That’s IMSI, spelled I M S I. SW: And do you have a burning math question? Maybe you have an idea for a story on how mathematics and statistics connect with the world around us. Send us an email with your idea! SH: You can send your feedback, ideas, and more to Sam AT IMSI dot institute. That’s S A M at I M S I dot institute. IM: We’d also like to thank Blue Dot Sessions for the music we use in Carry the Two. SW: Lastly, Carry the Two is made possible by the Institute for Mathematical and Statistical Innovation, located on the gorgeous campus of the University of Chicago. We are supported by the National Science Foundation and the University of Chicago. SW: I just want that on a poster SH: Wow, I lost my spot entirely on the script SW: Yes, in my head I’m like are we the baddies? SH: I’m gonna have to, it’s gonna be hard for me to decide whether or not to keep that in the (laughs) SW: (laughs) SW: We introduced someone to Star Trek by watching First Contact. And they were like, “This is actually good” and I was like “That is not the take I thought I’d hear” SH: No, not it’s not. And it’s not the right take either. Uh SW: (laughs) SW: Go dogs! No (laughs) please cut that. (laughs) SH: (laughs) That’s staying in! SW: (continued laughing) No! So stupid!