Emerging Technologies Episode 5: Computation Imaging

KW - Yeah, so many people point to the Apollo program as being a place where this notion of having a simulator, where that idea was maybe first put into practice or put into practice in a way that was highly impactful. And so, when NASA launched the spacecraft up into space, The story goes that they would also launch a simulator on the ground in Houston and have that simulator follow along with the real space mission. (Intro Music Starts) SH: Hello everyone, I am Sam Hansen SW: And I’m Sadie Witkowski. SH: And you are listening to Carry the Two, a podcast from the Institute for Mathematical and Statistical Innovation aka IMSI. SW: This is the podcast where Sam and I talk about the real world applications of mathematical and statistical research. (Intro music ends) SW: (Very excited) So from that clip can I assume we are talking spaceflight? Cause as much as I love the story of Apollo I don’t think we can call it emerging technology SH: No, we definitely can not. And while the emerging technology we are talking today is definitely used in modern spaceflight, the important part of that clip wasn’t the putting a craft into space SW: Then what was it? SH: Here, let’s hear a bit more from our guest KW: Now, on the Apollo program, it wasn't a digital simulator, it was actually a physical simulator. SW: Oooooh, it was the simulator part SH: Exactly, and while these advanced digital simulation style models have been around for decades, since 2010, thanks again to NASA, they have been given a fancy new name KW: A digital twin is a virtual model of a physical system. And what makes this virtual model, this computer model special is that it is not static, but this digital twin is living. SW: Living? How can a DIGITAL thing be living? KW: We can collect data or make observations of the system in the physical world. feed those observations, feed that data into the virtual model, update the model so that it's dynamically changing and representing changing conditions in the physical world, and then use that virtual representation to drive changes in the other direction. And then you can imagine this bi-directional interaction in kind of a continual loop where we're constantly back and forth between the virtual and the physical. SH: This is Karen Willcox, one of the first ever speakers at IMSI, and KW: I'm a professor of aerospace engineering and engineering mechanics at the University of Texas at Austin. Here at UT Austin, I'm also the director of the Odin Institute for Computational Engineering and Sciences, and I am also an external faculty member at the Santa Fe Institute. SH: While I was talking digital twins with Karen I asked for an… SW: (Interrupting) Um Sam, before we get into that can we just talk about how cool a name digital twin is? I mean it is good from a communications perspective in that I totally know what it means when I hear it but also from a marketing one because it sounds exciting and new in a way that say computational materials science doesn’t SH: Materials science catching strays is not quite what I expected, but I can only agree with you Sadie. Of all the technologies we have covered this season Digital Twins is the most exciting sounding for sure. Karen too thinks that the creation of the name really helped what had to that point been a very diffuse set of research doing bidirectional modeling across a whole wide range of disciplines KW: And with the use of the term, I think it really gives some identity to this idea that shows up in different places, but really makes the digital twin a thing for us to think about. It really takes it to a new level. SH: A very exciting level at that KW: A digital twin is like a beacon to which many different disciplines can be attracted and collaborate SH: Which is one of the many reasons that Digital Twins just so happens to be the focus of the current long term research program at IMSI SW: Lucky! You are going to have a great time attending those workshops and chatting with the participants SH: I know it, and registration is still open for at least three of the workshops at our website imsi.institute. And it is always free to attend online SW: I am definitely going to check those out! SH: As should everyone else. But now back to what I was talking about before you interrupted me, while I was talking digital twins with Karen I asked for an example of one that most people could easily understand and she brought up modern flight simulators which have come a long way from the preprogrammed versions KW: But then where the virtual and the mathematical modeling can come in is that now you can start to represent the dynamics of the aircraft, the dynamics of weather, clearly without having the physical weather inside the building or the physical aircraft that you're sitting in. SW: That is so cool, and must lead to such better training SH: I mean I am not a pilot but I can only assume that training under real world conditions, especially for things like weather and airflow which are so complex, has to be an upgrade. And flight simulators weren’t even the only aircraft based digital twins that Karen brought up KW: Some of the work in my group in Digital Twins has been focused on unmanned aerial vehicles, UAVs. I actually have a small UAV in my group that we designed and built together with Aurora Flight Sciences. That is the physical twin that is a test bed for us. SW: UAV, like a drone? SH: That’s right, but instead of a remote controlled camera platform or weapon of war, which are two of the most common uses right now, think delivery platform or autonomous flying taxis. And KW: You know, imagine Sam, you're the operator of one of these, these fleets of autonomous aerial vehicles, and maybe you have a thousand vehicles in your fleet. You want to be able to track the health of all those vehicles. SW: Yeah you would, and I would want you tracking them because if one of your uav’s crashes into me I will be so pissed SH: I promise to take it hard too SW: Hey, that sounded sarcastic SH: Nooooo, not at all. I promise SW: I don’t believe you. Thankfully I also don’t believe you will be in charge of a fleet of UAVs anytime soon SH: Well you are right about that. If only because of all the maintenance KW: I can imagine my son standing on a street corner with a slingshot and his friends, you're trying to see if he can hit the Amazon drone that just flew past. You would like to know if you got hit accidentally or otherwise on a given day. And clearly you don't want to have to employ a team of people who are going to sit there and monitor these drones as they fly in this uncontrolled urban area. SW: And here I was just thinking of standard wear and tear, I hadn’t even started to consider human behavior SH: Which is why Karen and her lab’s work creating a digital twin for their drone is so important KW: Could we imagine a future where we create a digital twin for every one of your drones in your fleet, that digital twin is a living model that's following along so that when the UAV wakes up in the morning, or by the way, it probably doesn't wake up because it probably flies all night, the data are being collected, being assimilated into the digital twin so that the digital twin is a living, dynamically evolving representation of the health of that vehicle. SH: And these twins can then help you make decisions KW: And then you as the decision maker can use that digital twin to do adaptive mission replanning. What if you do get hit by the slingshot? What do you do? Do you land safely? Do you continue to deliver your package? Do you try to make for the nearest base? And also to guide your decisions about maintenance and just generally help you manage your fleet. SH: For example KW: If you've got high value packages, you give them to your healthiest, most trusted drones versus the ones that may be delivering less packages. SW: This is really cool and useful, but so far we have really only talked about various aerospace engineering based applications of Digital twins, but you mentioned earlier though that they can exist in a diffuse set of disciplines and I can’t help but wonder what they are KW: One of the things that we've been doing in my group is that we've been collaborating with oncologists here at UT Austin and also at MD Anderson Cancer Center to try to take some of the mathematical and computational methods we've developed for aerospace engineering and bring those over to the cancer setting. SH: In this case instead of looking at when a UAV should be repaired they are looking instead at when cancer patients should get imaging KW: One of my PhD students is looking at this. This is really an optimal experimental design question, which is to say, if you could bring a cancer patient in for an additional image, what would the value be? How much additional information would you gather? How much would it reduce the uncertainty you have in characterizing the state of this cancer patient and ultimately in making decisions about their treatment plan? SH: And if it is decided a new image would be useful, there is the questions of when as well KW: If you bring in for an additional image soon after the last image, things haven't progressed too much and you don't gain that much more information. But if you wait too late to bring in for an additional image, you may gain a lot of information about what goes on, but you haven't then left yourself much of a horizon to make adaptive control or treatment decisions. And so it's a real trade off. And again, this is really an optimal experimental design question. SW: Wow, I can see this being so important in health care going forward. I readily welcome the coming birth of my digital twin SH: Do you think she will also have a podcast with my digital twin? Maybe Carry the Digital Two? SW: Talk all about the applications of math and stats in the digital world? SH: I mean I would listen SW: Of course you would SH: Normally this would be the time that I would start bringing in how math and stats helps make digital twins work but instead I want to bring in another guest to give a sense of just how big and varied digital twins are. AM: My name is Anna Michalak SH: She is the AM: Founding director of the Climate and Resilience Hub at the Carnegie Institution for Science, and I'm also by courtesy at the Stanford-Door School of Sustainability. SH: Anna is also one of the organizers of the upcoming IMSI workshop titled Application of Digital Twins to Large-Scale Complex Systems, which is why it surprised me when she told me AM: In my day-to-day life as a scientist, I do not use the term digital twin. I don't think I have used that term in any paper I've written, in any proposal I've written. SW: Ok, definitely not what I was expecting from a guest in a digital twins episode but please continue SH: Before we discuss why an organizer of a digital twins workshop doesn’t use the term in her day to day work, we should discuss what science Anna and her research group focuses on AM: Half of our group works on questions related to the global carbon cycle. So where and when and how are greenhouse gases being emitted from the Earth's surface and then where are they ending up and what’s controlling the ability of systems like the terrestrial biosphere or the oceans to essentially offset part of the carbon that we are emitting. SH: And the other half of the group is AM: Interested in how climate is impacting water quality outcomes. So we're used to thinking about how climate impacts water quantity. So things like drought or flooding or extreme rainfall. But if you think about water sustainability, understanding water quality impacts of climate is equally important. SH: And both of these areas have something very important and relevant to the ideas of digital twins. For example with greenhouse gases AM: One of the really interesting and difficult aspects of that science is that beyond something like a laboratory scale where you can literally take a plant and put it in laboratory, there are no ways that we can directly measure the exchange of greenhouse gases at the Earth's surface SH: Which means AM: Instead, what we do is we work with a very large variety of proxy data to some degree. SH: And for water quality AM: We need to be able to measure both the things that cause water quality to change, as well as being able to track how that water quality itself is changing. And the challenge there is that there are relatively few systems globally that have actually been tracking water quality on a consistent basis for a long amount of time. And because of that, in some cases, we need to turn to remote sensing observations to some degree. SW: So they do have data, but it isn’t quite like the immediate and exact data that you would get from a sensor on a UAV SH: No, it is not. Which is why AM: In my field, the term digital twin is a little bit of a squishy concept. SH: Not that the computational and mathematical tools that make up a digital twin are not commonly used in her fields, just that the time and geographic scales can vary wildly from the engineering fields where it first caught on. So Anna sometimes sees digital twins used in earth systems science as a fancier way to say computer model, it often refers to specific classes of models AM: It's used to represent the fact that some of the models that we use are now getting to spatial scales and time scales that are much more refined and therefore much more closely tied to things that you might observe on the ground or to things that are relevant for policy or management that are at regional or urban scales. SH: And while they are not quite real time data acquisition and use of proxy data may make earth systems scientists use the term digital twins more loosely they are still using the same tools and techniques. So, are you ready to dive into the math and stats now Sadie? SW: Always! SH: That’s what I like to hear, and that is just what we will do after taking a moment to share about a University of Chicago Podcast Network show our listeners may enjoy (Ad Music starts) SH: Have you ever wondered who you are but didn't know who to ask? Well, then join Professor Eric Oliver as he poses the nine most essential questions for knowing yourself to some of humanity's wisest and most interesting people. Nine Questions with Eric Oliver, part of the University of Chicago Podcast Network. (Ad Music Ends) KW: So we really emphasize the computation, but the reality is that digital twins are absolutely built on mathematics and statistics. SW: Great to hear from Karen again, but what are these math and stats that the digital twins are built on? KW: We could start with the virtual representation, those are the models, the models of the system that are in the computer. Those models are mathematical or statistical models, they may be mathematical models like partial differential equations that are the governing laws of how the physical processes evolve in space and time. Or they could be statistical models using your favorite neural network or your favorite statistical data fit model. But the core of that digital twin virtual representation are mathematical and statistical models. SH: Then once they have the models, the next step is to get the data to flow from the real world into the model KW: If we start with the flow from physical to virtual, this is the flow where we are acquiring data or observations from the physical world and feeding them into our models. That mathematically is data assimilation or an inverse problem or a parameter estimation problem. And so again, there we see mathematical methods for how you mathematically pose that task of acquiring data and updating a model. And there is a long history in the mathematical community of solving inverse problems and data assimilation at scale. SH: Finally you have the other direction of the bi-drectional data flow KW: The flow from the virtual back to the physical world, this is where we're now using the virtual representation to effect change in the physical world. And there we are talking mathematically about an optimal control problem, an optimal design problem, an optimal experimental design problem. So again, the mathematics and the statistics are absolutely the building blocks of what goes on that flow in the digital twin. SW: Ok, there is so much math and stats involved with digital twins. SH: I know. Some we have definitely talked about a lot on Carry the Two SW: For sure, I mean there is barely an episode that goes by where we are not talking about using models to numerically represent processes in the real world and I think talked about inverse problems before SH: I am pretty sure they came up in the last episode about computation imaging. The basic idea is that you start with an end state and trying to figure out the state the system was in the beginning SW: That’s right, but what about optimal control and optimal experimental design problems? SH: These are problems where you are trying to control the parameters of a system or experiment to optimize for some feature. Such as choosing the best route for an aircraft to maximize speed while minimizing fuel use given the air currents or designing an experiment to limit the occurrence of bias or variance SW: That makes sense, but it does leave one more. What is data assimilation? KW: So data assimilation is a mathematical formulation for taking observations in the real world, so collecting data and assimilating it into a computer model, meaning that we may be updating estimates of different parameters in the model, updating estimates of states or initial conditions SH: And while we did not know it, we rely on data assimilation to help us out in our day to day lives SW: We do? How? KW: Data simulation has been making our weather forecasts better and more reliable for many years now. The process of acquiring data from sensors that are deployed out across the globe, taking that data, using the data to adjust the parameters in the physics-based models that are used to model and represent the weather, adjusting those parameters so that they are better matching the data that we've been seeing in the past, and then using the adjusted models to predict out into the future. SW: Ooooh, of course. Weather forecasts rely on so much data SH: And assimilating that data has helped me from getting soaked many times SW: Yeah, as long as we remember to actually check it before running out the door SH: There is that. And there is also one other mathematical and statistical concept that is central to digital twins KW: With all these ingredients, the role of uncertainty quantification is absolutely essential because if we're going to have digital twins that are updating models based on sparse, imperfect, noisy data and then issuing guidance for decisions that may be critical, maybe absolutely high consequence decisions, we have to be characterizing uncertainty. And so the need to have methods that quantify uncertainty and embed it in all those elements, uncertainty in the models, statistical formulations of the inverse problem and the data simulation task and then optimization under uncertainty in the decision task, again, those are essential ingredients. SH: And as use of digital twins continue to expand across scientific fields uncertainty quantification will only get more vital KW: But then also recognizing that a digital twin in a medical setting might have very different needs or it just might mean differently what it means to trust a digital twin in the medical domain as opposed to what it might mean to trust a digital twin in, say, an engineering domain SW: You can say that again, while I really do not want unmaintained UAVs falling out of the sky I am much more invested in making sure that my doctor and I fully understand the uncertainty baked into my medical digital twin before making a decision based off of it SH: Then it is good that Karen has been working on developing techniques to embed uncertainty quantification directly into digital twins KW: You know, I've worked and I've worked with collaborators on statistical formulations of inverse problems where uncertainty is carried along at every stage so that as you are doing this mathematical process of data assimilation, you're not just updating the models, but you're also carrying along the uncertainty that you have due to the fact that the data is imperfect or that, you know, your models may also be imperfect. SH: And making it as quick as possible KW: I've personally also done a lot of work in multi-fidelity methods that aim to accelerate uncertainty quantification. As soon as we start talking about embedding uncertainty quantification in an inverse problem or in an optimal control problem, all of a sudden that computational task becomes much more computationally demanding. And if we layer on top of that the need that digital twins may have to issue decisions in real time or in very rapid turnarounds, we have to be computationally efficient. And so that's where surrogate modeling, reduced order modeling, and multi-fidelity methods really come into play. SW: I have to say, hearing all of this that digital twins sound super exciting but also super complicated and, frankly, expensive in many different ways like sensors and labor and computation to name a few SH: That they are, and it is why Karen thinks that it is very important to build them in a very purposeful manner. For her, KW: There's the question of what you could phrase as fit for purpose. What's the purpose? What are the decisions you want to drive with with the digital twin and how good does it need to be? SH: And the answers to those questions then bring up KW: The resolution and fidelity of the models, but also the constraints on the computational power you have to execute things and the time you have to make a decision. SH: Because, after all KW: You could have the most exquisite model for forecasting the weather, but if it only finished making its predictions, if it took seven days to make a five-day prediction, it's not terribly helpful. SW: Yeah, a two day late warning isn’t going to keep me dry SH: No, no it is not KW: We realize that there are these trade-offs between fidelity, complexity of the model, how much data we can ingest, the rate at which we might want to do updating the models. And all of these different aspects have to now be assessed. The trade-offs have to be made. And the digital twin has to be sort of intentionally designed accounting for all of that. SH: And in order to design in this intentional way, Karen thinks we need to think of digital twins a bit more concretely KW: I think it's valuable for us to start thinking of the digital twin as an asset, as a thing, just like we think about the physical UAV or the cancer patient or the ice sheet as physical things. Let's think about the digital twin as also being an asset, as being a thing. SW: That makes total sense, it is how successful software companies think of their products too. Sure in the end it may just bit bytes on a hard drive but they view their code as having a lifecycle that includes its concept, its support, and even its end of life the same as a vehicle or power plant SH: Exactly, and digital twins start in the same place as your example KW: And the lifecycle starts with the conceptual design, the architecture, thinking about how do you even architect this digital twin. What are the data that you maybe have already, that already exist in your current systems, but what new data you might want to acquire if you were willing to invest in additional sensing capabilities? Thinking about the decisions that you might want to drive, and again, they could be a combination of things you already do, but things and new things that could be enabled by the digital twin. So really architecting at a conceptual level, the digital twin and the role it could play in your system. SH: Conceptual design isn’t the end of the design phase either. Once it has been hashed out the detailed work begins KW: And when we talk about detailed design of the digital twin, we're now talking about making those decisions of how sophisticated, how high fidelity should the models be? What are the constraints on computational power? What are the constraints on computational time? How do we really design the digital twin so that it meets our system requirements? SH: The design is then implemented using math, stats, and coding work that manufactures the digital twin. Followed by KW: After manufacturing typically comes tests and evaluation, validation, verification. SH: At which point the digital twin should be able to be made operational. Not that this means the work is done KW: With operation, again, I sometimes use the analogy, can you imagine a world where engineers designed a bridge and built it and never had a plan for inspections and maintenance. That is not what we do in the physical world, so that is not what we should be doing in the digital world. We should be creating these digital twins and then having plans for maintaining, sustaining and upgrading SW: Yes please, I want these digital twins to be sustained. I mean I would love all code and software to be maintained, but the idea of these digital twins is to test out and drive decisions and policies and that makes it feel even more important SH: And once we start considering sustaining and maintaining digital twins the idea of end of life comes in. And sunsetting a digital twin starts to feel even more important when we think about how personal they may be, such as the medical digital twins we have talked about KW: What happens when that patient is no longer undergoing treatment? What happens to the digital twin? What happens to the data and the models that have been built? So thinking very intentionally about all those, all those phases. SW: Ahhhh, I hadn’t even started to think about the implications there. I mean if my treatment is over I kind of like the idea of the digital twin being deleted, but I can see how it could be useful to keep them around if I have a recurrence of my symptoms SH: I get that, but even if the twin is deleted does that mean the data that was collected should also be deleted? Or should that be kept for the next time you need a digital twin to help personalize your care SW: That is too many questions Sam, that is supposed to be my role on the show. You provide the answers SH: Sadly there are no answers yet to these questions. But I can promise you there are great people like Anna and Karen looking into them. In fact it is the fact that there are so many open questions still in the digital twins space that makes IMSI’s current digital twins long program so exciting. Because as Anna told me AM: We all tend to come in with relatively naive assumptions about how a particular tool is used in a discipline other than our own. We assume that there are more similarities than there are and exposing those differences in of itself has value because it forces us to think about how we use a particular set of tools within our own disciplines in a more critical way. SH: Which is why interdisciplinary workshops are so important for math, stats, and science. Not only will it be a chance for experts from across IMSI’s research themes from data to health to materials to uncertainty quantification but once those experts start to understand these differences AM: If done well, there's the second half of the meeting where you realize that if you drill down to a deeper level, there are actually commonalities between these applications and there are lessons to be shared and that we can learn about from each other. And that tends to be where the really juicy and interesting parts of the conversations take place. SH: Conversations that can be really important and impactful AM: And so this exchange of ideas across different disciplines, but with a focal point, in this case digital twins, that is common to all of our applications, tends to make for just really interesting conversations. And then I tend to come away really thinking about my own science in a different way. SW: I just love those types of conversations, the ones that help you get a new perspective on your own work. So many of my best ideas have happened after one of those moments and interdisciplinary spaces are always the most likely to generate them SH: Which is perfect since, as Karen shared, digital twins are interdisciplinary by their very nature KW: Just as we think of the ingredients of a digital twin and all the things we've talked about, we've talked about mathematics, we've talked about statistics, we've talked about computer science, software engineering, we've talked about the domain science, we've talked about human decision makers, behavioral concerns. So in order to build the very best digital twin for your favorite application, you've got to bring all those ingredients together. And in order to advance the frontiers of what's possible with digital twins, we need to be able to bring those different fields together. SH: And she doesn’t just mean researchers either KW: Academics need to be there and pushing forward the frontiers of the methods, but the collaborations need to be built with the clinicians all the way out to the clinicians, the people in the clinics who are going to be using the digital twins and everything in between. SW: YES! I mean I have my PhD, I love academics and researchers but we do have to keep in mind that we are not real life, and often there are real people out there who may be impacted by our work, sometimes unintentionally. SH: Too true, and if digital twins become common place we need everyone to be a part of their construction SW: From your lips, to… Well to the ears of everyone who is working on digital twins, many of whom you will be able to say it to directly SH: I know, which is why I am now off to figure out exactly how I am going to bring this up… SW: Good luck… (outro music) SH: Don’t forget to check out our show notes in the podcast description for more Karen and Anna, including links to their talks they all gave at IMSI and the work that we discussed on today’s episode SW: And if you like the show, give us a review on apple podcast or spotify or wherever you listen. By rating and reviewing the show, you really help us spread the word about Carry the Two so that other listeners can discover us. SH: And for more on the math research being shared at IMSI, be sure to check us out online at our homepage: IMSI dot institute. We’re also on Bluesky at IMSI dot institute, as well as instagram at IMSI dot institute! That’s IMSI, spelled I M S I. SW: And do you have a burning math question? Maybe you have an idea for a story on how mathematics and statistics connect with the world around us. Send us an email with your idea! SH: You can send your feedback, ideas, and more to sam AT IMSI dot institute. That’s S A M at I M S I dot institute. SW: We’d also like to thank Blue Dot Sessions for the music we use in Carry the Two. SH: Lastly, Carry the Two is made possible by the Institute for Mathematical and Statistical Innovation, located on the gorgeous campus of the University of Chicago. We are supported by the US National Science Foundation and the University of Chicago.