Episode Transcript
(Intro Music Starts)
SH: Hello it is your host Sam Hansen and I am excited to welcome you back to Carry the Two, the podcast about how math and stats impact the world around us from the Institute for Mathematical and Statistical Innovation. While we’re in between our more in-depth seasons, we like to bring you something a little different in mini-season format. And for this mini season, we are going to highlight some of the amazing researchers who have presented at IMSI over the past year. Our fifth guest is
RY: Hi, my first name is Rose and last name is Yu. My Chinese name is Qi Yu, but I publish under the name of Rose Yu. I'm an associate professor in the computer science department of UC San Diego. I'm also affiliated with the Halıcıoğlu Data Science Institute.
SH: Rose joined us at IMSI for a workshop on Learning Collective Variables and Coarse Grained Models where she presented a talk titled Automatic Symmetry Discovery from Data. So, without further ado let’s get into my conversation with Rose Yu
(Intro music ends)
SH: I'm going to start off with, with a couple of questions that may seem simple, but I think are very important to get into the work that you shared at IMSI. So what are dynamical systems? And if at all possible, could you provide an example?
RY: So in mathematical, like, study, dynamical system is basically systems that evolve over time, anything that changes over time. And dynamical system is a more formal definition of that. So an example for that, be it our Earth, and Earth is changing all the time with the weather is changing, with the clouds moving around, and different precipitation.
SH: Great. And then the next, the next question to make sure that we are all in the same page as well is, when you are talking about symmetry, what, what do you mean by symmetry?
RY: Yeah, so symmetry, I guess the most common notion of symmetry is something that is symmetric. Say, if we look at the wings of a butterfly and the wings of a butterfly are symmetric. So that's a very easy example of that. But if you think about from some mathematical operation on an object, that's that if the butterfly's wind is symmetric, that means if I flip, you know, one side of the wind to the other side, it will look exactly the same.
So in general, symmetry refers to this kind of transformation that leaves something unchanged. If I rotate an object, let's say if I rotate a ball, then the shape of the ball doesn't change. The ball will look exactly the same after I rotate it, regardless of which angle I use. Then it's obvious that the ball would have a perfect rotational symmetry.
SH: And why is it important to be able to find and discover these symmetries?
RY: A lot of times when we build AI algorithms, we wanted to make the AI algorithms like sample efficient, meaning they can learn from a few examples, just like humans. Humans can learn to walk, learn to ski, but just seeing a few examples. And one of the hypotheses in this line of study is that one of the reasons that humans can do this thing so fast is because there's an internal symmetry in how we recognize things in the world. So this kind of idea has already been built into the design of AI algorithms.
For example, one of the very famous neural network architectures in deep neural network is called a convolutional neural network. The reason that it's so successful is that it utilizes the fact, there's a lot of symmetry in the data. So convolutional neural network is very good at recognizing patterns in images. So if we think about, like, an image of a rose, when we zoom into the image, and we can always recognize this is a rose picture regardless of the size of the image or if we shift the image by some pixels, to us it's still pretty obvious that it's a picture of a rose. So CNN basically built this kind of principles into its design, such that the output of this neural network does not change whether or not we shift the images by some pixels or ways we do some transformations of that images.
So there has been a lot of success of building in symmetries of the data into the design of neural network architectures. So the examples I gave in terms of images are pretty obvious because there's a translational symmetry as in we shift the image by left and right pixels. The object, the concept of the rose doesn't change. But a lot of the real-world data, the symmetries are not obvious to us.
For example, if we're looking at the data of atmospheric like in the earth, you want to look at how the temperature on the globe changed over time, then it's actually not obvious what the underlying symmetries are. In order to build neural network that can leverage the symmetry, we have to know what symmetries to use. And that's the reason we wanted to discover symmetry automatically. So then after the discovery, we can use the discovered symmetry as a inductive bias to design neural network architectures. Instead of human, try different type of symmetry and like brute force.
SH: Inductive bias is an interesting term. What does that mean in the machine learning context?
RY: Inductive bias, it really just means what type of prior knowledge that we think we should use to design our models and algorithms. So, let's say if I have an intuition of the image being translation invariant, then I can just build that prior knowledge into the design of my model. The other case is, let's say I have a pretty good idea that the underlying model is not very complicated.
So I can choose, you know, maybe I will use a very simple model. Say a linear regression model rather than a very complex model like a multilayer deep neural network. And the choice of such model class is another type of inductive biases. You know, it actually resembles how human reasoning works. We have thousands of thousands of years of evolutionary inductive biases that are built into our reasoning system. And then whenever we make a prediction about certain phenomena in the real world or try to reason about some phenomena in the world, we will combine the evolutionary inductive bias we have with the data that we see right now.
And that's kind of how machine learning models work. They will use the inductive bias that's designed by engineers and scientists. And then also combine that with the data it was fed into to make a better prediction.
SH: When you are trying to create and work with these, these machines to try and find these symmetries that you're looking for, what, what were the sort of important goals that you were hoping to meet with the systems you were helping to create?
RY: One of the goals that we hope is to basically reduce the effort of searching in a large space of symmetries. So before our algorithm, people who use this kind of inductive bias to design neural networks have to try a lot. And there are so many possible symmetries that we could use to design our neural network. So it becomes a very tedious process. And our goal is to lift that burden and let the machines do that for humans and do that automatically so that we can discover the symmetry from data.
And another motivation for this work is according to Noether's theorem, there is a very intricate connection between laws of conservation and the concept of symmetry.
For example, conservation of momentum would correspond to time translational symmetry. So if we can discover the symmetries automatically from data, then we also hope that it can shed light on what kind of physical laws that the system have. We can then draw conclusions about unknown phenomenon in the data and that would help our scientists to make better and faster discoveries.
(Ad music starts)
SH: If you're enjoying the discussions we're having on this show, there's another University of Chicago podcast network show you should check out. It's called The Chicago Booth Review Podcast.
What’s the best way to deliver negative feedback? How can you use AI to improve your business strategy? And why is achieving a soft landing so hard?
The Chicago Booth Review Podcast addresses the big questions in business, policy, and markets with insights from the world’s leading academic researchers. It’s groundbreaking research — in a clear and straightforward way. Find the Chicago Booth Review Podcast wherever you get your podcasts.
I also want to let you know that New York Times bestselling author Steven Strogatz is giving a public lecture on mathematics for biological discovery and biology as inspiration for mathematics on Wednesday, May 7th in downtown Chicago, hosted by the NSF-Simons National Institute for Theory and Mathematics in Biology. Strogatz’s research has helped to uncover the mysteries of the human sleep-wake cycle, the collective behavior of firefly swarms, and more. Get free tickets at NITMB.org.
(Ad music Ends)
SH: With those goals, as you just explained them, how exactly can you get machines to find symmetry from just data?
RY: Yeah, so the idea that we build upon is actually coming from a very classic paper in deep learning. It's called the Generative Adversarial Network. So it was actually awarded the Test of Time award at NeurIPS, which is one of our top conferences in AI. So the idea of GANs or Generative Adversarial Network is that you wanted to discover something by basically training two neural networks as a game. And you have one neural network that generates different type of possibilities. And you have another network that tries to decide whether this possibility is real or fake.
So GANs was used to generate new images to discover in the space of all possible images, which are the images that are real, that are realistic. So GANs were super successful at, like, generating realistic images because of that. It has a generative network that generates all kinds of possibilities and it has another discriminator network that discriminates whether this image is real or fake. In contrast to generating images, we are trying to generate symmetries. So in our work, we also have this two neural network set up. We have a generator network that generates possible symmetries. And then we have another discriminator network that tries to decide whether the generated symmetry is real or fake.
So because we can quantify whether some symmetry is real or fake, so that will become the training objective function of the discriminator. And then once we train the neural network to a point that the discriminator is confused. So basically, it cannot tell whether it's a real symmetry or whether it's a fake symmetry. Now we know we're successful because the generator network has really fine symmetry that is almost like realistic in the data.
SH: While you were training these networks, what were, what were some of the some of the, some of the obstacles that you ran into? What were some of the things that you had to, had to overcome either in the initial work or maybe while you were trying to expand it out even further?
RY: One of the challenges is just because GANs are very hard to train. So in general, because we need to train two neural networks, and then they played a game together. So one is trying to fool the other and the other one is trying to decide whether the generated output is real or fake. So this kind of two network joint training procedure is a very hard optimization problem. And it's a known problem for GANs and also, and also extends to our setting. So the optimization itself is challenging. And then we used some engineering tricks to overcome that difficulty.
The other challenge that is unique to our problem setting is, when we try to find these symmetries from data, there is a possibility that we can find a very trivial symmetry, which is identity, right? So if we remember the definition of symmetry, it means when we transform something and that object doesn't change, that was what we say symmetry. But a very trivial symmetry can just be we don't do anything to that object. So if we don't do anything to that object, of course that object doesn't change. But then that's not the very interesting symmetry that we found. So we call it like a trivial symmetry.
So sometimes during the training process of our models, it can also fall into this very trivial symmetry that we don't actually want to discover. So in the, in the algorithm, we basically redefined an objective function such that it tried to avoid this type of trivial symmetry.
SH: I don't think I'm necessarily spoiling too much by saying that you do now have algorithms that can help find these symmetries. How are you hoping that, that these automatic symmetry discovery algorithms will help with, like, dynamical systems or even out past that?
RY: One of the applications that we showed in the paper is that when we apply our algorithm to high energy particle physics data, so imagine we have particles being accelerated close to the speed of light and then they collect each other that will generate new particles. So it's a very famous Large Hadron Collider experiments. So people have been searching for new particles and hence new physics in these kind of experiments.
And in our paper, we showed that there is, you know, a symmetry called Lorentz symmetry that was very well known in particle physics. But our algorithm without any knowledge about particle physics actually found the Lorentz symmetry purely from data. So that was a very surprising moment for us and also our friends in particle physics. And so then we basically showed if you just use this discovered Lorentz symmetry to build neural network, then the performance of that neural network is almost the same as neural networks specifically build for particle physics prediction problems. Right, but then also remember that that's a very specialized neural network for particle physics prediction problem that was built by a bunch of really smart particle physicists. But then our model does not have that kind of expertise in domain science, but still it was able to match the performance of that specialized design model.
So in terms of future work, I think this, this line of research essentially open up the possibility of automated scientists. So we're saying that a lot of the procedures in scientific discovery can be repetitive and then we can use machine learning to automate that, so that our scientists can really focus their energy on the most creative part of scientific discovery instead of spending a lot of time on tedious repetitive procedure of the work. So I can imagine this automatic symmetry discovery framework to help scientists find new symmetries in their data, which may lead to new laws of physics in the data.
And recently we had a follow-up paper that demonstrated that if we use the discovered symmetry from data to inform governing equation discovery, which means if we gave the model a lot of these trajectories of an apple falls from a tree and the model can spit out the Newton's law, that kind of setting. And then we found that the symmetry can actually help the model discover much simpler and cleaner equations. So that's like another application that we have just published and we're extending that kind of work to help more efficient scientific discovery.
SH: Oh that's, that's really quite incredible. And I look forward to continuing to see where this goes. I have one, one last question and that is, how was it for you to be a part of, of the workshops at the Institute for Mathematical and Statistical Innovation? Like have you, have you, been able to meet up with other interesting people, maybe any new collaborations, anything like that?
RY: Yeah, so I was involved because there was a workshop at IMSI and they invited me to give a talk. Unfortunately, because I had a very small child at the time, I didn't travel in person, so I only gave the talk remotely.
The positive thing I would say, I got a lot of really great questions, especially from domain science researchers in molecular dynamics, in biology, and chemistry. And people were interested in finding symmetries in the dynamic system in that domain as well, because molecules move all the time, they jiggles a lot, like atoms jiggles a lot. So then there's a question about what kind of symmetries exist in this kind of data. Similarly, in biology, we can look at proteins and cells, and they also have this kind of nature of dynamical systems that everything is moving all the time. But what is the underlying hidden symmetry in this kind of biological data?
So I got a lot of these questions from domain science that I'm not familiar with. And after the talk, I also received several emails from folks who were at the audience. They asked very, very good questions. Yeah, then some people who listened to the talk, and they later met up with me in other conferences in person, and I was able to make some really good connections that way.
SH: That's wonderful, and Rose, thank you so much for giving me your time today and talking about your work.
RY: Thanks a lot for asking these great questions.
(Outro music starts)
SH: As always, don’t forget to check out our show notes in the podcast description for more about Rose’s research and to watch his talk.
And if you like the podcast, please make sure to follow, subscribe, or like the show in your podcast app of choice
It is the best way to make sure you hear all of the episodes of Carry the Two
For more on the math research being shared at IMSI, be sure to check us out online at our homepage: IMSI dot institute. That is I M S I dot institute. We’re also on Blueskey at imsi.institute, twitter at IMSI underscore institute, as well as instagram at IMSI dot institute!
Do you have any mathematical or statistical questions? Maybe you have an idea for a story on how mathematics and statistics connects with the world around us.
Send us an email with your idea!
You can send these ideas, your feedback, and more to sam AT IMSI dot institute. That’s S A M at I M S I dot institute.
We also like to thank Blue Dot Sessions for our Music.
Lastly, Carry the Two is made possible by the Institute for Mathematical and Statistical Innovation, located on the gorgeous campus of the University of Chicago. We are supported by the National Science Foundation and the University of Chicago.
