MLOps Innovator Series: Bridging the Gap to Observability

Webinar Transcript

Dan: Hello, and welcome to another edition of MLOps Innovators. I am here with Danny from WhyLabs. Danny, tell us, how are you doing, my friend? Good to have you on.

Danny: Thanks. It's good to be on. To be honest, Dan, I'm still waking up a little bit, but I'm excited to be here and talk about observability and monitoring and all the things that work better than coffee for me.

Dan: [laughter] Well, you look fresh as the day is new, my friend. So I'm not worried about this in the least. I wish I looked that good this early in the morning. So tell everybody a little bit about yourself and your history. I think you went to a amazing school. You've done all kinds of statistics and you met the president, you went to space. [laughter]

Danny: You might be overselling me a little bit on that one. [laughter] Maybe I'll look less impressive.

Dan: Tell us a bit about the background, how you got into this whole space and what you're doing now.

Danny: Yeah. So I went to UCLA for my undergrad and I studied statistics with a focus in computation which basically just meant that I took some CS courses and got really into machine learning by my senior year. It was a little bit of a weird journey to get there. I'd actually started as a poly-sci major, which I'd gotten enough credits that I graduated with a double major because I've been planning to be a lawyer. But then I started taking statistics classes and I was like, "Whoa, this stuff is way harder, but also way more interesting and way more applicable to so many situations in real life." And I just fell in love with the ways that we use statistics and probability theory all the time throughout our lives. And then I interned at a company called Qubole, which at the time was offering a data platform as a service. It offered managed Apache Hive, Apache Spark, Presto, Apache Airflow. And I really just fell in love with machine learning in the data space and was like, "Wow. I want to build my career around that." So since then, I worked at Imply. I took a brief stint where I was going to work as a data scientist at BCG Gamma in Singapore, but then COVID hit, and then I moved back to the United States. I worked at a company called Imply, which offers Apache Druid as a managed service. And now I work at WhyLabs where we help companies get ML observability and really figure out what's going on with their models once they're in production.

Dan: I mean, that is a cool journey. What's interesting is sometimes when these spaces are developing, they have a very different trajectory of how people get into them, right? So I mean, this is sort of my fourth technological revolution. And in the early days when I was working in the internet people said, "Why are you wasting time with that platform? This is never going to do anything. Don't waste your time there." They said the same thing about Linux. And I think people get very used to these industries that have been around for a super long time. There's already a huge set of courses at college or whatever, but you have to kind of come to it from another space and kind of zoom in and figure out, "Oh, I was a lawyer and I like statistics and probability," and all of a sudden, "Oh, machine learning is this new thing." And to me that's the most exciting time to be in a space, is when it's still a little bit undefined and it's still rough around the edges. I don't know if you feel the same, but.

Danny: Yeah. Absolutely. I mean, I really love getting into a space while it's still being defined and really finding the intersections of spaces because I think that's where the most interesting innovations happen, right? And that's where our space came from, right? And machine learning is kind of the intersection of statistics and computer science. Data scientists are like statisticians, computer scientists, and hackers all mixed into one person. And I think that's where really amazing cross-pollination and innovation can happen is when you're taking people from different disciplines and bringing them together to see how problems in one space relate to solutions from another space.

Dan: Totally. That interdisciplinary thinking I think is absolutely critical. And it just makes for-- it makes the space super exciting in the very beginning and I frankly can't work in a space once it's been super well-defined. I get bored by that point when it's just kind of turning the knobs. I love it in the beginning when you have to think of the strategy and think outside the box and it's incredible to be there. But let's loop around to a bit of WhyLabs. WhyLabs' been out there, they've been killing it. They did an awesome [raids?]. The CEO is really-- I've seen her everywhere in the news and just doing fantastic work. I've just talked to another team that's simulation science that you guys were co-writing a paper with and they're doing some amazing work with the team. So tell us a bit about-- since WhyLabs essentially are working on logging. There's a number of folks in there in that space. It's fast developing, it's really fast moving. But tell us a little bit about kind of the history of logging, but also really why is it different? I think that's where we're really going to go to. Right? It's like logging has been around for forever, right? And I had an IT consulting company for a decade and there's logs for everything. But why is it different in the ML space and kind of walk folks through what's changed when we get to the MLOps world?

Danny: Totally. And I think to kind of segue it, I mean, where we exist in MLOps is really exciting because it's an intersection of a machine learning space, right, that's like how do you deploy models and DevOps which is a more classic software engineering space. So getting to sit at the middle and look over data scientists' shoulders to see what's going on in the world of data science and then also look over the shoulders of SREs and DevOps engineers and say, "What's going on in DevOps," and being able to pull innovations from DevOps into MLOps to enable producing, productionizing, deploying, and then monitoring and ultimately getting observability on machine learning models is a really fruitful experience. And you mentioned that there are a few companies that are popping up in the space and I actually want to challenge you on that one, Dan. I have yet to see anybody else who's doing machine learning logging. There's a lot of companies out there that are trying to offer monitoring, they're trying to offer observability, but none of them are taking the classic DevOps approach of having a logging agent installed alongside the machine learning model within the machine learning system and then sending those logs, these artifacts that are being collected to an observability platform. I hope they catch on and realize there's a reason why the DevOps world isn't deploying massive pieces of software alongside their Software 1.0 applications. But for now, we're really able to differentiate as the only company that's offering an open-source data logging standard within machine learning.

Dan: All right. So I remember when I talked to your team in the early days and I forget who said it directly, but I usually quote a few times, right, when I talk about the Infrastructure Alliance or parts of the space we're developing. And I remember, I said, look, what is different about the logging in the traditional world versus machine learning? I think the definition they gave me that was fascinating to me was in the regular IT space, you don't much care about anything except the current state of the world. Right? So in other words, if I have a web server, I care whether it's up or down, it's very binary. Right? And I may care about the history a little bit if that server has been bouncing over time, but once I kind of solve that problem, I can kind of discard the history. Whereas when you're looking at matching learning, you're talking about inference and training over an extended period of time. So you really absolutely need the entire history and so you really need a completely kind of different backend to be able to deal with it. Because if you're looking to do a knowledge detection or something like that, you have to know all of the decisions that the engine has made. Would you say that that's kind of held true as you have kind of gone through the evolution of the company and anything you think would kind of add to that?

Danny: I think understanding historical state is absolutely critical to observability, right? If we want to be able to understand the system by looking at its outputs, we need to understand how changing the inputs has changed the outputs over time in order to understand the system as a whole. So that's definitely held true. The other thing that I would highlight is that the things that are being logged, literally the events or the metrics about the system are a superset of the logs or the events about the system that you want to be tracking in Software 1.0, right? If we think about a deployed machine learning model, this is still an application that's been deployed. It's still a piece of software at the end of the day. But in addition to wanting to monitor the system that the application is running on, so the CPU consumption, all of that stuff, and in addition to wanting to make sure that we're getting literally logs from, hey, we successfully moved the data from this pipeline to this machine learning model, you actually also want to be able to introspect the data that's being moved. I think that's one of the really big-- I mean, that's honestly why you can't use something like StatsD in combination with Prometheus and Grafana, these really classic application monitoring tools, for ML monitoring because ultimately those will give you a Software 1.0 view of your system as if it's a static, stable system that doesn't have variable inputs over time. But the crazy thing that we have to deal with in Software 2.0 is that we've got huge variations in the inputs to the system and to the thing that the system is trying to model, which is the outside world. So because of the increased amount of dynamicness, we actually need to be tracking different things, and we need to be able to look historically over time, as you said.

Dan: All right. So there's a lot in that answer that we can kind of dig into, and I'm going to peel it apart a little bit for the folks listening and watching. But let's start on kind of one part of this is, define observability. You dropped that term a couple of times, I feel like it's showing up in your marketing, I feel like it's showing up everywhere. I feel like it's a great-sounding word, but it's also one of those words where it's like God. Everybody hears the word God. You think it's uniform, but it's not. Everybody hears their own thing, the agnostic, and the atheist hears something, and the Christian hears something else, right? So tell me what observability is, give me a definition of it, and kind of how it's important.

Danny: Yeah. I can evangelize the altar of the observability god. So observability as a word actually comes from control theory, so related to CS, but not very distinctly in our space, where it literally means the ability to get insight into the state of your system by looking at its outputs. So this sounds pretty straightforward. If we dig into it a little bit more, we realize that in machine learning systems, this definitely requires the ability to look at historical state. And the really incredible thing that observability does is it allows you to debug, right, at a much deeper level because it allows you to look and introspect into your system, understand what's happening in your system, so that if there's a problem in your system, like if your machine learning model degrades in performance, if you've got bad data being passed through your data pipeline, whatever it might be-- there's so many causes of problems in machine learning systems. But whatever the cause of the problem in the machine learning system, observability is absolutely table stakes to be able to understand what's happening in it, so that you can debug, so that you can do a postmortem, so that you can go in and fix it, right? So it goes beyond just writing comments in your code that prints when something happens. It's really about being able to introspect deeply into a system to see what's going on with it.

Dan: You said one thing there that we'll come and touch on. Another thing I want to touch on here is absorbing bad data into the system. So kind of define the bad data. I mean, are we just talking about a corrupted file or are we talking about something that's mislabelled, right? And so that we're feeding in dogs and cats, and somebody labeled the dog a cat or put the bounding box in a ridiculous area around the wrong thing. How does kind of observability help us kind of solve the bad data and what do you mean by that deal?

Danny: Yeah. That's a great question, and the answer is all of it. I was using bad data kind of as shorthand. At WhyLabs, we differentiate between two very distinct but very common problems that we see in machine learning systems. One is related to data drift, so it's changing data over time. The real world changes. It causes changes in the data that's being generated. This new data is being passed to the model. The model may no longer be representative of the real world, and then you got model performance degradation. And this is talked about a lot by basically everyone doing ML monitoring, right? Everybody talks about data drift. But to be honest, data drift is only part of the problem. Another pretty significant part of the problem lies more with data engineering than it does with the broader outside world and fundamental cool statistical concepts like covariate shift. So when I talk about bad data, I mean it in contrast to changing data. So we can have bad data where, yeah, a file gets corrupted, or a bunch of nodes get introduced, or something that should be between zero and one suddenly has a two in it, or there's a bounding box that's in a crazy location.

Yeah. All of these are examples of something going wrong in the data engineering pipeline before data gets passed to the machine learning model that has caused problems in the data that can only be remediated by going back to the data engineering processes and either rolling back to a previous version of the pipeline or doing some kind of debugging. And that's really what we mean when we talk about AI observability, is going beyond just monitoring your model. Yeah, of course, you have to monitor your model in production. You wouldn't deploy a piece of software and not monitor it. But in addition to monitoring the model, when something happens, when you've got a problem, you need to be able to say, "I know how to fix my problem." And that's what observability is. It's the ability to figure out what's causing model failure or model performance degradation and being able to go in and debug it.

Dan: Are you talking about, essentially, just building heuristics? What you talked about in this example there where maybe the-- that you're putting a boundary around and it's supposed to be one or zero and there's a two in there. Are you having to go in and define these kinds of things? Because I also feel like certain types of data problems are very difficult, I think, to automate, right? In other words, if I-- and [inaudible] talks about this. We'll talk about him in a bit, but he talks about how a lot of people focus too much on the model, right? This is the data-centric AI thing. And maybe you've got five different models and they can do 0.2% different performance. But then you go back and you look at all the bounding boxes around it and you realize you thought you gave clear instructions to the labeling team, but of course, the labeling team is diverse and people are human, so they're going to do different things. And suddenly somebody interpreted one of your rules as this and put the bounding box here and another person put it here. So you go back, you find the rules, understand them, and say, "Look. Let's go ahead and redo 30% of these things." And all of a sudden, you're getting 20% better improvement. I feel like that's something that only, at this point in time, kind of humans can easily handle. That kind of inside into what's wrong with the data. But is there a way for this sort of observability to help with that level of harm or are we really talking about building kind of heuristics around kind of basic things, basic sort of bounding boxes around various types of data integrity?

Danny: Yeah. That's a great question. And the answer is people take different approaches. To me, there's kind of two layers. There's the very heuristic focused, having to manually define things approach, and then there's the fully automated, using machine learning to monitor machine learning. So if we take a look at the first one, that's something that you can do with our open-sourced library whylogs. You can set constraints based on, "I know that I want my data to be constrained to be between zero and one. So throw me a warning if it goes outside of this." You can also use something like Great Expectations, which is another open-source library that kind of also allows you to manually define set thresholds and to get a warning when you get past this threshold. Now, to be honest, this helps a lot of people a lot. Great Expectations is a really popular library. People use constraints in our library quite a bit. And the reason why is because there are situations where you just need this manual heuristic-level setting. But the real world tends to get messy and complicated. And the customers that we serve have sometimes massive models with thousands of features and we're the only ones that can support them because we've put in a lot of work into setting up good automation around monitoring.

So we can say, "Look. You've sent us seven data profiles from whylogs, and so far, the data has always been between zero and one. But now we've got a two in here. That seems weird. Here's a warning. There's something out of distribution, we think." And what's important here is that the customer didn't need to manually tell us, "Data should always be between zero and one," because we had the historical view of the system-- harkening back to an earlier fact. We had this historical view of the system, and now, as a result of new data, we can say this is or isn't an anomaly. So users can come in and after they've gotten an alert they can say, "This wasn't a helpful alert. Actually, this data can vary beyond zero and one." But what we found is that, in addition the manual heuristic level, it's also very helpful when you're dealing with big models or with diverse models or just with data that you don't totally have a firm grasp on to be able to automate and use automated anomaly detection rather than set heuristics.

Dan: So machine learning monitoring, machine learning essentially turtles all the way down. I wonder if the long-term is essentially what that really looks like anyway, right? You have advanced models in a number of sort of lesser models monitoring in various levels because the complexity is challenging. In other words, there's an unknown unknown there, right? It's great if you know precisely what the boundaries should be around everything and you can define a bunch of manual characteristics. At the same time, maybe you don't know what those characteristics are over the long term time. That is in fact probably why you're studying the data, to come up with an interesting insight because you don't know all the variables that are in there. Or the variable that you are focusing on as a human being is very different than what the model is focusing on. In fact, we saw that in the earlier kind of [inaudible] statistics being applied to ham and spam, right? Being applied to spam email, right? In that, humans would write a rule in the early days. I'm old enough, unfortunately, to remember the pre-ML days where you'd write this rule and you go, "Oh, I'm a programmer. I can figure this out. Somebody wrote, 'Dear friend. How dare you.' And I'll write a rule that every time it says 'dear friend.'" And what you have is this towering inferno of things that the programmer focused on, whereas the base might be statistically be focusing on the code for the letter read in the HTML, right? Or some other weird something that we didn't understand and those are the things that are hardest to pick out. So I think that combination that you have of understanding the basic heuristics you can define but also understanding there's going to be things we don't. I think is incredibly important for people to understand.

Danny: I think spam is a really good example too because it's a perfect example of a constantly shifting distribution, right? Ham versus spam is an adversarial problem, you've got somebody spamming who wants their spam to get through the sensors and you've got somebody wanting to censor it. So the person who's developing these spam emails is constantly going to be poking at your black box, they're constantly going to be testing to figure out what is this model using to determine if something is spam or not. So you've got to make sure that your model can be monitored for its accuracy, for its performance, for changes in the input over time so that you can update it in response to changes in the incoming data. Honestly, spam is a great example of why ML observability and ML monitoring is so important.

Dan: The adversarial is a good example, I actually knew a fellow who was involved in email marketing in the early days of the net, was good friend who passed but was quite a genius at actually poking the black box. And figuring out those kinds of, "Okay, we've got all the systems; here's the state-of-the-art." And they would load up the various state-of-the-art and try to figure out, "Okay, what can we add into this thing to kind of get around it? Can we hide the graphic? Can we do this, whatever?" And it was a constant back-and-forth effort and it's very different from sort of probing kind of traditional software, as well, right? Whereas, when you're probing traditional software, you're maybe looking for a race condition or the library that's out of date that you can attack, right? It's a very different kind of thing where it's in many ways in machine learning you're looking for an attack on the logic, right, and attack in the sort of gaps within that logic itself. So it's quite fascinating to think about it from that standpoint. But let's talk a bit about the data-centric AI. So we talked about Andrew. We talked about him earlier, the illustrious Andrew, I feel half the time he's talking about stuff that seems pretty obvious when I think about it, but I realized it's not obvious for people. And one of the things that he seemed to hammer home--

Danny: I want to emphasize a useful mental model that I've picked up on recently, which is there are things that are intuitive which is it seems obvious when somebody says it to you. There are things that are obvious which are things that you actually already knew. Supply and demand is a super intuitive system, but before it was developed nobody had thought about, "Oh, yeah, maybe price gets set as a response to supply and demand." So yeah, I think a lot of things that Andrew says-- because he's so innovative or super intuitive because he's got such a good reading on the pulse of what's going on in the machine learning world. But there is a lot of really new things that he's saying that haven't bubbled up to the surface before he's the one bubbling them up.

Dan: And that's fair, right? Look, when you think about the history of innovation since [inaudible] MLOps innovators-- but when you really think about it-- I think about, actually, innovation a lot as an abstract when I do my futurism writing. And you're right that once a solution is there it's very obvious. So in other words, once everyone figures out how to run the four-minute mile, then all of a sudden, 20 more people that year run the four-minute mile. Whereas before that everybody thinks it's impossible. Everybody loves that example, that it's sort of-- a lot of people come close to running the four-minute mile, so it's sort of a hack one. But it is very true that once a solution comes out, it's usually so simple and elegant that it seems-- but Mr. Fuller talks about this a lot, right? That it would seem-- it seems so obvious at the time, but when you have all these people trying to figure out something that's never existed before, that's really hard, right? And I talked about this when I'm working on stuff within the AI Infrastructure Alliance, for instance, where if you go, "Look, let's define all the categories of machine learning." Right? So we'll start with a blank page, and you get 200 people together, everyone's going to flounder around for six months. But if you're able to go through the insightful process, put down a bunch of things, and then give them something to work with, then people are able to, "Oh, wait a minute. Yeah, no. This, this, this doesnt make sense. Let's define it this way." But starting from scratch is very hard.

Danny: It's really hard, yeah.

Dan: Once it exists, though, all of a sudden, boom, there's this exponential explosion of understanding, right? So I think that's a key observation in terms of what Andrew's saying. And one of the things that Andrew's been saying a lot has been this idea of data-centric AI. And again, this idea that you're going way too much into the model. And frankly, the algorithms are getting fantastic, right? We're kind of understanding these more and more. Transformers seem to be able to do everything, and there's a backlash now. They're like, "No, [ComNets?] can do everything. We're going to tweak them with this." And suddenly you're like, "They can be kind of universal and deal with texts and sound and everything else." But at the same time, what he also says is that most people don't have that kind of gigantic level of internet data, right? So AI really started-- it started in the universities and all that, but then when it started to get commercialized, it was the big FAANG companies and such that had these internet scale data sets where they're like, "Cool, let's throw some stuff in it." Right? And Google has that famous paper where they throw 50 algorithms at it and they all converge to the same sort of level of diminishing returns after 8 bazillion points of data. But most people don't have it. They have a smaller curated data set, and--

Danny: Hand-labeled, yeah

Dan: Hand-labeled, hand-understood, human intuition, working in concert with the AIs. So how do you think kind of whylogs fits in with that sort of data-centric approach, and how does it help with that company that's trying to build-- that's trying to instill all the knowledge of a bunch of really smart humans into a data set that the machine learning models can learn from in a programmatic way? How do you folks kind of assist with that?

Danny: Yeah, so I mean, we obviously think of ourselves as very much aligned to the DCAI story, right? Whylogs, our open-source library, allows you to generate data profiles, these statistical summaries of your data, so that you can understand your data better, so that you can monitor your data pipelines and your machine learning system in production by looking at the inputs and the outputs and the performance. And I mean, Andrew Ng thinks that we're [laughter] very much aligned with this vision too, which is why he funded us in this last [inaudible] thing that you were bringing up. So yeah, to me, WhyLabs is a very obvious compliment to any company that's trying to do DCAI, especially when they get to the production stage, right? Because we really understand that when it comes to monitoring your model, when it comes to monitoring your machine learning system as a whole, yeah, you need to look at the performance of the model. You need to look at the performance of the computers that are running the model. But ultimately, you need to have visibility into the entire data pipeline, which means logging data all throughout the process rather than just treating monitoring and observability as something that happens all the way at the end of the process. Which is, historically, how people have thought about monitoring within the ML space, and to be honest, even how I talked about monitoring in ML when I was first getting into MLOps before it was even a word a few years ago. So yeah, so to me, our vision of what does it mean to get observability into an AI system is ultimately very tied to the DCAI approach because we fundamentally believe that you need to be logging your data in order to be able to debug problems with your ML system.

Dan: Yeah, great. And that kind of goes to the heart of the difference between sort of DevOps and MLOps, right? Whereas DevOps essentially says, "Cool." The data is secondary. The programmer is designing the logic, right? So if I design a program where there's a login script, they only touch the data as a secondary thing, right? Maybe they get the username and password to log you in but I as the programmer write all of the logic. In machine learning, there are all these little steps where the data comes in and then the algorithm is learning essentially from that data and making decisions on it. The data becomes primary and there's this shift. So exactly what you're saying where there's-- at the end of it, we only care about the end state of the code. Is the code working? Is it responding within an X number of milliseconds? And you still need that in the machine learning space. You still want to know if the inference has slowed down.

You still need your SLIs. But there's so much more to it and that is understanding like-- but all the data that's feeding that training and that learning over time, it's still good. So let's talk about-- we're sort of getting to the end of our wonderful session here but-- we've talked a lot about kind of history of machine observability and monitoring. We've talked about kind of the current state of the art and-- let's talk a bit about-- just maybe finishing off with this. Where is it going? What is it developing to? What does an ideal system look like? Even if it doesn't fully exist at this point. What are we going to get to in 5, 10, 20 years? Maybe you don't want to project that far ahead but I love it when we get that far ahead. But what does an ideal system look like for people running a massive number of models in production, training, we're talking, thousands of inferences. Even tens of thousands, and we're talking 10 or 20 years, ambient intelligence. What does this look like kind of monitoring and dealing with that?

Danny: Yeah. Great question. So this is something that we spent a lot of time thinking about and talking about at WhyLabs. You mentioned our CEO Alessya. I mean she's a great thought leader in the space. She was talking about AI observability before anybody else was. Back when everybody else was still using ML monitoring and just kind of this baseline, "Yeah we should just monitor the performance of a model," she was already going like, "Well, no. We need to do more than that. We need to understand, okay, something's gone wrong. How do I know how to fix the problem?" So we're continuing to look ahead. And honestly, a lot of what looking ahead means is just looking over the shoulder of our neighbors in DevOps and seeing what are they doing? What's happening in that world?

And one of the things that we see is it's kind of still a gap in ML Ops is a lot of things are still very manual, right? You go in and you check your AI observability platform. Maybe it's WhyLabs, maybe it's somebody else. You go in and you do some manual debugging to figure out what's wrong. But in DevOps, CI/CD is like a pretty automated process where like if an SRE is getting involved, something's really gone wrong, right? In DevOps you've got a lot of automation set up where, in response to certain pipeline events, you have system rollbacks or patching or all of these different things that could happen. So where we see our platform going, where we see the space as a whole going, is much more in the automation direction, right? Not the human-in-the-loop manual change. But automating the process like auto MLOps or some people are calling it continual learning. Getting in there and saying, yeah, when something's wrong with the system, we should be automating as much as possible of fixing the system. So you were talking about turtles all the way down. I was talking about machine learning models, monitoring machine learning models. Taking it to the next step, what we need to do is automate the processes that we have in place to repair problems in the system because, ultimately, what we do as technologists is try to automate things. And constantly, we're trying to automate ourselves out of jobs. And the next job that we should be trying to automate ourselves out of, or at least minimize the amount of manual work that we have to do, is repairing a pipeline, is retraining a model. A lot of that stuff should be getting solved through more automated systems.

Dan: Automated interventions, right, in other words-- I talked about this a lot with the sort of AI red teams concept which is now sort of taking off and was kind of crazy when I was talking about it was the idea that you would have, "Okay. I need unit tests, essentially, for these types of things," and those unit tests might have an action associated with them, right? In other words, maybe it needs to go send inference to an older version of the model, or maybe it needs to pause the pipeline, or maybe it needs to alert a system, or all three of those things need to happen. There could be a number of things that sort of remediate those different issues. And you even see something like that kind of when you talk about the DevOps side. I remember when I was working at Red Hat, they had come and taken their Ansible platform and turned it into a SaaS. And half the time you'd run a report constantly on our system and then attempt to remediate a certain level of those problems, right? In other words, the system is down. Try to restart Apache a couple of times and if it's related to this bug, alert the administrator that they probably want to update so that it stops crashing, right? These kinds of things, I think we can get to in ML. But how long do you think it takes before we get there? I feel like in many ways, we're still kind of tiptoeing around it. People don't realize, in some respects, how early this space is. There's some amazing breakthroughs that have happened in a few years. But how long does it take? I mean look, I always know, and know when I say this, that the hardest thing about futurism is actually pinning down an actual number.

Danny: A number of [inaudible]. [We're going to have?] flying cars, at some point.

Dan: Yeah. Yeah. Yeah. At some point, we're going to have flying cars, but plus or minus 50 years. But really, how long do you think it takes to kind of get to at least some level of automation in this space before we get there?

Danny: So from our side, WhyLabs is already providing some level of automation to our customers, right? We're already solving one of the more trivial versions of this, which is simply setting up a rule system that if you've got model performance degradation caused by data drift, we can send an alert to you that you can receive and then automatically retrain and redeploy your model, right? So if you got your data pipeline set up with, I don't know, SageMaker or TeachableHub or UbiOps, whoever you're using to deploy the model, whoever you're using to train the model, if you can put that together in a pipeline with something like ZenML, then we can actually trigger a call to that system in order to retrain and redeploy a model. Now I say this as like, "This is what we have now," and like, "This is cool that we've got this level of automation." But as I've been talking about, MLOps is a lot more than just retrain your model, right? I mean that's some of the best-case scenario that you've got, baseline, ground truth data that you can use right away to retrain your model. If you don't have that, then you've got to label your data which is why we're partnered with Superb and why you can use cool platforms like Snorkel or Scale AI in order to get more data. But we can trigger the process of getting more data.

But even more than that, a lot of the problems that arise in ML systems, as I talked about at the beginning of the call, is not just about data drift. Everyone's talking about data drift right now because it's cool and fun and statistics are awesome. Yeah, but there's a lot of engine mirroring stuff that goes wrong upstream. So being able to automate repairing the data pipeline upstream, I think that's going to be kind of the next frontier. I think we're going to have some version of that within the next couple of years. I think we'll have a pretty good version of it within the next five years for sure. And when I say, "We," I mean I think the industry is going to catch up with WhyLabs, and we're going to need to figure out how to keep innovating beyond what we're thinking about right now.

Dan: Innovating's the dilemma. Everyone's got to keep innovating to survive, so. All right. Well, let's start to wrap it up here in, I would say, one sort of rapid-fire question. Are you reading anything awesome right now that you want to share with the team? What's on your Kindle? Or maybe you're a full, real book person. And I've run across few of them. Anything you want to share with the world, that's awesome, man.

Danny: Yeah. I'm actually an everything-reading person. So I have a book on my phone, I've got a physical book that I'm reading, and I've got an audiobook that I listen to. So my audiobook is The Idiot by Dostoevsky. My mom bought it for me. I don't think she meant anything by the title, but [it felt like it?]. And then other than that, I've been on a big climate science fiction kick, so I'm reading The Ministry for the Future by Kim Stanley Robinson who's a pretty prominent sci-fi author. And then I'm also reading Termination Shock by Neal Stephenson. And what's interesting is drawing this parallel between these two prolific, super impactful modern sci-fi authors, both of whom I adore and have read almost everything that they've published, and seeing them both thinking about the same issues that are going to be facing humanity: global warming, climate change, etc.

Dan: Amazing. I've been in the exact opposite direction with my sci-fi reading: gigantic space operas that have nothing to do with reality. So I just finished the Peter F. Hamilton's Salvation series which is just mind-blowingly crazy aliens with time weapons and a bunch of stuff. And then I've been blazing through The Expanse, and that's been fantastic. So they're pure escapism and politics and human nature, but well beyond sort of [the immediate day?]. But I think both are what makes sci-fi great, that gigantic escapism and also just being able to comment on the present, almost. The sci-fi is almost really about the present in a dressed-up form. So that's it. I appreciate--

Danny: I feel like The Expanse, you're-- I don't know how much escaping you're doing. Everything that they're talking about is so relevant to us. [laughter] I mean, they dress it up with the aliens and the gateways to other planets, but in actuality, what's happening there is a very human conflict between a lot of different human drives.

Dan: Yes. And they get that a lot. They really put in an amazing amount of stuff about the nature of human nature and our ability to keep being the most beautiful kind of creatures in the world and also the most violent and stupid ones too. I love that level of it where they're able to take this beautiful kind of escapist space opera and kind of layer in very important themes about the nature of humanity. So awesome. It's always good to end on science fiction. And I appreciate you joining us today. It was a lot of fun, and I think we had an amazing conversation. So thanks for being on the show.

Danny: Yeah. Thanks for having me, Dan. This was really cool to get to flesh out these concepts with you, and I hope your listeners get value from listening to us.

Dan: I'm sure that they will. And that's why they're tuning in every week, to understand where this whole industry is going. So thanks again for your time and it's most appreciated. [music]