Riskthinking.AI was in the early phases of ramping up their internal AI infrastructure when they took on the CovidWisdom project. Their team specializes in measuring the financial risk of climate change. Companies and governments work with them to figure out how to make the best decisions when it comes to uncertain futures. Should the electric company rebuild transformers in the exact same spots that caused forest fires in the past or should they put them in a different configuration to reduce the chance of starting another fire in the future? How quickly can a company ramp up a solar farm and where should they put it?
But early on, their team realized they had experts in predicting the future but not in building AI architecture. They had data scientists working on laptops, pulling and pushing data over VPNs to remote work spots, and even building their own Docker containers. They wanted to find a way to let data scientists do what they do best, build models and make predictions, while data engineers build them a rock solid platform to do that work. The less data scientists have to deal with data wrangling, versioning data, and pushing the pulling data to pipelines the more they can focus on the future. They needed to move from ad hoc to MLOps and that’s where Pachyderm came into the picture.
Forecasting uncertain futures is a tricky business. There’s lots of experimentation and trying out different ideas and looking at the data in different ways. Pachyderm let Riskthinking.AI’s scientists focus on the complexity of models rather than the complexity of figuring out which model was trained on which version of the dataset. It gave them the foundation to work with data and deploy any ML tool they wanted inside their machine learning loop.
As Riskthinking.AI’s data scientists got more comfortable with pachctl and the command line, they used Pachyderm to run multiple models simultaneously and to visualize backtesting results with easy to understand images. The backtesting results showed how a particular model outperformed other models and helped data scientists to understand the model performance changes over time using Pachyderm’s powerful data lineage. The best performing model got automatically pushed to the application for the current day.
But those visualizations weren’t just for the data science team. They could also share their progress with non-technical or less technical stakeholders. That let everyone look at the models and understand how they were changing and getting better over time. That’s a key decision that any data science team can learn from because if you can’t demonstrate what you’re doing to a wider team, it’s hard for them to know the impact your team is making on the real world and the bottom line of the business.
As Riskthinking.AI ramps up their efforts to understand complex, dynamic systems they’re moving to tackle the growing threat of climate change. They want to capitalize on Pachyderm to process millions of data points about how our world is changing around us. That will allow companies and nations to make better and better decisions about how their choices today affect tomorrow.
We live in an uncertain world. Too often there are millions of variables we can’t see or account for easily. It’s a world of missing and hidden information and that’s where machine learning can make a big difference. It lets people make better predictions about the future by crunching through data points that human minds might struggle to see.
Of course, in the end, the world is a messy place and even the best models can’t make politicians make good choices. Some countries heeded the wisdom that advanced AI models showed about the pandemic and some choose to believe what they wanted to believe, but that doesn’t change the fact that AI modeling of dynamic systems is here to stay. As AI continues to develop in the coming years, expect more and more companies, cities and nations to turn to machine learning to drive their biggest decisions.
In an uncertain world who couldn’t use a little more certainty?