Transform your data pipeline
Learn how companies around the world are using Pachyderm to automate complex pipelines at scale.
Request a DemoIn the early days of the pandemic, when uncertainty reigned about the best policies to protect people around the world, Riskthinking.AI’s team of veteran forecasters went to work. As 2020 went by, we saw machine learning deployed in everything from discovering top vaccine candidates, to tracking super spreader events, but the Riskthinking.AI team took a different approach. Could they predict the best way to implement societal-level responses like lockdowns with minimal damage to daily life and the economy?
After getting the call from the Canadian government, their big data and financial modeling experts got to work assessing the economic impact of major pandemic policies. They crafted the CovidWisdom app, a dashboard that tracked the different approaches to shutting down. Their models showed the economic challenges of the pandemic in bold detail. It allowed city and state officials to visualize what would happen if they implemented no restrictions at all, all the way to complete lockdowns. Even better, they were able to model whether targeted lockdowns of specific types of places and events could work. That let world leaders see what would happen if they decided to shut down concerts, bars and gyms but nothing else or if they targeted only concert halls and bars, as well as dozens of other variations.
Riskthinking.AI was in the early phases of ramping up their internal AI infrastructure when they took on the CovidWisdom project. Their team specializes in measuring the financial risk of climate change. Companies and governments work with them to figure out how to make the best decisions when it comes to uncertain futures. Should the electric company rebuild transformers in the exact same spots that caused forest fires in the past or should they put them in a different configuration to reduce the chance of starting another fire in the future? How quickly can a company ramp up a solar farm and where should they put it?
Understanding complex and dynamic systems requires a level of predictability and reproducibility. We found that with Pachyderm we were able to get to results faster and we had the confidence that we could always recreate them.
But early on, their team realized they had experts in predicting the future but not in building AI architecture. They had data scientists working on laptops, pulling and pushing data over VPNs to remote work spots, and even building their own Docker containers. They wanted to find a way to let data scientists do what they do best, build models and make predictions, while data engineers build them a rock solid platform to do that work. The less data scientists have to deal with data wrangling, versioning data, and pushing the pulling data to pipelines the more they can focus on the future. They needed to move from ad hoc to MLOps and that’s where Pachyderm came into the picture.
Forecasting uncertain futures is a tricky business. There’s lots of experimentation and trying out different ideas and looking at the data in different ways. Pachyderm let Riskthinking.AI’s scientists focus on the complexity of models rather than the complexity of figuring out which model was trained on which version of the dataset. It gave them the foundation to work with data and deploy any ML tool they wanted inside their machine learning loop.
As Riskthinking.AI’s data scientists got more comfortable with pachctl and the command line, they used Pachyderm to run multiple models simultaneously and to visualize backtesting results with easy to understand images. The backtesting results showed how a particular model outperformed other models and helped data scientists to understand the model performance changes over time using Pachyderm’s powerful data lineage. The best performing model got automatically pushed to the application for the current day.
But those visualizations weren’t just for the data science team. They could also share their progress with non-technical or less technical stakeholders. That let everyone look at the models and understand how they were changing and getting better over time. That’s a key decision that any data science team can learn from because if you can’t demonstrate what you’re doing to a wider team, it’s hard for them to know the impact your team is making on the real world and the bottom line of the business.
In an uncertain world who couldn’t use a little more certainty?
As Riskthinking.AI ramps up their efforts to understand complex, dynamic systems they’re moving to tackle the growing threat of climate change. They want to capitalize on Pachyderm to process millions of data points about how our world is changing around us. That will allow companies and nations to make better and better decisions about how their choices today affect tomorrow.
We live in an uncertain world. Too often there are millions of variables we can’t see or account for easily. It’s a world of missing and hidden information and that’s where machine learning can make a big difference. It lets people make better predictions about the future by crunching through data points that human minds might struggle to see.
Of course, in the end, the world is a messy place and even the best models can’t make politicians make good choices. Some countries heeded the wisdom that advanced AI models showed about the pandemic and some choose to believe what they wanted to believe, but that doesn’t change the fact that AI modeling of dynamic systems is here to stay. As AI continues to develop in the coming years, expect more and more companies, cities and nations to turn to machine learning to drive their biggest decisions.
In an uncertain world who couldn’t use a little more certainty?
Learn how companies around the world are using Pachyderm to automate complex pipelines at scale.
Request a Demo