Using Data Pipelines to Create Complex Maps
Woven Planet is building the safest mobility in the world. A subsidiary of Toyota, Woven Planet innovates and invests in new technologies, software, and business models that transform how we live, work and move. With a focus on automated driving, smart cities, robotics and more, Woven Planet builds on Toyota’s legacy of trust to deliver secure, connected, reliable, and sustainable mobility solutions for all.
Faster Data Processing with Parallel Pipelines
Toggles Easily Between Structured and Unstructured Data
Automatic Data Versioning for Reproducibility
Handles Petabyte-Scale Machine Learning Workloads
The ChallengeWoven Planet’s Automated Mapping team is building automotive-grade maps for use in the automated-driving vehicles of today and the autonomous-driving vehicles of tomorrow. Key to this effort is the use of aerial orthographic projection. While this kind of aerial- and satellite-derived data have long been used in the development of consumer-grade navigational maps, using these kinds of data to meet the rigorous requirements of automated driving—and to do so at continental scale—is an unprecedented challenge.
For automated driving applications, maps need a level of detail, accuracy and precision far beyond those of their consumer-grade counterparts. Developing this level of granularity requires processing voluminous amounts of data. In order to achieve this at scale, Woven Planet’s Automated Mapping team is pioneering new, cutting-edge machine learning solutions.
In selecting an orchestration system as part of this effort, the Automated Mapping team had two key priorities: First was ensuring their pipeline had maximum flexibility—the ability to scale to meet elastic workloads and to easily toggle between structured and unstructured datasets. And second was long-lived pipeline stability that would support continuous, region-based map updates—critical to maintaining map accuracy in a constantly evolving world.
Why Woven Planet Chose Pachyderm
Woven Planet’s Automated Mapping team had strong expertise in Kubernetes (K8s); and as Pachyderm employs the powerful K8s’s orchestration engine to rapidly scale up workers to deal with elastic workloads, it offered a solution that could be easily integrated into the Automated Mapping team’s workflow and pipeline.
Also supporting the Automated Mapping team’s decision was Pachyderm’s ability to handle petabytes of machine learning workloads, which made it well-suited for the next-generation processes Woven Planet’s Automated Mapping team is pioneering.
Woven Planet's Automated Mapping team uses Pachyderm for a range of processes that augment their ingestion-to-training pipeline. With data sourced from an array of satellite providers and databases, the Automated Mapping team adopts a unique approach to image recognition that divides images into various geolocations and sections. Pachyderm pipelines complement this approach, offering hundreds of pipelines that allow for parallel processing of different road features. This allows the Automated Mapping team to extract semantic information (i.e., what does a road feature mean for the vehicle) from the data far more efficiently than via a linear approach.
In addition to parallel processes, there is also a sequential coordination, with a waterfall-like cascade of different threads continually transforming the work of the pipelines that came before it. As one pipeline completes its work, processing different sections of the imagery at the same time, the results can be collated from their distributed workers before passing onto the next stage for new transformations. Combined with Kubernetes' tremendous scalability at the worker level, the Automated Mapping team’s use of Pachyderm speeds up time-to-value, helping Woven Planet deliver maps that have continental-scale coverage and always-on accuracy.
Lastly, the Woven Planet engineers have enjoyed working with the Pachyderm engineering and support teams, giving credence to the truism that a powerful platform is great but it’s only as good as the support that comes with it.
See Pachyderm In Action
Watch a short 5-minute demo which outlines the product in action