80% of data is unstructured
So why do most AI/ML tools only handle structured data?
Pachyderm’s automated versioning and data-driven pipelines easily scale to petabytes of video, audio, text and genomics data.
Unstructured data is anything that doesn’t easily fit into a database or the rows and columns of a spreadsheet. That includes everything from videos for film and television, music and voice audio, large reams of text like a novel, and biotech driven data like genetics.
There are four key categories of unstructured data
Learn about how Pachyderm unlocks the power of your data in video / audio, imagery, text, and biosciences / health.
Video / Audio
Pachyderm’s parallel processing engine lets your team tear through huge audio and video datasets 10 to 100X faster than processing them linearly.
Whether you’re working in Media & Entertainment doing automatic subtitle generation, or dynamic ad insertion into online videos, or processing closed circuit video feeds, or track packages and cargo movements, or monitoring industrial machines on factory floors, Pachyderm can help you do it faster and more reliably.
Some good examples of unstructured data are:
- Streaming videos
- TV and movies
- Closed captioned video feeds
- Call center audio
- Meeting recordings
- Weather data
See how LogMeIn sped up their audio processing by 99%.
“The fact that we’re able to prepare our data so fast helped them to run a lot of training. Prior to using Pachyderm, we thought we’d never be able to execute those training sessions so fast. But because the data preparation process became so short, the research team was able to deliver much faster and create a lot of new models because of it.”
RTL Nederlands uses Pachyderm to do advanced machine learning on video. It detects dialogue so that inserting ads doesn’t cut a speaker off mid-sentence, does shot segmentation, facial and object recognition, and more, all in a modular set of pipelines. It also selects the ideal thumbnail to drive incredible user attention, because a thumbnail might be the only thing a viewer sees before deciding to click or keep going.
See Case Study
See how Fraunhofer uses Pachyderm to scale their cutting edge speech recognition experiments:
“Today, our workload runs in under a day due to incremental processing, thanks to Pachyderm. We were able to push out more models by training and serving them in parallel.”
Imagery
Images challenge most AI/ML platforms because you’re dealing with lots of little files or big binary files. Shoehorning images into databases meant for highly structured data simply doesn’t work well.
Pachyderm lets you rip through big stores of images fast, feeding them to parallelized workers, so you can do everything from finding hidden streets and buildings in satellite imagery, to spotting pedestrians in autonomous vehicle vision systems, to facial recognition for security.
Some good examples of unstructured data are:
- Satellite imagery
- Photographs
- Medical imagery such as CT
- MRI scans
- Radiology pictures
- Microscope images
- Astronomy images
RTL Nederlands, part of Europe’s largest broadcast group, wanted to use artificial intelligence (AI) to make video content more valuable and discoverable for millions of subscribers. Pachyderm delivered the data-driven machine learning (ML) automation, scale and reproducibility the team needed to work with massive amounts of unstructured video data.
See Case Study
Defense and Logistics Support
A major government contractor uses Pachyderm’s robust image processing pipelines on high resolution satellite imagery to track airfield and vehicle movements at bases around the world. Increasingly, modern militaries need the most up-to-date intelligence from a massive amount of data sources. They need a way to fuse all that data together into a comprehensive picture of logistics and that’s where Pachyderm helps support the democratic defense industries of the world today.
Home, Building and Real Estate Support
A top-tier commercial and residential real-estate support company wanted to help homeowners and commercial real estate developers make highly detailed damage assessments and see what upgrades would look like before they made a single change. But their data science teams had multiple overlapping projects, all with duplicate data scattered across.
Text
We’ve known how to deal with structured text for decades, so it’s no surprise that most companies focus on the tried and true database as their backend. But for the tremendous amounts of unstructured text pouring into datacenters, everything from financial reports, to chat logs, to Tweets and Slack posts, databases just don’t work very well.
Pachyderm’s file system based approach lets you deal with the surge of new unstructured text so you can do next-generation NLP, business analytics, legal document analysis and more.
Some good examples of unstructured data are:
- Novels
- Wikipedia Pages
- Tweets
- Slack
- Reddit and Discord posts
- Call center logs
- Meeting minutes
- Legal contracts
See how LivePerson scaled out processing call center audio to feed their NLP models doing transcription and sentiment analysis with Pachyderm:
“The difference was an order of magnitude faster… If it took 10 hours on the old system then it would only take an hour with Pachyderm” George Bonev, PhD – Machine Learning Engineer, LivePerson”
Insurance
A best-in class insurance provider uses Pachyderm’s significant text processing skills to handle health record reports coming from millions of providers so they can help doctors deliver better treatments.
Biosciences / Health
Healthcare, biotech and pharmaceutical companies looking to leverage the next generation of machine learning to improve their understanding of genetics look to Pachyderm to unlock hidden insights in the universal code of organic life.
Whether you’re doing advanced drug discovery, risk analysis for trials, repurposing existing drugs, looking for new targets for existing drugs, Pachyderm can help you dramatically speed up your data pipelines and get drugs and cutting edge-therapies to market faster and more successfully.
Some good examples of unstructured data are:
- Health reports
- Unstructured text from pathology
- Radiology and doctor’s note taking
- Clinical reports
- DNA data
- Research
- Published papers
See how Epona Data Science made their model throughput effectively continuous. Every single model they have and every sample, especially genetics samples, runs through the pipeline, gets tested and uploaded to the website in minutes.
See Case Study
See how Agbiome cuts down on manual steps and automatically processes their genetics data with speed and agility.
“Pachyderm helps us convert our existing data science pipelines from manually managed scripts to scalable, repeatable end-to-end workflows; enabling us to focus more on developing transformative technology to drive agriculture forward instead of wrangling infrastructure.”
Wrangling unstructured data easily and effectively
Database driven platforms simply don’t provide the right backend to manage the massive amounts of unstructured data your machine learning teams are struggling with right now. To truly unlock the power of your data you need an AI/ML platform that treats unstructured data as a first-class citizen, not an afterthought.
Pachyderm delivers a number of essential features for wrangling unstructured data easily and effectively:
- Work directly with any file type in your code, no schema or complex queries required
- Scales to petabytes of information with highly parallel processing with no extra code, and incremental processing of only new or changed data
- Track every version of your code, models and data with automatic versioning and lineage tracking
- Powerful deduplication of data with content-based chunking keeps storage costs low regardless of the number or size of files
- Supports any tool, framework or language including Python, R, Rust, Java, C++ or BASH
- Runs on any cloud or on-prem environment using standard object stores like S3, Azure Blob Storage, and Google Cloud Storage
Contact us to learn more about how Pachyderm handle Unstructured Data
Database driven platforms simply don’t provide the right backend to manage the massive amounts of unstructured data your machine learning teams are struggling with right now. To truly unlock the power of your data you need an AI/ML platform that treats unstructured data as a first-class citizen, not an afterthought.
