The MLOps Ecosystem

When it comes to MLOps, no one tool can do it all. It takes an array of best-of-breed tools to truly automate the entire ML lifecycle. That’s why we’re proud founding members of the AI Infrastructure Alliance (AIIA) a global organization of more than 50 of the most cutting edge companies in AI/ML, working together to build seamless integration points and clean APIs for machine learning.

The Machine Learning Lifecycle

Exp Train Shape Bottom Line Shape

  • Prepare

    Data, Collection, Preparation, Labeling

    Straight Line Shape mobile line mobile line

  • Experiment

    Exploration, Development, Feature, Engineering

    mobile line mobile line

  • Train

    ML Model Training, Evaluation/Selection, Monitoring

    mobile line

  • Deploy

    Model Deployment & Serving, Inference Monitoring

    Straight Line Shape

The AI Infrastructure Alliance Wants to Build a ‘Canonical Stack’

Read Article

newstack

Band of AI startups launch ‘rebel alliance’ for interoperability

Read Article

venturebeat

The Machine Learning Tech Stack

Use the diagram below to see where Pachyderm and other AIIA members fit into the ML Tech Stack

NOTEBOOKS CODE REPO DATA ENGINEERING ORCHESTRATION PIPELINE DATA ENGINEERING ENGINE EXTERNAL DATA SOURCES SYNTHETIC DATA LABELING MODEL TESTING / VALIDATION ENGINE MODEL REPO EXPERIMENTATION ENGINE TRAINING ENGINE MONITORING FEATURE STORE METADATA STORE DATA FOUNDATION CLOUD OR ON-PREM INFRASTRUCTURE DEPLOYMENT ENGINE MODEL SECURITY ALERTING ENGINE LOGGING ENGINE SERVING ENGINE DATA SCIENCE EXPERIMENTATION PIPELINE EXPERIMENT TO SERVING SUCH AS GIT RBAC ROLE BASED ACCESS CONTROL GENERATION/ AUGMENTATION USE CASE DEPENDENT INGESTION TO DEPLOYMENT EXPERIMENT TO SERVING DATA INGESTION AND TRANSFORMATION STAGE USE CASE DEPENDENT PRIMARILY FOR STRUCTURED DATA VERSIONING DATA LAKE WITH LINEAGE TRACKING INGESTION TO SERVING (SUCH AS KUBERNETES/IAAS/OBJECT STORE/OS) EXPERIMENTS AND TESTING TRAINING AND TUNING DEPLOY TO PRODUCTION MAKE PREDICTIONS IN APPS DASHBOARDS

Integrations

Pachyderm integrates with a wide variety of solutions in the ML Tech Stack, and provides sample code and examples for the following partners.

Cloud Deployment in a Snap with Azure

Streamlined cloud deployment of Pachyderm on Azure for integrated ML data pipelines with automated scale and lineage.

Azure & Pachyderm Jupyter Notebook example

ML Lifecycle Management with Seldon

Deploying Seldon and Pachyderm together lets you pull in data from anywhere, build complex models and push them to production with ease.

Read the Integration Deep Dive

Version control for data labeling

Now, Label Studio and Pachyderm are integrated to make it super simple to automate data versioning and lineage for labeling datasets – never lose track of your labels or data.

Integration Video + Example

Experimentation and data engineering unified

Uniting ClearML’s comprehensive experimentation and visualization with Pachyderm’s data orchestration engine delivers a complete ML experience.

Read article

Other partners

See Pachyderm In Action

Watch a short 5-minute demo which outlines the product in action

Try Pachyderm Today

Request a Demo