The MLOps Ecosystem

When it comes to MLOps, no one tool can do it all. It takes an array of best-of-breed tools to truly automate the entire ML lifecycle. That’s why we’re proud founding members of the AI Infrastructure Alliance (AIIA) a global organization of more than 50 of the most cutting edge companies in AI/ML, working together to build seamless integration points and clean APIs for machine learning.

The Machine Learning Lifecycle

Exp Train Shape Bottom Line Shape
  • Prepare

    Data, Collection, Preparation, Labeling

    Straight Line Shape mobile line mobile line
  • Experiment

    Exploration, Development, Feature, Engineering

    mobile line mobile line
  • Train

    ML Model Training, Evaluation/Selection, Monitoring

    mobile line
  • Deploy

    Model Deployment & Serving, Inference Monitoring

    Straight Line Shape
mlops tusks mlops hero shapes left mlops hero shapes right

Articles

  • The AI Infrastructure Alliance Wants to Build a ‘Canonical Stack’

    Read Article newstack
  • Band of AI startups launch ‘rebel alliance’ for interoperability

    Read Article venturebeat

The Machine Learning Tech Stack

The diagram below illustrates where Pachyderm and other AIIA members fit into the ML Tech Stack

Download the PDF
NOTEBOOKS CODE REPO DATA ENGINEERING ORCHESTRATION PIPELINE DATA ENGINEERING ENGINE EXTERNAL DATA SOURCES SYNTHETIC DATA LABELING MODEL TESTING / VALIDATION ENGINE MODEL REPO EXPERIMENTATION ENGINE TRAINING ENGINE MONITORING FEATURE STORE METADATA STORE DATA FOUNDATION CLOUD OR ON-PREM INFRASTRUCTURE DEPLOYMENT ENGINE MODEL SECURITY ALERTING ENGINE LOGGING ENGINE SERVING ENGINE DATA SCIENCE EXPERIMENTATION PIPELINE EXPERIMENT TO SERVING SUCH AS GIT RBAC ROLE BASED ACCESS CONTROL GENERATION/ AUGMENTATION USE CASE DEPENDENT INGESTION TO DEPLOYMENT EXPERIMENT TO SERVING DATA INGESTION AND TRANSFORMATION STAGE USE CASE DEPENDENT PRIMARILY FOR STRUCTURED DATA VERSIONING DATA LAKE WITH LINEAGE TRACKING INGESTION TO SERVING (SUCH AS KUBERNETES/IAAS/OBJECT STORE/OS) EXPERIMENTS AND TESTING TRAINING AND TUNING DEPLOY TO PRODUCTION MAKE PREDICTIONS IN APPS DASHBOARDS

Integrations

Pachyderm integrates with a wide variety of solutions in the ML Tech Stack, and provides sample code and examples for the following partners.

  • From ingestion to serving

    Deploying Seldon and Pachyderm together let’s you pull in data from anywhere, build complex models and push them to production with ease.

    Read Article
    plus
    seldon
  • Labeled data is great but versioned, labeled data is awesome.

    Now you can label all the data in your machine learning pipelines while keeping immutable copies as your labels continually change.

    Watch Video
    plus
    label studios
  • Experimentation and data engineering unified

    Uniting ClearML’s comprehensive experimentation and visualization with Pachyderm’s data orchestration engine delivers a complete ML experience.

    Read Article
    plus
    ClearML Logo

Other Partners

  • UbiOps Logo
  • RedHat Logo
  • Neu.ro Logo
  • YData Logo
  • Arize Logo
  • superb ai

Stories from the AIIA Ecosystem

  • The AI Infrastructure Alliance and the Evolution of the Canonical Stack for Machine Learning

    Read Article
  • AI/ML startups align to build a canonical stack and compete with the incumbents

    Read Article
  • The Future of ML and AI Infrastructure and Ethics with Dan Jeffries (Pachyderm, AI Infrastructure Alliance)

    Read Article
  • Seldon and Pachyderm: Two Foundational Pieces of the Machine Learning Loop

    Read Article
  • Fiddler on AIIA

    Read Article
  • Time is right for AI Infrastructure Alliance to Define Better Rules

    Read Article
  • Spell ML on AIIA

    Read Article
  • TerminusDB on AIIA

    Read Article
  • Bloomberg Press release

    Read Article