Pachyderm is a team of passionate individuals who love all things data, open source, and ML/AI. Oh, and we also love infrastructure tools and building developer communities!
Pachyderm is an enterprise-grade, open source data science platform that makes explainable, repeatable, and scalable ML/AI a reality. Our platform brings together version control for data with the tools to build scalable end-to-end ML/AI pipelines while empowering users to use any language, framework, or tool they want. What makes Pachyderm a natural choice for data science teams is that they can iterate quickly and know that everything is tracked and 100% reproducible.
What would data analytics infrastructure (namely Hadoop) look like if we rebuilt it from scratch today? We think it would be containerized, modular, and easy enough for a single person to use while still being scalable enough for a whole company. Tools like Docker and Kubernetes provide the perfect building blocks for us revolutionize data infrastructure!
Pachyderm is “Git for Data Science.” We offer complete version control for data and give your data science team the same first-class development tools as software developers. Pachyderm is ideal for building machine learning pipelines and ETL workflows because we track every model/output directly to the raw input datasets that created it (aka: Provenance).
Since everything in Pachyderm is a container, data scientists can use any languages or libraries they want (e.g. Spark, R, Python, OpenCV, etc) without any additional infrastructure overhead.