Introducing Pachyderm 2.8

We are excited to announce the general availability of Pachyderm 2.8! This release includes a multitude of exciting features and enhancements including Global Pipeline Configurations, a more convenient Console experience, and IntelliSense support for Pipelines. This release focuses on bringing our users a powerful, more intuitive, and streamlined experience across the whole Pachyderm platform. 

Global Pipeline Configuration

Users are now able to efficiently bulk create and manage a massive scale of Pachyderm pipelines with the new Global Pipeline Configuration feature. This new feature introduces the concept of Cluster default configurations. When users set default parameters on the Cluster default configuration, pipelines inherit these parameters. Pipeline creation and updates are streamlined, allowing users to iterate faster and reduce risk.

• Scale Pachyderm pipelines with new Global Pipeline Configurations 
• Set default parameters for pipelines on the cluster level 
• Pipelines can inherit these parameters, saving time and effort 

Users can create a base pipeline configuration using Global Pipeline Configuration, allowing for better organization, maintenance, and consistency when creating and updating pipelines. This also unlocks rapid pipeline deployment as users can quickly create pipelines. Cluster default configuration reduces the complexity for new users, who can quickly onboard, and for existing users, who can better manage the growing number of pipelines. 

• Consistent, organized, and easily-maintained pipelines with Global Pipeline Configurations 
• Quickly deploy and iterate new Pipelines using Cluster default configuration 
• Simplify Pipeline creation for new users getting started and existing users managing Pipelines at scale 

Find out more about Global cluster defaults in our docs. 

Do Even More in the Web Interface

Users can now do more than ever without having to leave Console in your web browser. Create and update pipelines in Console, using VS Code IntelliSense to help you write your code. Users can also configure cluster-level pipeline defaults to apply common settings to all your pipelines. Teams can see how projects are connected in the DAG viewer, and re-run jobs with a single click. Users can filter jobs and commits by global ID and see the details in the sidebar. And you can preview more file types in Console, including images, videos, audio, programming languages, markdown, HTML, XML, text, and CSV.

• Create and update pipelines in Console, powered by VS Code IntelliSense 
• Configure cluster-level pipeline defaults using Global Pipeline Configuration 
• Connections between projects are now visible in the DAG viewer 
• A new action is available to re-run jobs with a single click 
• Global ID filters now show relevant job and commit info in the sidebar 
• More file types can be previewed in Console
  • 15 image formats 
  • 24 programming languages 
  • 17 video file extensions
  • 10 audio formats 
  • Markdown, HTML, XML, text, and CSV all have improved support

IntelliSense Support for Pipelines

Pachyderm 2.8 introduces IntelliSense support for Pipeline specifications, a feature that will boost user’s productivity and accuracy. With IntelliSense, users receive real-time auto-completion and feedback as they draft pipeline specifications, reducing the risk of manual input errors. Users can also use IntelliSense with VSCode, which supports both JSON and YAML formats for your pipeline specifications. IntelliSense is based on a JSON Schema file, which means that you can customize it to work with your preferred editors, linters, and other tools if you are familiar with JSON schema support. Finally, we have integrated IntelliSense into Console, so users can benefit from the streamlined specification process within the Console code editing interface when you create or update a pipeline. 

• Real-time auto-completion and suggestions, aiding users in swiftly defining pipelines while minimizing errors 
• Pachyderm’s IntelliSense is optimized to work seamlessly with both JSON and YAML formats 
• Integration with your preferred editors, linters, and other tools, based on a JSON Schema file 

For detailed installation and usage instructions. More details are provided in our docs.

Pachyderm SDK Tutorials and Examples

With the previous release of Pachyderm 2.7.0, we enthusiastically announced our new official Python client, the Pachyderm SDK.  

If you are as excited as we are about the new Pachyderm SDK, we encourage you to check out some of our tutorials and examples. Learn how to create your first project here or walk through examples including Breast Cancer Detection, Distributed Image Processing, or Spout Pipelines. Feedback is always encouraged and welcome!

Summary

Pachyderm 2.8 offers users the most performant and intuitive version of Pachyderm to date. With these new features and enhancements, customers will experience greater ease of use and flexibility when working across the platform, from Pipelines to the Console. For a comprehensive list of features, read the release notes. 

Are you new to Pachyderm and searching for a data pipelining or MLOps solution? Take the next step and schedule a demo tailored to your environment.