We’re excited to announce the last major release for the 1.x version of Pachyderm before the big release of Pachyderm 2.0.
Pachyderm 2.0 is well underway and expected to release this summer with lots of tremendous improvements. The Pachyderm 1.x line will continue to receive minor updates to address bugs or implement security patches until 2022.
What’s included in Pachyderm 1.13?
- Introducing the Pachyderm Innovators Program
- Autoscaling for Standby pipelines
- Helm Chart
- New Activation Process and Scaling Limits
Install/Upgrade to Pachyderm 1.13
Pachyderm Innovators Program
It seems like everyday someone joins our Slack channel or asks a question in the MLOps community Slack about the best way to implement Pachyderm. We love that, and we love all the cool ways Pachyderm is being used to solve some of today’s biggest AI/ML challenges. That’s why we’re launching a new springboard program for all those pioneers out there trying to accomplish big dreams with machine learning.
Starting on April 5th, startups, non-profits, and Proof-Of-Concept projects at for-profit companies can apply for free or heavily discounted Pachyderm Enterprise licenses. We want to make you successful and the whole team here at Pachyderm are ready and willing to help. All you need to do is schedule some time with us.
Who can apply for the Innovators program?
- Non-profits (those who have a 501(c) or can show proof of tax exempt status).
- Startups who have less than 100 employees or have been operating for less than 4 years.
- Full-time Students or part-time in a post-graduate program (Master, P.H.D).
- Proof of Concept technical directors – If you have an idea and need to prove it before you go Enterprise, we want to help.
If you meet the requirements above and want to apply, email us at innovator_program@pachyderm.com.
Important changes to the Pachyderm Community Edition
As of 1.13, Pachyderm Community Edition limits teams to 16 concurrent pipelines. To run more than 16 or to set parallelism to greater than 8, you will need an active token to access Pachyderm’s enterprise features. You can get a FREE 30 day token by registering at Trial, or by getting in touch with us directly.
Users running previous versions of Pachyderm Community Edition won’t be affected. Those who upgrade to 1.13 and already have at-scale workflows (>16 pipelines or parallelism >8) will continue to run, but you won’t be able to scale further without an active Pachyderm Enterprise Token.
We want to establish closer relationships with our users so we can continue to provide an even higher level service to our community of users . Please don’t hesitate to reach out to us if you have any questions or concerns about the Community Edition changes.
Autoscaling for Standby Pipelines
One of Pachyderm’s best features is our ability to use parallel processing to scale computational workloads horizontally across an entire cluster rather than being bound to a single machine. As of 1.13, our parallel processing capabilities become more dynamic with the introduction of autoscaling.
Prior to 1.13, a pipeline with a parallelism of three would always spin up three worker nodes at the start of a job, regardless of the amount of data you’ve got to process. Now with autoscaling enabled, a pipeline will only spin up the number of workers required to process the job, up to a specified maximum. This allows your jobs to scale up for large-scale workloads, while conserving resources (and saving money!) when processing smaller workloads.
Autoscaling pipelines will automatically have standby enabled as well. When a job finishes, the pipeline will go into standby mode and all worker nodes will shut down.
Learn how to configure autoscaling
New reprocess_spec Option for Pipelines
Another great benefit built into Pachyderm by default is the ability to skip over unchanged data. However, you might need to alter this behavior for specific use cases and force the reprocessing of all of your datums systematically. This is especially useful when your pipeline makes an external call to other resources, such as a deployment or triggering an external pipeline system.
Beta Release Of Pachyderm Helm Chart
This one’s been a long time coming but we’re happy to announce that there’s now an officially supported Helm chart for Pachyderm. Users can leverage this asset to shorten deploy times significantly as well as start to standardized how Pachyderm should be deployed within your org.
Take a look a the Beta Helm Chart
Pachyderm 1.9 and 1.10 End-Of-Life(EOL)
As we ramp up for Pachyderm 2.0 it’s time to send out some of our older releases: 1.10 (2020) & 1.9 (2019) out to pasture so we can focus on meeting our deadlines. Our Enterprise customers can contact us anytime, however all community support will be directed towards upgrading to a supported version of Pachyderm.
You can find out more info about the Pachyderm Product Support Lifecycle here.
