Over the last five years, we’ve built PipelineDB into a business with minimal initial funding. As of recently, we’ve made the determination that with a relatively niche product, our business did not warrant seeking additional capital to fuel venture-scale growth, however. This realization placed us at a distinct crossroads — do we continue growing organically or leverage everything we’ve learned and join forces with a better-capitalized company to help the idea behind it to take root more broadly?
After much consideration, we decided on the latter and to join Confluent, whose founders are the original creators of Apache Kafka. Confluent is a company with an incredible amount of momentum. Its vision of a changing world in which data are fluid streams of events rather than static silos of records has always been perfectly aligned with ours. And after five thrilling years of going it alone, the time has come for us to continue our mission as part of a once-in-a-generation company that is reshaping the entire industry.
What does this mean for PipelineDB the product?
We will no longer offer the commercial extension of PipelineDB, and the open-source PipelineDB core will remain at
1.0.0. Critical bugs will still be fixed whenever possible. Any such maintenance will come from the PipelineDB open-source community and will not have any affiliation with Confluent. Official releases will remain available, as will the Apache-licensed PipelineDB codebase.
What does this mean for PipelineDB users?
We certainly understand the long-term implications of continuing to run a production database that will not be maintaining a regular release cycle. We want to be as helpful as possible in helping you think through longer-term migration plans to something with regular releases moving forward.
If you’re already using Kafka, running your continuous query workload with KSQL is a really good option. KSQL enables you to run continuous SQL queries on Kafka messages, sending the output to another stream/topic or materializing ongoing results in a persistent store, much like a PipelineDB continuous view. KSQL has gotten remarkably good since its initial release, and Confluent is making a significant investment in it.
If you’re based more on PostgreSQL than on Kafka, and have a workload for which your event history fits on disk, TimescaleDB is probably a good option for you. TimescaleDB is a PostgreSQL extension that provides advanced and highly performant automated partitioning on time-series data, enabling you to ingest raw data at very high throughput as well as query it efficiently. Their next release includes a feature called "Continuous Aggregates" that, similar to PipelineDB's continuous views, materializes results of views over raw data, but also stores aggregates over the entire history of your data.
There are many other viable options, but these are probably the top two for most of our users. If you’re considering something else, please don’t hesitate to reach out to us at email@example.com if you’d like any guidance on long-term workload migration.
Finally, we would like to end by expressing our sincerest gratitude to all of the PipelineDB employees, investors, advisors, users, and customers who have made these last few years possible. PipelineDB largely became what it is as a result of the excellent feedback and ongoing interactions we’ve had with so many of you. Every open-source community is different, and we’ve always felt lucky that our users have been such thoughtful, kind, patient, and clever people.
Working with all of you has been the greatest honor of our lives.
Derek and Jeff