Data analytics firm Databricks has announced it is to open source its core declarative ETL framework, Apache Spark Declarative Pipelines, which it launched in 2022 as Delta Live Tables, making it available to the Apache Spark community.
The offering allows users to run it anywhere Apache Spark is supported, rather than just on the Databricks platform, reflecting the company’s commitment to open ecosystems.
Databricks’ key rival Snowflake recently launched Openflow, a data integration service for getting any data from any source into its platform, while Databricks’ offering is designed to go from source to usable data.
Databricks’ framework will be committed to the Apache Spark codebase in an upcoming release, with an unclear timeline.