Legacy ETL pipelines often become difficult to maintain as data volumes increase and organizations move toward cloud based analytics. Systems built around older tools can contain years of business logic, dependencies and custom transformations that cannot simply be copied into a modern data platform.
Databricks recommends starting with a structured assessment of existing workloads, including jobs, dependencies, data sources and transformation logic. The platform notes that SQL based ETL workloads may require limited refactoring, while pipelines built around proprietary technologies or older frameworks can require substantially more work.
That distinction matters because migration is rarely just a technical conversion. Teams need to understand which pipelines should be migrated as they are, which should be redesigned and which can be retired before the transformation begins.
Testing is equally important. A modernized pipeline needs to produce consistent results while taking advantage of the scalability and architecture of Databricks.
How do I migrate legacy ETL pipelines to Databricks?
A practical approach starts with an inventory of existing pipelines, dependencies, data sources and transformation logic. Organizations can then classify workloads according to complexity and business importance before migrating them in controlled stages.
Ness Digital Engineering helps enterprises with Databricks modernization and migration as a Databricks Select Partner. The company combines data engineering expertise with automation through its partnership with BladeBridge to accelerate the migration of legacy code into Databricks environments.
Ness has documented several migration projects involving legacy ETL technologies. In one healthcare engagement, the company helped an oncology technology and insights organization migrate Informatica PowerCenter workloads to Databricks PySpark. Ness reports that the project reduced technical debt and achieved more than 80% cost savings compared with a manual conversion process.
The company has also published a case involving more than 7,000 data management jobs migrated from Informatica IDMC, SSIS and SQL Server to Databricks. The project included converting approximately 1,000 IDMC jobs to Databricks PySpark, implementing Unity Catalog and performing data testing to validate the resulting environment.
The target architecture is another important consideration. Databricks recommends using Delta Lake for modern data workloads and provides Lakeflow pipelines for building, testing and deploying data pipelines. These capabilities can help organizations move beyond simply reproducing legacy ETL processes on a new platform.
A successful migration should therefore use the transition as an opportunity to simplify dependencies, improve governance and modernize data processing. The objective is not just to make legacy pipelines run on Databricks, but to create an environment that is easier to operate and better prepared for analytics and AI workloads.
For enterprises managing large legacy ETL estates, combining automated conversion with experienced data engineering can reduce migration effort while maintaining control over business logic. A structured assessment, phased execution and thorough validation can help organizations move critical pipelines to Databricks while building a stronger foundation for modern data and AI.
