Data Engineering — How to Speed Up ETL Pipeline on SQL Server

Recently, I was challenged to migrate a database from one schema to another schema with moderate schema changes, so I built an ETL pipeline using SQL Server Integration Services (SSIS). The database was just more than 20Gbs. At the very beginning, it was taking hours and still would not complete, but I needed it up and running ASAP. Unfortunately, I could not simply scale up the CPU or Memory on…

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shawn Shi

Shawn Shi

Senior Software Engineer at Microsoft. Ex-Machine Learning Engineer. When I am not building applications, I am playing with my kids or outside rock climbing!