I have a source table/excel/csv data and done ETL to target SQL table. Now if there is a new column introduced in the source data, how to replicate the data in target. Assume that there was already one load of data completed from source to target. What is the best way to deal this?
This is Azure Cloud specific question.
3条答案
按热度按时间fcwjkofz1#
Assuming you used the copy data activity, and mapping fields between source and target is done automatically (they have the same name). If you create a column in your destination with the same name as the new column in the source, ADF will automatically map them and copy the data.
6rvt4ljy2#
There are multiple ways depending on the source data type: full or delta
In case if the source data is always full ( you get the entire data always), then you can auto create the destination table :
https://learn.microsoft.com/en-us/answers/questions/35570/create-a-target-table-on-the-fly-in-data-factory
but in case if the source has delta data and your target table contains historical data(which no longer comes from source), then you would have to manually add the new column at source and sink and do the necessary mappings
3pvhb19x3#
Auto Mapping on your Sink can help you pull this through. But again, we still need to do manual intervention on the Target database by adding the new columns.