I have a pipeline that copies data from source (dynamics) to SQL server datawarehouse. There is a ForEach activity which iterates over the list of all the tables and in ADF copy activity the data is copied. Also, the data copy is incremental and that is achieved by using SQL query to load the data incrementally.
However, sometimes new columns are added to the source system but not yet exsist in the destination table. Right now my pipeline stops working and throws an error. Is there a way to skip the newly added columns of the source system in ADF?
2条答案
按热度按时间ogq8wdun1#
You can use the query option in the source and write the query to get the required columns in the select list from the source table.
Or you can edit the mapping in your copy activity and map only the required columns.
mzaanser2#
Another option is to use Data Flow instead of the Copy Activity. Set the Source and Sink but leave the Mapping blank. It will copy any columns whose names match exactly and ignore any others.
Be aware that this may increase your pipeline cost.