azure 通过API获取记录并批量发送- DataFactory

zxlwwiss 于 2023-04-22 发布在其他

关注(0)|答案(1)|浏览(175)

我必须将数据从sql server中一个包含65023记录的表更新为post类型API，该api在每个批处理的主体中包含100条记录，端点为https://api.local/batch/update，主体包含一批100条记录，主体如下

{ "inputs": [
     {
     "id": "{id[1]}",
       "properties": 
       {
         "field1": "{field1}",
 "field2": "{field2}"
       }
     },
     {
     "id": "{id[2]}",
       "properties": 
       {
       "field1": "{field1}",
 "field2": "{field2}"
       }
     }
....
]
}

其中id，field1，field2分别对应id，field1，field2列。请求体中不能发送超过100条记录，必须发送所有65023记录。

Azure

来源：https://stackoverflow.com/questions/75995980/get-records-and-send-them-in-batches-by-an-api-datafactory

1条答案

按热度按时间

ppcbkaq51#

如果您想通过post方式将数据从SQL server批量更新到Rest API，您需要设置writeBatchSize选项来批量写入特定数量的数据，并根据您的需求设置为100**。**

首先选择您的源作为SQL数据库和表。
然后选择sink作为rest API，并使用post方法给您的Api，并以特定数量的批次写入数据，并根据您的要求将writeBatchSize设置为100。
以下是我的Pipeline JSON，供大家参考：

{
    "name": "pipeline4",
    "properties": {
        "activities": [
            {
                "name": "Copy data1",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "AzureSqlSource",
                        "queryTimeout": "02:00:00",
                        "partitionOption": "None"
                    },
                    "sink": {
                        "type": "RestSink",
                        "httpRequestTimeout": "00:01:40",
                        "requestInterval": 10,
                        "requestMethod": "POST",
                        "writeBatchSize": "100",
                        "httpCompressionType": "none"
                    },
                    "enableStaging": false
                },
                "inputs": [
                    {
                        "referenceName": "AzureSqlTable1",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "RestResource1",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": []
    }
}

**参考：**REST as a Sink

赞(0）回复(0）举报 2023-04-22

我来回答

azure 通过API获取记录并批量发送- DataFactory

1条答案

相关问题

热门标签

最新问答