NodeJS 发送多个HTTP请求

643ylb08  于 2023-08-04  发布在  Node.js
关注(0)|答案(2)|浏览(126)

我需要写一个程序来查找使用项目ID的项目信息。
API一次只接受一个项目,所以我只能对每个项目执行一个查询。API限制为5个同时请求。任何额外的结果都将给予HTTP 429错误。
如果有一个JavaScript对象,其中包含所有具有ID的项目
如何检索所有给定ID的信息,而不触发同时请求限制,并且不对已经看到的项目ID执行不必要的查询。

import fetch from "node-fetch";

let itemObject = [
  { itemName: "", itemID: "" },
  { itemName: "", itemID: "" },
  { itemName: "", itemID: "" },
  { itemName: "", itemID: "" },
];

async function sendIDRequests() {
  try {
    const response = await fetch("https://url/items/:ID", {
      headers: {
        Authorization: "",
      },
    });
    if (!response.ok) {
      throw new Error(`${response.status} ${response.statusText}`);
    }
    response
      .text()
      .then((res) => console.log(res))
      .catch((err) => {
        throw new Error(err);
      });
  } catch (error) {
    console.error(error);
  }
}

sendRequests()

字符串

nzrxty8p

nzrxty8p1#

我想到了两种方法。批处理和滑动窗口方法。批处理可能更容易,但使用滑动窗口将是更有效的实现。

Promise.all()批量处理

这种方法创建一批请求,最多可达指定的batchSize,并且只有在一批中的所有请求都完成后,才发出下一批请求。
您需要在这里添加一些错误处理,以防请求失败。

import fetch from "node-fetch";

// list of items that you might want to use to compose your URL (not actually used here)
let itemObject = [
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
];

(async () => {
    // number of concurrent requests in one batch
    const batchSize = 4;
    // request counter
    let curReq = 0;
    // as long as there are items in the list continue to form batches
    while (curReq < itemObject.length) {
        // a batch is either limited by the batch size or it is smaller than the batch size when there are less items required
        const end = itemObject.length < curReq + batchSize ? itemObject.length: curReq + batchSize;
        // we know the number of concurrent request so reserve memory for this
        const concurrentReq = new Array(batchSize);
        // issue one request for each item in the batch
        for (let index = curReq; index < end; index++) {
            concurrentReq.push(fetch("https://postman-echo.com/get"))
            console.log(`sending request ${curReq}...`)
            curReq++;
        }
        // wait until all promises are done or one promise is rejected
        await Promise.all(concurrentReq);
        console.log(`requests ${curReq - batchSize}-${curReq} done.`)
    }
})();

字符串
预期结果:

sending request 0...
sending request 1...
sending request 2...
sending request 3...
requests 0-4 done.
sending request 4...
sending request 5...
sending request 6...
sending request 7...
requests 4-8 done.
sending request 8...
sending request 9...
sending request 10...
sending request 11...
requests 8-12 done.

带信号量的滑动窗口

这种方法使用sliding window,并在另一个请求完成时立即调度新的请求,同时始终保持请求计数低于或等于任何时候n并发请求的最大数量。你需要实现的是一个Semaphore
在JavaScript中有一个名为async-mutex的库。
这里是一个使用这个库的示例程序,它同时向Postman Echo API发送2个请求。在信号量允许的情况下,永远不会有更多的请求并发运行(在您的情况下,限制是5,这里是2)。

import { Semaphore } from "async-mutex";
import fetch from "node-fetch";

// list of items that you might want to use to compose your URL (not actually used here)
let itemObject = [
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
];

(async () => {
    // allow two concurrent requests (adjust for however many are required)
    const semaphore = new Semaphore(2);

    itemObject.forEach(async (item, idx) => {
        // acquire the semaphore
        const [value, release] = await semaphore.acquire();
        // at this point the semaphore has been acquired and the job needs to be done 
        try {
            console.log(`sending request ${idx}...`)
            const response = await fetch("https://postman-echo.com/get")
            if(!response.ok){
                console.log(`request failed with status code ${response.status}`)
            }
        }
        catch (error) {
            console.log("request failed.")
        }
        finally {
            console.log(`request ${idx} done...`)
            // release the semaphore again so a new request can be issued 
            release();
        }
    })
})();


预期输出(顺序可能不同):

sending request 0...
sending request 1...
request 1 done...
sending request 2...
request 2 done...
sending request 3...
request 3 done...
sending request 4...
request 0 done...
sending request 5...
request 4 done...
sending request 6...
request 5 done...
sending request 7...
request 6 done...
sending request 8...
request 7 done...
sending request 9...
request 8 done...
sending request 10...
request 9 done...
sending request 11...
request 10 done...
request 11 done...

kadbb459

kadbb4592#

等待单个API调用完成

尝试await sendRequests()-由sendRequests()返回的挂起的promise被丢弃,因为它没有被传递给await运算符或添加了thencatchfinally子句。
如果你想让await sendRequests()response.text()启动的promise链的子句被执行之后被实现,而不是简单地被定义(这在sendRequests内部同步发生),在response.text()之前添加一个return语句:

return response.text()
 .then //  ... rest of promise chain code

字符串
这会强制await sendRequests()等待承诺链处理的执行。

计算未完成的请求

尝试将sendRequests重命名为sendRequest(单数),并编写一个节点模块(可能是sendRequests),用于记录已发出但仍在等待响应的请求。它将为单个请求返回一个promise,但不会发出新的获取操作,直到未完成请求的计数低于允许的限制。
这种模块的复杂性取决于设计标准:

  • 它是否由单个节点服务器、单个API、单个帐户使用
  • 它是否必须支持多个API URL,多个帐户和多个调用者。

使用模块化工厂函数或类构造函数来创建定制的sendRequests函数的通用解决方案对于您的用例来说可能是多余的,也可能不是。

相关问题