NodeJS 带分页的AWS s3列表对象

kyvafyod  于 2023-02-03  发布在  Node.js
关注(0)|答案(3)|浏览(154)

我想使用awss3实现分页。对象ms.files中有500个文件,但我想一次只检索20个文件,下次检索20个文件,以此类推。

var params = {
  Bucket: 'mystore.in',
  Delimiter: '/',
  Prefix: '/s/ms.files/',
  Marker:'images',
};
s3.listObjects(params, function(err, data) {
  if (err) console.log(err, err.stack); 
  else     console.log(data);          
});
ep6jt1vc

ep6jt1vc1#

在一次列出所有对象时遇到了这个问题,如果响应被截断,它会为下一次调用提供一个标记isTruncated = true和一个continuationToken
如果你在es6上,你可以这样做,

const AWS = require('aws-sdk');
const s3 = new AWS.S3({});

const listAllContents = async ({ Bucket, Prefix }) => {
  // repeatedly calling AWS list objects because it only returns 1000 objects
  let list = [];
  let shouldContinue = true;
  let nextContinuationToken = null;
  while (shouldContinue) {
    let res = await s3
      .listObjectsV2({
        Bucket,
        Prefix,
        ContinuationToken: nextContinuationToken || undefined,
      })
      .promise();
    list = [...list, ...res.Contents];

    if (!res.IsTruncated) {
      shouldContinue = false;
      nextContinuationToken = null;
    } else {
      nextContinuationToken = res.NextContinuationToken;
    }
  }
  return list;
};
guykilcj

guykilcj2#

对于正在使用@aws-sdk/client-s3和TypeScript的任何人,下面是一个从桶中提取所有对象的示例:

import {
  S3Client,
  ListObjectsV2Command,
  ListObjectsV2CommandInput,
  _Object,
} from "@aws-sdk/client-s3";

export const fetchObjects = async (bucket: string) => {
  const objects: _Object[] = [];

  async function fetchObjectsWithPagination(
    bucket: string,
    continuationToken?: ListObjectsV2CommandInput["ContinuationToken"]
  ): Promise<void> {
    const s3 = new S3Client({ region: "YOUR_S3_BUCKET_REGION" });
  
    const result = await s3.send(
      new ListObjectsV2Command({
        Bucket: bucket,
        ContinuationToken: continuationToken,
      })
    );

    objects.push(...(result.Contents || []));
  
    if (result.NextContinuationToken) {
      return fetchObjectsWithPagination(bucket, result.NextContinuationToken);
    }

    return;
  }

  await fetchObjectsWithPagination(bucket);

  return objects;
}
1rhkuytd

1rhkuytd3#

Mr Jarmod共享的解决方案:

var params = {
  Bucket: 'mystore.in',
  Delimiter: '/',
  Prefix: '/s/ms.files/',
  Marker:'',
  MaxKeys : 20
};
s3.listObjects(params, function(err, data) {
  if (err) console.log(err, err.stack); 
  else     console.log(data);          
});

相关问题