如何在Deno中下载大文件?

zzwlnbp8  于 2022-09-18  发布在  Java
关注(0)|答案(2)|浏览(206)

我正在尝试下载一个10 GB的文件,但只有4 GB被保存到磁盘上,而且内存正在大量增加。

const res = await fetch('https://speed.hetzner.de/10GB.bin');
const file = await Deno.open('./10gb.bin', { create: true, write: true })

const ab = new Uint8Array(await res.arrayBuffer())
await Deno.writeAll(file, ab)
xbp102n0

xbp102n01#

你在缓冲React,这就是记忆不断增长的原因。

更新最新版本

Deno.open现在返回一个FsFile,该e1d1e在.writable属性中包含WritableStream,因此您可以通过管道将响应发送给它。

const res = await fetch('https://speed.hetzner.de/10GB.bin');
const file = await Deno.open('./10gb.bin', { create: true, write: true })

await res.body.pipeTo(file.writable);
file.close();

旧版本

您可以迭代res.body,因为它当前是实现Symbol.asyncIterator并在每个块上使用Deno.writeAllReadableStream

for await(const chunk of res.body) {
    await Deno.writeAll(file, chunk);
}
file.close();

您还可以使用writerFromStreamWriter将其通过管道传输到文件

import { writerFromStreamWriter } from "https://deno.land/std@0.156.0/streams/mod.ts?s=writerFromStreamWriter";

---

You can also use `fromStreamReader`  from `std/io` (`>= std@v0.60.0`) to convert `res.body` to a `Reader` that can be used in `Deno.copy`

从“https://deno.land/std@v0.60.0/io/streams.ts”;常量res=等待提取导入{FromStreamReader}(‘https://speed.hetzner.de/10GB.bin’);常量文件=等待Deno.Open(‘./10gb.bin’,{Create:TRUE,WRITE:TRUE}))

Const读取器=from StreamReader(res.body!.getReader());等待Deno.Copy(读取器,文件);file.lose();

---

Regarding why it stops at 4GB I'm not sure, but it may have to do with `ArrayBuffer` / `UInt8Array` limits, since 4GB is around 2³² bytes, which is the limit of `TypedArray`, [at least in most runtimes][2].

  [1]: https://doc.deno.land/https/github.com/denoland/deno/releases/latest/download/lib.deno.d.ts#Deno.copy
  [2]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length
cyvaqqii

cyvaqqii2#

下面是另一个简短的版本。

const res = await fetch('https://speed.hetzner.de/10GB.bin');
const file = await Deno.open('./10gb.bin', { create: true, write: true });

await res.body?.pipeTo(file.writable);
file.close();

相关问题