来自AWS S3的生 rust 极线?

omvjsjqw  于 2022-11-12  发布在  其他
关注(0)|答案(1)|浏览(198)

Polars指南展示了loading a file from S3的例子。不幸的是,它使用了python库pyarrow和一个函数from_arrow,这似乎也是python特有的。
我想知道是否有可能在纯Rust中做同样的事情?或者我最好的选择是使用Python FFI?
更新:目前似乎不可能,但work in progress

2izufjch

2izufjch1#

虽然不直接支持,但可以通过使用aws sdk将s3读取到内存中的对象,然后将其传递给文件读取器来实现。

use aws_sdk_s3::Region;

use aws_config::meta::region::RegionProviderChain;
use aws_sdk_s3::Client;
use std::borrow::Cow;

use polars::prelude::*;

# [tokio::main]

async fn main() {
    let region = "region";
    let bucket = "bucket";
    let key = "key";
    let region = Region::new(Cow::Borrowed(region));

    let region_provider = RegionProviderChain::default_provider().or_else(region);
    let config = aws_config::from_env().region(region_provider).load().await;
    let client = Client::new(&config);

    let req = client.get_object().bucket(bucket).key(key);

    let res = req.clone().send().await.unwrap();
    let bytes = res.body.collect().await.unwrap();
    let bytes = bytes.into_bytes();

    let cursor = std::io::Cursor::new(bytes);

    let df = CsvReader::new(cursor).finish().unwrap();

    println!("{:?}", df);
}

Cargo.toml

[dependencies]
aws-config = "0.49.0"
aws-sdk-s3 = "0.19.0"
polars = { version = "0.24.3", features = ["lazy", "parquet"] }
tokio = { version = "1.21.2", features = ["full"] }

相关问题