我使用csv2parquet成功地将一个7GB以上（270万行）的CSV文件转换为 parquet 文件。
过程很简单：

• 首先，我必须使用来自csvkit的csvclean来清理CSV（但您可能不需要这个）
• 使用csv2parquet生成JSON模式
• 手动编辑模式，因为它可能不适合您
• 使用csv2parquet生成parquet文件
• 额外好处：使用DuckDB直接在parquet文件上测试简单的SQL查询

如果您在https://world.openfoodfacts.org/data下载我们的CSV导出文件，则可能会重现此过程

# Not needed for you, just in case you want to reproduce
wget https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv
csvclean -t en.openfoodfacts.org.products.csv

# Generate the schema
./csv2parquet --header true -p -n en.openfoodfacts.org.products_out.csv products_zstd.pqt > parquet.shema
# It has to be modified because column detection is sometimes wrong.
# From Open Food Facts CSV, for example, the code column is detected as a an Int64, but it's in fact a "Utf8".
nano parquet.shema

# Generate parquet file.
# Using -c for compression is optional.
# -c zstd appears to be the best option regarding speed/compression.
./csv2parquet --header true -c zstd -s parquet.schema en.openfoodfacts.org.products_out.csv products_zstd.pqt

# Try a query thanks to DuckDB. It's as fast as a database!
time ./duckdb test-duck.db "select * FROM (select count(data_quality_errors_tags) as products_with_issues from read_parquet('products_zstd.pqt') where data_quality_errors_tags != ''), (select count(data_quality_errors_tags) as products_with_issues_but_without_images from $db where data_quality_errors_tags != '' and last_image_datetime == '');"
┌──────────────────────┬─────────────────────────────────────────┐
│ products_with_issues │ products_with_issues_but_without_images │
├──────────────────────┼─────────────────────────────────────────┤
│ 29333                │ 4897                                    │
└──────────────────────┴─────────────────────────────────────────┘

real    0m0,211s
user    0m0,645s
sys 0m0,053s