使用pig和json存储来自csv的多级json

ffx8fchx  于 2021-06-24  发布在  Pig
关注(0)|答案(1)|浏览(254)

我有以下格式的csv文件

customerid, period, credit, debit
 100, jan-2017, 500, 300
 100, jan-2017, 300,0
 100, feb-2017, 200,100
 100, mar-2017, 200,10
 200, jan-2017, 100, 200
 200, feb-2017,100,200

现在,我的要求是首先按客户id分组,然后按时段分组,合并事务,并使用apache pig脚本创建一个分层json,如下所示。

{
    {
        "customerid": 100,
        "periods": [{
            "period": "jan-2017",
            "transactions": [{"credit": 500,"debit": 300},....]
        }, {
            "period": "feb-2017",
            "transactions": [...]
        }, {
            "period": "mar-2017",
            "transactions": [....]
        }]
    }, {
        "customerid": 200,
        "periods": [{
            "period": "jan-2017",
            "transactions": [.....]
        }, {
            "period": "feb-2017",
            "transactions": [.....]
        }]
    }
}

我是相当新的Pig,但设法写了下面的脚本

Data = LOAD 'data.csv' USING PigStorage(',') AS (
    company_id:chararray,
    period:chararray,
    debit:chararray,
    credit:chararray)

CompanyBag = GROUP Data BY (company_id);

final_trsnactionjson = FOREACH CompanyBag {
    ByCompanyId = FOREACH Data {
        PeriodBag = GROUP Data BY (period);

        IdPeriodItemRoot = FOREACH PeriodBag{
            ItemRecords = FOREACH Source GENERATE debit as debit, credit as credit
            GENERATE group as period, TOTUPLE(ItemRecords) as transactions;
        }   
    }
    GENERATE group as customerid, TOTUPLE(PeriodBag) AS periods;
};

但这给了我以下的错误

mismatched input '{' expecting GENERATE

关于如何使用pig生成嵌套的json,我搜索了很多,但是没有找到任何好的指针。我哪里出错了?提前谢谢你的帮助

zed5wv10

zed5wv101#

请使用jsonloader提供的清管器。
https://pig.apache.org/docs/r0.11.1/func.html#jsonloadstore 您可以在“as”中提供嵌套架构
使用com.twitter.elephantbird.pig.load.jsonloader('-nestedload')可以更简单地处理任何嵌套的json数组。

相关问题