我有以下格式的csv文件
customerid, period, credit, debit
100, jan-2017, 500, 300
100, jan-2017, 300,0
100, feb-2017, 200,100
100, mar-2017, 200,10
200, jan-2017, 100, 200
200, feb-2017,100,200
现在,我的要求是首先按客户id分组,然后按时段分组,合并事务,并使用apache pig脚本创建一个分层json,如下所示。
{
{
"customerid": 100,
"periods": [{
"period": "jan-2017",
"transactions": [{"credit": 500,"debit": 300},....]
}, {
"period": "feb-2017",
"transactions": [...]
}, {
"period": "mar-2017",
"transactions": [....]
}]
}, {
"customerid": 200,
"periods": [{
"period": "jan-2017",
"transactions": [.....]
}, {
"period": "feb-2017",
"transactions": [.....]
}]
}
}
我是相当新的Pig,但设法写了下面的脚本
Data = LOAD 'data.csv' USING PigStorage(',') AS (
company_id:chararray,
period:chararray,
debit:chararray,
credit:chararray)
CompanyBag = GROUP Data BY (company_id);
final_trsnactionjson = FOREACH CompanyBag {
ByCompanyId = FOREACH Data {
PeriodBag = GROUP Data BY (period);
IdPeriodItemRoot = FOREACH PeriodBag{
ItemRecords = FOREACH Source GENERATE debit as debit, credit as credit
GENERATE group as period, TOTUPLE(ItemRecords) as transactions;
}
}
GENERATE group as customerid, TOTUPLE(PeriodBag) AS periods;
};
但这给了我以下的错误
mismatched input '{' expecting GENERATE
关于如何使用pig生成嵌套的json,我搜索了很多,但是没有找到任何好的指针。我哪里出错了?提前谢谢你的帮助
1条答案
按热度按时间zed5wv101#
请使用jsonloader提供的清管器。
https://pig.apache.org/docs/r0.11.1/func.html#jsonloadstore 您可以在“as”中提供嵌套架构
使用com.twitter.elephantbird.pig.load.jsonloader('-nestedload')可以更简单地处理任何嵌套的json数组。