pig：计算文件中不同元组的频率

f3temu5u 于 2021-06-21 发布在 Pig

关注(0)|答案(1)|浏览(321)

我有一个包含json条目的文件，如下所示：

{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "case", "child_dep": "nn", "parent_dep": "nsubj", "child": "martin"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "case", "child_dep": "nn", "parent_dep": "nsubj", "child": "martin"}

我想计算文件中不同json对象的频率。我在pig中看到了使用group by和count（）函数的其他答案。我不确定我是否正确使用它们，但我没有得到所需的结果。我的输出应该如下所示：

{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua", "count": "3"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "case", "child_dep": "nn", "parent_dep": "nsubj", "child": "martin", "count": "2"}

顺序并不重要。有人能给我一些建议吗？

JSON count apache-pig

来源：https://stackoverflow.com/questions/39927653/pig-count-frequency-of-distinct-tuples-in-a-file

1条答案

按热度按时间

7cwmlq891#

这里是可以使用的代码，所有字段的条件都要分组如果你想要其他格式，你可以从元组中读取feild并使用任何其他格式

A = LOAD '/user/root/test12.json' USING JsonLoader('child_pos:chararray,               parent_pos:chararray, parent:chararray, child_dep:chararray, parent_dep:chararray, child:chararray');
B = GROUp A by (child_pos, parent_pos, parent, child_dep, parent_dep, child) ;
C = FOREACH B GENERATE group, COUNT(A.child_pos) as COUNTX;
STORE C into 'user/data/json_out.json' USING JsonStorage();

out put is ... 
{"group":    {"child_pos":"NN","parent_pos":"NN","parent":"case","child_dep":"nn","parent_dep":"nsubj","child":"martin"},"COUNTX":2}
{"group":{"child_pos":"NN","parent_pos":"NN","parent":"fighter","child_dep":"nn","parent_dep":"nsubj","child":"virtua"},"COUNTX":3}

赞(0）回复(0）举报 2021-06-21

我来回答

pig：计算文件中不同元组的频率

1条答案

相关问题

热门标签

最新问答