使用pig拉丁语将列拆分为组

ig9co6j1 于 2021-06-21 发布在 Pig

关注(0)|答案(2)|浏览(312)

我有一张有两列的table(code:chararray，sp:双）
我想将第二个字段sp划分为不同的组（例如，基于类似（<25）、（>25<45）、（>=45）的条件）。
输入

code sp
t001 60.0
t001 75.0
a003 34.0
t001 60.0
a003 23.0
a003 23.0
t001 45.0
t001 10.0
t001 8.0
a003 20.0
t001 38.0
a003 55.0
a003 50.0
t001 08.0
a003 44.0

期望输出：

code    bin1     bin2        bin3
       (<25)   (>25 <45)    >=45
t001    3          1          4 
a003    3          2          2

我正在尝试下面的脚本：

data = LOAD 'Sandy/rd.csv' using PigStorage(',') As (code:chararray,sp:double);

data2 = DISTINCT data;

selfiltnew = FOREACH data2 generate code, sp;
group_new = GROUP selfiltnew by (code,sp);

newselt = FOREACH group_new GENERATE selfiltnew.code AS code,selfiltnew.sp AS sp;

bin1 = filter newselt by sp < 25.0;
grp1 = FOREACH bin1 GENERATE newselt.code AS code, COUNT(newselt.sp) AS (sp1:double);

bin2 = filter newselt by sp < 45 and sp >= 25;
grp2 = FOREACH bin3 GENERATE newselt.code AS code, COUNT(newselt.sp) AS (sp2:double);

bin3 = filter newselt by sp >=75;
grp3 = FOREACH bin3 GENERATE newselt.code AS code, COUNT(newselt.sp) AS (sp3:double);

newbin = JOIN grp1 by code,grp2 by code,grp3 by code;

newtable = FOREACH newbin GENERATE grp1::group.code AS code, SUM(sp1) AS bin1,SUM(sp2) AS bin2,SUM(sp3) AS bin3;

data2 = FOREACH newtable GENERATE code, bin1, bin2, bin3;
dump newtable;

如何使用pig拉丁语获得正确的输出？

hadoop2 apache-pig

来源：https://stackoverflow.com/questions/38418944/column-splitting-into-groups-using-pig-latin

2条答案

按热度按时间

d5vmydt91#

在使用count之前必须使用group by
source:count
使用
使用count函数计算一个包中的元素数。count要求前面的group all语句用于全局计数，group by语句用于组计数。

bin1 = filter newselt by sp < 25.0;
grouped1 = GROUP bin1 by (newselt.code);
grp1 = FOREACH grouped1 GENERATE group AS code, COUNT(newselt.sp) AS (sp1:double);

赞(0）回复(0）举报 2021-06-21

webghufk2#

通过看到你想要的输出 DISTINCT 是需要的。此外，也不需要执行您正在执行的某些步骤。请注意，如果源被空格分隔，则应使用 PigStorage(' ') 而不是 PigStorage(',') 按照@inquisitive\u mind所指出的，代码如下：

data = LOAD 'Sandy/rd.csv' using PigStorage(' ') As (code:chararray,sp:double);
bin1 = filter data by sp < 25.0;
grouped1 = GROUP bin1 by code;
grp1 = FOREACH grouped1 GENERATE group AS code, COUNT(bin1.sp) AS (sp1:double);
bin2 = filter data by (sp >= 25.0 AND sp<45);
grouped2 = GROUP bin2 by code;
grp2 = FOREACH grouped2 GENERATE group AS code, COUNT(bin2.sp) AS (sp2:double);
bin3 = filter data by sp >= 45.0;
grouped3 = GROUP bin3 by code;
grp3 = FOREACH grouped3 GENERATE group AS code, COUNT(bin3.sp) AS (sp3:double);
result= JOIN grp1 BY code, grp2 by code, grp3 by code;
final_result = FOREACH result GENERATE grp1::code as code, grp1::sp1 as bin1, grp2::sp2 as bin2, grp3::sp3 as bin3;

输出如下：

赞(0）回复(0）举报 2021-06-21

我来回答

使用pig拉丁语将列拆分为组

2条答案

相关问题

热门标签

最新问答