避免pig中多关系连接的前缀

4xrmg8kj 于 2021-06-24 发布在 Pig

关注(0)|答案(2)|浏览(375)

我正在尝试在pig中执行一个星型模式类型的join，下面是我的代码。当我用不同的列连接多个关系时，每次都必须在前一个连接的名称前面加上前缀才能使其工作。我相信应该有更好的方法，我无法通过谷歌找到它。任何提示都会很有帮助。
i、我想避免在一列前面加上这样的前缀“h864:：h86:：hs\u 8\u d:：hs\u 8\u desc”。

hs_8 = LOAD 'hs_8_distinct' USING PigStorage('^') as (hs_8:chararray,hs_8_desc:chararray);
hs_8_d = FOREACH hs_8 GENERATE SUBSTRING(hs_8,0,2) as hs_2,SUBSTRING(hs_8,0,4) as hs_4,SUBSTRING(hs_8,0,6) as hs_6,hs_8,hs_8_desc;

hs_6_d = LOAD 'hs_6_distinct' USING PigStorage('^') as (hs_6:chararray,hs_6_desc:chararray);
hs_4_d = LOAD 'hs_4_distinct' USING PigStorage('^') as (hs_4:chararray,hs_4_desc:chararray);
hs_2_d = LOAD 'hs_2_distinct' USING PigStorage('^') as (hs_2:chararray,hs_2_desc:chararray);

H86 = JOIN hs_8_d BY hs_6, hs_6_d BY hs_6 USING 'replicated' ;
H864 = JOIN H86 BY hs_8_d::hs_4, hs_4_d BY hs_4 USING 'replicated' ;
H8642 = JOIN H864 BY H86::hs_8_d::hs_2, hs_2_d BY hs_2 USING 'replicated' ;

hs_dim = FOREACH H8642 GENERATE hs_2_d::hs_2,hs_2_d::hs_2_desc,H864::hs_4_d::hs_4,H864::hs_4_d::hs_4_desc,H864::H86::hs_6_d::hs_6,H864::H86::hs_6_d::hs_6_desc,H864::H86::hs_8_d::hs_8,H864::H86::hs_8_d::hs_8_desc;

apache-pig

来源：https://stackoverflow.com/questions/17255842/avoiding-prefixes-in-multi-relation-join-in-pig

2条答案

按热度按时间

hs1rzwqc1#

pig总是在字段的前面加上 bagname:: 在联接后消除字段的歧义。我不认为你能避免不幸的事。

赞(0）回复(0）举报 2021-06-24

vulvrdjw2#

通过向连接中添加额外的foreach，可以稍微简化别名。看看统计数据，这不会给管道增加额外的mr职位。原来的和这将产生4个Map只工作。
例如：

H86 = foreach (JOIN hs_8_d BY hs_6, hs_6_d BY hs_6 USING 'replicated') generate 
        hs_8_d::hs_2 as x1, 
        hs_8_d::hs_4 as x2, 
        hs_8_d::hs_6 as x3, 
        hs_8_d::hs_8 as x4,
        hs_8_d::hs_8_desc as x5, 
        hs_6_d::hs_6 as x6,
        hs_6_d::hs_6_desc as x7;

H864 = foreach (JOIN H86 BY x2, hs_4_d BY hs_4 USING 'replicated') generate 
          H86::x1 as y1,
          H86::x2 as y2, 
          H86::x3 as y3,
          H86::x4 as y4,
          H86::x5 as y5, 
          H86::x6 as y6, 
          H86::x7 as y7,
          hs_4_d::hs_4 as y8,
          hs_4_d::hs_4_desc as y9;

H8642 = foreach (JOIN H864 BY y1, hs_2_d BY hs_2 USING 'replicated') generate 
          H864::y1 as z1, 
          H864::y2 as z2,
          H864::y3 as z3, 
          H864::y4 as z4, 
          H864::y5 as z5, 
          H864::y6 as z6, 
          H864::y7 as z7,
          H864::y8 as z8, 
          H864::y9 as z9, 
          hs_2_d::hs_2 as z10, 
          hs_2_d::hs_2_desc as z11;

hs_dim = FOREACH H8642 GENERATE z10, z11, z8, z9, z6, z7, z4, z5;

如果您有一包元组，那么datafu的aliasbagfields可能会有所帮助。

赞(0）回复(0）举报 2021-06-24

我来回答

避免pig中多关系连接的前缀

2条答案

相关问题

热门标签

最新问答