pig执行两个分拣袋的交叉

vmdwslir 于 2021-06-03 发布在 Hadoop

关注(0)|答案(3)|浏览(265)

我已经从hdfs加载了2个分类数据包。现在我想执行合并连接或设置相交对他们返回（3，孤儿的风暴），（7，穆里尔的婚礼）作为结果。
我有一些问题，让它与datafu或Pigmergejoin功能工作。
我尝试了下面提到的天真解决方案，但它没有利用我的数据被排序的优势。

vegas = LOAD 'vegas' USING PigStorage() AS (B1:bag{T1:tuple(id:int, name:chararray)});
macau = LOAD 'macau' USING PigStorage() AS (B2:bag{T2:tuple(id:int, name:chararray)});
vegast = FOREACH vegas GENERATE FLATTEN(vegas.$0) AS (id:int,name:chararray);
macaut = FOREACH hotel GENERATE FLATTEN(macau.$0) AS (id:int,name:chararray);

F = join vegast by id, macaut by id;
-- o/p: (3,Orphans of the Storm), (7,Muriel's Wedding)
-- describe vegas
--vegas: {B1: {T1: (id: int,name: chararray)}}
-- data for vegas
--({(3,Orphans of the Storm),(6,One Magic Christmas),(7,Muriel's Wedding),(8,Mother's Boys),(9,Nosferatu: Original Version)})

-- describe macau
--macau: {B1: {T1: (id: int,name: chararray)}}
--data for macau
--({(1,The Nightmare Before Christmas),(3,Orphans of the Storm),(4,The Object of Beauty),(7,Muriel's Wedding)})

有人能告诉我找到两个用pig分拣的袋子交叉点的最佳方法吗？

hadoop apache-pig

来源：https://stackoverflow.com/questions/22656506/pig-perform-intersection-of-2-bags-which-are-sorted

3条答案

按热度按时间

laawzig21#

如果关系是按联接字段排序的，则可以合并联接它们。

F = join vegast by id, macaut by id USING 'merge';

更多信息请参见pig文档：http://pig.apache.org/docs/r0.13.0/perf.html#merge-连接

赞(0）回复(0）举报 2021-06-04

9ceoxa922#

如果有人在datafu或pigmergejoin中得到setintersection来工作，请提供提示
SETFU指南。如果您正在加载以下结构或在执行连接后实现它（请注意，必须对行李进行分类）

DESCRIBE relationWith2Bags
relationWith2Bags: {B1: {(id: int,name: chararray)},B2: {(id: int,name: chararray)}}
--let it contain only 1 tuple with sorted bags from the question
--B1: {(3,Orphans of the Storm),(6,One Magic Christmas),(7,Muriel's Wedding),(8,Mother's Boys),(9,Nosferatu: Original Version)}
--B2: {(1,The Nightmare Before Christmas),(3,Orphans of the Storm),(4,The Object of Beauty),(7,Muriel's Wedding)}

intersect = FOREACH relationWith2Bags GENERATE datafu.pig.sets.SetIntersect(B1, B2);
DUMP intersect
--({(3,Orphans of the Storm),(7,Muriel's Wedding)})

赞(0）回复(0）举报 2021-06-04

wrrgggsh3#

我们在xplenty（hadoop平台即服务）遇到了同样的问题，我们决定走简单的道路，在jruby udf中实现set操作。
为了执行它，您需要在节点上安装jruby。
代码见此处：https://gist.github.com/saggineumann/9804083

赞(0）回复(0）举报 2021-06-03