pig中的连接或过滤器更昂贵的是什么?

wgx48brx  于 2021-06-25  发布在  Pig
关注(0)|答案(1)|浏览(358)

如果在pig中选择加入或过滤,哪个性能更高?

ercv8c1e

ercv8c1e1#

联接总是代价高昂的,因为您必须为表1中的每个元组扫描第二个表。考虑下面的例子

A = LOAD 'data1' AS (a1:int,a2:int,a3:int);

DUMP A;
(1,2,3)
(4,2,1)
(8,3,4)
(4,3,3)
(7,2,5)
(8,4,3)

B = LOAD 'data2' AS (b1:int,b2:int);

DUMP B;
(2,4)
(8,9)
(1,3)
(2,7)
(2,9)
(4,6)
(4,9)

X = JOIN A BY a1, B BY b1;

DUMP X;
(1,2,3,1,3)
(4,2,1,4,6)
(4,3,3,4,6)
(4,2,1,4,9)
(4,3,3,4,9)
(8,3,4,8,9)
(8,4,3,8,9)

当我们加入x时,我们遍历b中的每个元组,得到a中的每个元组。对于filter,我们只遍历一次数据集,并对每个元组执行filter操作。

X = FILTER A BY a3 == 3;

DUMP X;
(1,2,3)
(4,3,3)
(8,4,3)

相关问题