如何通过(b,a)过滤(a,b)关系?

w8f9ii69  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(340)

我有这样一个泛型关系:

DUMP A;
(a, b)
(a, c)
(a, d)
(b, a)
(d, a)
(d, b)

看到有一对(a,b)和(b,a);但是(d,b)没有一对。我想过滤掉那些“未配对”的元组。
最终结果应该是:

DUMP R; 
(a, b)
(a, d)
(b, a)
(d, a)

我怎么能把这个写在Pig身上?
我可以用下面的代码解决,但是交叉操作太贵了:

A_cp = FOREACH L GENERATE u1, u2;
X = CROSS A, A_cp;
F = FILTER X BY ($0 == $3 AND $1 == $2);
R = FOREACH F GENERATE $0, $1;
utugiqy6

utugiqy61#

这是我的 DESCRIBE A ; DUMP A ; :

A: {first: chararray,second: chararray}
(a,b)
(a,c)
(a,d)
(b,a)
(d,a)
(d,b)

这是解决问题的一种方法:

A = LOAD 'foo.in' AS (first:chararray, second:chararray) ;
-- Can't do a join on its self, so we have to duplicate A
A2 = FOREACH A GENERATE * ;

-- Join the As so that are in (b,a,a,c) etc. pairs.
B = JOIN A BY second, A2 BY first ; 

-- We only want pairs where the first char is equal to the last char.
C = FOREACH (FILTER B BY A::first == A2::second)
    -- Now we project out just one side of the pair.
    GENERATE A::first AS first, A::second AS second ;

输出:

C: {first: chararray,second: chararray}
(b,a)
(d,a)
(a,b)
(a,d)

更新:正如winnienicklaus指出的,这可以缩短为:

B = FOREACH (JOIN A BY (first, second), A2 BY (second, first))
    GENERATE A::first AS first, A::second AS second ;

相关问题