使用pig对大型Dataframe进行反规范化

dwbf0jvd 于 2021-06-21 发布在 Pig

关注(0)|答案(1)|浏览(274)

我有一个大的ish（21gb）制表符分隔的数据框的形式

DOCID_1    TERMID_1    TITLE_1    YEAR_1    AUTHOR_1
DOCID_1    TERMID_2    TITLE_1    YEAR_1    AUTHOR_1
...
DOCID_n    TERMID_n    TITLE_n    YEAR_n    AUTHOR_n

也就是说，（docid，termid）对总是唯一地标识一行。我需要的是一个Dataframe，其中docid单独唯一地标识一行，termid被折叠成一个逗号分隔的字符列表。例如，

DOCID_1    TERMID_11, TERMID_12, ..., TERMID_n    TITLE_1    YEAR_1    AUTHOR_1
...
DOCID_n    TERMID_n1, TERMID_n2, ..., TERMID_n    TITLE_1    YEAR_n    AUTHOR_n

有人能想出一个好的方法来做这件事吗？

apache-pig

来源：https://stackoverflow.com/questions/23091003/use-pig-to-denormalize-a-large-data-frame

1条答案

按热度按时间

vcirk6k61#

SEMINORMALIZED = LOAD 'so.txt' USING PigStorage(',') AS (
    doc_id:chararray
    ,term_id:chararray
    ,title:chararray
    ,year:chararray
    ,author:chararray
);

KEYS = FOREACH SEMINORMALIZED GENERATE 
    doc_id
    ,term_id
;

ATTRIBUTES = FOREACH SEMINORMALIZED GENERATE
    doc_id
    ,title
    ,year
    ,author
;

ATTRIBUTES = DISTINCT ATTRIBUTES;

GROUPED = GROUP KEYS BY doc_id;

ZNF = FOREACH GROUPED GENERATE
    group AS doc_id
    ,KEYS.term_id; AS term_ids

DENORMALIZED = JOIN ZNF BY doc_id, ATTRIBUTES BY doc_id;

赞(0）回复(0）举报 2021-06-21

我来回答

使用pig对大型Dataframe进行反规范化

1条答案

相关问题

热门标签

最新问答