pig:twitter sentiment 分析

uttx8gqw 于 2021-06-02 发布在 Hadoop

关注(0)|答案(3)|浏览(401)

我正在尝试实现twitter情绪分析。我需要获得所有积极的tweets和消极的tweets，并将它们存储在特定的文本文件中。
示例.json

{"id": 252479809098223616, "created_at": "Wed Apr 12 08:23:20 +0000 2016", "text": "google is a good company", "user_id": 450990391}{"id": 252479809098223616, "created_at": "Wed Apr 12 08:23:20 +0000 2016", "text": "facebook is a bad company","user_id": 450990391}

包含所有肯定词和否定词列表的文本

weaksubj    1   bad     adj     n   negative
strongsubj  1   good    adj     n   positive

Pigscript:-

tweets = load 'new.json' using JsonLoader('id:chararray,text:chararray,user_id:chararray,created_at:chararray');

dictionary = load 'dictionary.text' AS (type:chararray,length:chararray,word:chararray,pos:chararray,stemmed:chararray,polarity:chararray);

words = foreach tweets generate FLATTEN( TOKENIZE(text) ) AS word,id,text,user_id,created_at;

sentiment = join words by word left outer, dictionary by word;

senti2 = foreach sentiment generate words::id as id,words::created_at as created_at,words::text as text,words::user_id as user_id,dictionary::polarity as polarity;

res = FILTER senti2 BY polarity MATCHES '.*possitive.*';

描述res:-

res: {id: chararray,created_at: chararray,text: chararray,user_id: chararray,polarity: chararray}

但是当我转储res时，我看不到任何输出，但是它执行得很好，没有任何错误。
我在这里犯了什么错误。
请建议我。
莫汉五世

hadoop apache-pig twitter sentiment-analysis

来源：https://stackoverflow.com/questions/39586045/pigtwitter-sentiment-analysis

3条答案

按热度按时间

k3fezbri1#

我在这里看到两个错误
1:第2行-转储dictionary时，您将看到第1列中的所有记录，其余列显示为空。
解决方案：使用pigstorage（）指定适当的分隔符；

dictionary = load 'dictionary.text' AS     (type:chararray,length:chararray,word:chararray,pos:chararray,stemmed:chararray,polarity:chararray);

DUMP dictionary;
(weaksubj    1   bad     adj     n   negative,,,,,)
(strongsubj  1   good    adj     n   positive,,,,,)

第二个错误：第6行：更正肯定的拼写！使用类似

res = FILTER senti2 BY UPPER(polarity) MATCHES '.*POSITIVE.*';

赞(0）回复(0）举报 2021-06-02

mlnl4t2r2#

我看到拼写错误在：

res = FILTER senti2 BY polarity MATCHES '.*possitive.*';

不是吗 '.*positive.*' ?

赞(0）回复(0）举报 2021-06-02

ymzxtsji3#

根据我的建议，你应该使用自定义自定义自定义项来解决你的问题。现在可以使用elephant-bird-pig-4.1.jar、json-simple-1.1.jar。另外，如果你想看看这些例子，那么你可以使用这些情绪分析教程。如果您需要代码，那么您可以参考这些代码并根据教程和我的代码格式化您的代码，

REGISTER ‘/usr/local/elephant-bird-hadoop-compat-4.1.jar';
REGISTER '/ usr/local /elephant-bird-pig-4.1.jar';
REGISTER '/ usr/local /json-simple-1.1.1.jar’;
load_tweets = LOAD '/user/new.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS myMap;
extract_details = FOREACH load_tweets GENERATE myMap#'id' as id,myMap#'text' as text;
tokens = foreach extract_details generate id,text, FLATTEN(TOKENIZE(text)) As word;
dictionary = load '/user/dictionary.text' AS (type:chararray,length:chararray,word:chararray,pos:chararray,stemmed:chararray,polarity:chararray);
word_rating = join tokens by word left outer, dictionary by word using 'replicated’; describe word_rating;
rating = foreach word_rating generate tokens::id as id,tokens::text as text, dictionary::rating as rate;
word_group = group rating by (id,text);
avg_rate = foreach word_group generate group, AVG(rating.rate) as tweet_rating;
positive_tweets = filter avg_rate by tweet_rating>=0;

赞(0）回复(0）举报 2021-06-02

我来回答

pig:twitter sentiment 分析

3条答案

相关问题

热门标签

最新问答