我的mr程序有一些奇怪的问题,不知道为什么会这样。也许你能给我一个提示有什么问题?
我的Map器函数就是这样的:
Integer Click_ID = 0;
public void map(LongWritable key, Text value, Context context)
throws IOException , InterruptedException
{
String line = value.toString();
String []lineArr = line.split("\t");
String nm_uv_id = lineArr[0];
String session_id = lineArr[1];
String time_stamp = lineArr[2];
String click_counter = lineArr[3];
String is_robot = lineArr[4];
Click_ID++;
String full_line = Click_ID + "\t"+ nm_uv_id +"\t"+ session_id+"\t"+time_stamp+"\t"+click_counter+"\t"+ is_robot;
context.write(new Text(session_id), new Text(full_line));
}
到目前为止,一切正常-当我设置reducer的数量=0时,我的Map器生成预期的输出。
这是我的减速机的样子。我想做的是,对我的每个键迭代两次。为此,我尝试将iterable的每个值缓存在一个单独的arraylist中:
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
List<Text> cache = new ArrayList<Text>();
// first iterable
for (Text value : values) {
cache.add(value); }
//second iterable
for (Text entity : cache) {
context.write(key, entity); }
}
}
我用于mr的输入如下所示:
nm_uv_id_1 session_id_2 1234567891 1 is_robot_no
nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
nm_uv_id_1 session_id_2 1234567893 3 is_robot_no
nm_uv_id_1 session_id_2 1234567894 3 is_robot_no
nm_uv_id_1 session_id_1 1234567895 1 is_robot_no
nm_uv_id_1 session_id_1 1234567896 2 is_robot_no
nm_uv_id_1 session_id_1 1234567897 3 is_robot_no
nm_uv_id_1 session_id_1 1234567898 4 is_robot_no
nm_uv_id_1 session_id_1 1234567899 5 is_robot_no
nm_uv_id_1 session_id_1 1234567888 6 is_robot_no
nm_uv_id_1 session_id_1 1234567890 7 is_robot_no
nm_uv_id_1 session_id_1 1234567890 8 is_robot_no
nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
nm_uv_id_1 session_id_1 1234567890 10 is_robot_no
nm_uv_id_1 session_id_3 1234567890 1 is_robot_no
nm_uv_id_2 session_id_4 1234587890 1 is_robot_no
nm_uv_id_2 session_id_4 1234587890 2 is_robot_no
nm_uv_id_2 session_id_4 1234587890 3 is_robot_no
nm_uv_id_2 session_id_4 1234587890 4 is_robot_no
nm_uv_id_2 session_id_4 1234587890 5 is_robot_no
nm_uv_id_2 session_id_4 1234587890 6 is_robot_no
nm_uv_id_2 session_id_4 1234587890 7 is_robot_no
nm_uv_id_2 session_id_4 1234587890 8 is_robot_no
nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
nm_uv_id_2 session_id_5 1234587890 1 is_robot_no
nm_uv_id_2 session_id_5 1234587890 2 is_robot_no
nm_uv_id_2 session_id_5 1234587890 3 is_robot_yes
nm_uv_id_2 session_id_5 1234587890 4 is_robot_yes
nm_uv_id_2 session_id_5 1234587890 5 is_robot_no
nm_uv_id_2 session_id_5 123457890 6 is_robot_no
但是,我的输出文件如下所示:
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_no
session_id_3 15 nm_uv_id_1 session_id_3 1234567890 1 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
session_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
我不明白为什么减速机总是为一个特定的键写相同的键值对。我尝试了几件事,似乎第一个for循环,在那里我做缓存工作良好。当我编写context.write(key,value)时,我得到了预期的输出。然而第二,当我想使用第二个循环中的缓存时,程序为我写了一些奇怪的东西。
有人能帮忙吗?
1条答案
按热度按时间aoyhnmkz1#
它是重复使用相同的
Text
作为优化的缓冲区。因此,您需要手动克隆以缓存它。我会改变你的缓存循环: