java在arraylist中缓存iterable以在reducer中迭代两次是行不通的

uinbv5nw  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(430)

我的mr程序有一些奇怪的问题,不知道为什么会这样。也许你能给我一个提示有什么问题?
我的Map器函数就是这样的:

Integer Click_ID = 0;

  public void map(LongWritable key, Text value, Context context)
        throws IOException , InterruptedException
  {

      String line = value.toString();

      String []lineArr = line.split("\t");

      String nm_uv_id = lineArr[0];
      String session_id = lineArr[1];
      String time_stamp = lineArr[2];
      String click_counter = lineArr[3];
      String is_robot = lineArr[4];

      Click_ID++;

      String full_line =  Click_ID + "\t"+ nm_uv_id +"\t"+ session_id+"\t"+time_stamp+"\t"+click_counter+"\t"+ is_robot;

      context.write(new Text(session_id), new Text(full_line));

   }

到目前为止,一切正常-当我设置reducer的数量=0时,我的Map器生成预期的输出。
这是我的减速机的样子。我想做的是,对我的每个键迭代两次。为此,我尝试将iterable的每个值缓存在一个单独的arraylist中:

public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
    List<Text> cache = new ArrayList<Text>();

           // first iterable
    for (Text value : values) {
                                       cache.add(value); }

           //second iterable
        for (Text entity : cache) {

            context.write(key, entity);  }
    }

}
我用于mr的输入如下所示:

nm_uv_id_1  session_id_2    1234567891  1   is_robot_no
nm_uv_id_1  session_id_2    1234567892  2   is_robot_no
nm_uv_id_1  session_id_2    1234567893  3   is_robot_no
nm_uv_id_1  session_id_2    1234567894  3   is_robot_no
nm_uv_id_1  session_id_1    1234567895  1   is_robot_no
nm_uv_id_1  session_id_1    1234567896  2   is_robot_no
nm_uv_id_1  session_id_1    1234567897  3   is_robot_no
nm_uv_id_1  session_id_1    1234567898  4   is_robot_no
nm_uv_id_1  session_id_1    1234567899  5   is_robot_no
nm_uv_id_1  session_id_1    1234567888  6   is_robot_no
nm_uv_id_1  session_id_1    1234567890  7   is_robot_no
nm_uv_id_1  session_id_1    1234567890  8   is_robot_no
nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
nm_uv_id_1  session_id_1    1234567890  10  is_robot_no
nm_uv_id_1  session_id_3    1234567890  1   is_robot_no
nm_uv_id_2  session_id_4    1234587890  1   is_robot_no
nm_uv_id_2  session_id_4    1234587890  2   is_robot_no
nm_uv_id_2  session_id_4    1234587890  3   is_robot_no
nm_uv_id_2  session_id_4    1234587890  4   is_robot_no
nm_uv_id_2  session_id_4    1234587890  5   is_robot_no
nm_uv_id_2  session_id_4    1234587890  6   is_robot_no
nm_uv_id_2  session_id_4    1234587890  7   is_robot_no
nm_uv_id_2  session_id_4    1234587890  8   is_robot_no
nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
nm_uv_id_2  session_id_5    1234587890  1   is_robot_no
nm_uv_id_2  session_id_5    1234587890  2   is_robot_no
nm_uv_id_2  session_id_5    1234587890  3   is_robot_yes
nm_uv_id_2  session_id_5    1234587890  4   is_robot_yes
nm_uv_id_2  session_id_5    1234587890  5   is_robot_no
nm_uv_id_2  session_id_5    123457890   6   is_robot_no

但是,我的输出文件如下所示:

session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_1    13  nm_uv_id_1  session_id_1    1234567890  9   is_robot_no
session_id_2    2   nm_uv_id_1  session_id_2    1234567892  2   is_robot_no
session_id_2    2   nm_uv_id_1  session_id_2    1234567892  2   is_robot_no
session_id_2    2   nm_uv_id_1  session_id_2    1234567892  2   is_robot_no
session_id_2    2   nm_uv_id_1  session_id_2    1234567892  2   is_robot_no
session_id_3    15  nm_uv_id_1  session_id_3    1234567890  1   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_4    24  nm_uv_id_2  session_id_4    1234587890  9   is_robot_no
session_id_5    30  nm_uv_id_2  session_id_5    123457890   6   is_robot_no
session_id_5    30  nm_uv_id_2  session_id_5    123457890   6   is_robot_no
session_id_5    30  nm_uv_id_2  session_id_5    123457890   6   is_robot_no
session_id_5    30  nm_uv_id_2  session_id_5    123457890   6   is_robot_no
session_id_5    30  nm_uv_id_2  session_id_5    123457890   6   is_robot_no
session_id_5    30  nm_uv_id_2  session_id_5    123457890   6   is_robot_no

我不明白为什么减速机总是为一个特定的键写相同的键值对。我尝试了几件事,似乎第一个for循环,在那里我做缓存工作良好。当我编写context.write(key,value)时,我得到了预期的输出。然而第二,当我想使用第二个循环中的缓存时,程序为我写了一些奇怪的东西。
有人能帮忙吗?

aoyhnmkz

aoyhnmkz1#

它是重复使用相同的 Text 作为优化的缓冲区。因此,您需要手动克隆以缓存它。
我会改变你的缓存循环:

for (Text value : values) { cache.add(new Text(value)); }

相关问题