如何为hadoop输入格式的一个输入行生成多个键值对?

dl5txlt9  于 2021-06-04  发布在  Hadoop
关注(0)|答案(3)|浏览(275)

这是背景。我的mapreduce工作有以下输入(示例):

Apache Hadoop
Apache Lucene
StackOverflow
....

(实际上每行代表一个用户查询。我想要我的 RecordReader 类读取一行,然后将几个键值对传递给Map器。例如,如果 RecordReader 得到 Apache Hadoop ,然后我希望它生成以下键值对并将其传递给Map器:

Apache Hadoop - 1
Apache Hadoop - 2
Apache Hadoop - 3

(“-”是分隔符)我发现 RecordReader 传递键值 next() 方法:

next(key, value);

每次调用recordreader.next()时,只有一个键和一个值作为参数传递。那我该怎么完成工作呢?

5jdjgkvh

5jdjgkvh1#

试着不要给予key:-

context.write(NullWritable.get(), new Text("Apache Hadoop - 1"));
context.write(NullWritable.get(), new Text("Apache Hadoop - 2"));
context.write(NullWritable.get(), new Text("Apache Hadoop - 3"));
7cjasjjr

7cjasjjr2#

我想如果你想发送到Map使用相同的键;您必须实现您的所有者recordreader;例如,您可以创建一个mutlirecordreader来扩展linerecordreade;这里您必须更改nextkeyvalue方法;这是linerecordreade的原始代码:

public boolean nextKeyValue() throws IOException {
    if (key == null) {
      key = new LongWritable();
    }
    key.set(pos);
    if (value == null) {
      value = new Text();
    }
    int newSize = 0;
    // We always read one extra line, which lies outside the upper
    // split limit i.e. (end - 1)
    while (getFilePosition() <= end) {
      newSize = in.readLine(value, maxLineLength,
          Math.max(maxBytesToConsume(pos), maxLineLength));
      pos += newSize;
      if (newSize < maxLineLength) {
        break;
      }

      // line too long. try again
      LOG.info("Skipped line of size " + newSize + " at pos " + 
               (pos - newSize));
    }
    if (newSize == 0) {
      key = null;
      value = null;
      return false;
    } else {
      return true;
    }
  }

你可以这样改变它:

public boolean nextKeyValue() throws IOException {
    if (key == null) {
      key = new Text();
    }
    key.set(pos);
    if (value == null) {
      value = new Text();
    }
    int newSize = 0;

    while (getFilePosition() <= end && n<=3) {
      newSize = in.readLine(key, maxLineLength,
          Math.max(maxBytesToConsume(pos), maxLineLength));//change value --> key

     value =Text(n);
     n++;
     if(n ==3 )// we don't go to next until the N is three;
         pos += newSize;

      if (newSize < maxLineLength) {
        break;
      }

      // line too long. try again
      LOG.info("Skipped line of size " + newSize + " at pos " + 
               (pos - newSize));
    }
    if (newSize == 0) {
      key = null;
      value = null;
      return false;
    } else {
      return true;
    }
  }

我想这个适合你

vmjh9lq9

vmjh9lq93#

我相信你可以简单地用这个:

public static class MultiMapper 
        extends Mapper<LongWritable, Text, Text, IntWritable> {

    @Override
    public void map(LongWritable key, Text value, Context context) 
            throws IOException, InterruptedException {

        for (int i = 1; i <= n; i++) {
            context.write(value, new IntWritable(i));
        }
    }
}

这里n是要传递的值的数目。例如,对于指定的键值对:

Apache Hadoop - 1
Apache Hadoop - 2
Apache Hadoop - 3

n等于3。

相关问题