hadoop-从记录读取器到Map函数的多个文件

dvtswwa3  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(315)

我实现了一个定制的合并文件输入格式,以便为由一组文件组成的Map任务创建拆分。我创建了一个解决方案,将剥离的每个文件通过记录读取器,一切正常。现在,我尝试将整个文件集传递给map函数。
这是我的读卡器代码:

public class MultiImagesRecordReader extends
        RecordReader<Text[], BytesWritable[]> {
private long start = 0;
private long end = 0;
private int pos = 0;
private BytesWritable[] value;
private Text key[];
private CombineFileSplit split;
private Configuration conf;
private FileSystem fs;
private static boolean recordsRead;

public MultiImagesRecordReader(CombineFileSplit split,
        TaskAttemptContext context, Integer index) throws IOException {
    this.split = split;
    this.conf = context.getConfiguration();
}

@Override
public void initialize(InputSplit genericSplit, TaskAttemptContext context)
        throws IOException, InterruptedException {
    start = split.getOffset(0);
    end = start + split.getLength();
    recordsRead = false;
    this.pos = (int) start;
    fs = FileSystem.get(conf);
    value = new BytesWritable[split.getNumPaths()];
    key = new Text[split.getNumPaths()];
}

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
    if (recordsRead == true) {
        System.out.println("Sono nel next true"+InetAddress.getLocalHost());
        return false;
    } else {
        recordsRead = true;
        System.out.println("Sono nel next false"+InetAddress.getLocalHost());
        for (int i = 0; i < split.getNumPaths(); i++) {

            int fileLength = (int) split.getLength(i);
            Path path = split.getPath(i);
            byte[] result = new byte[fileLength];

            FSDataInputStream in = null;

            String file_path = path.toString();
            key[i] = new Text(file_path);
            try {
                in = fs.open(path);
                IOUtils.readFully(in, result, 0, fileLength);

            } finally {
                IOUtils.closeStream(in);
            }

            value[i] = new BytesWritable(result);
        }
        return true;
    }
}

使用这段代码,map函数可以正确地接收键和值的向量,但会重复。我的意思是,我期望map函数被调用一次,而不是多次调用。我做错什么了?

d4so4syb

d4so4syb1#

我想你知道 map()Mapper 将为读取器返回的每条记录调用 currentKey() , currentValue() 直到给定 Split 已经完成了。我知道map函数是为同一个键值对重复调用的(应该为单个键值对调用一次)。这意味着您的记录阅读器重复读取同一条记录(键值对)。我还实现了自定义的组合文件输入格式和记录阅读器

相关问题