将reducer方法的迭代器< pojo>对象存储到数组

ifmq2ha2  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(351)

我想存储迭代器值,这是reduce方法的参数,即

public void reduce(IntWritable key, Iterator<Pojo> values,
        OutputCollector<IntWritable, SubArrayWritable> output, Reporter reporter)

在自定义arraywritable类对象中,即pojoarraywritable对象。我已经创建了pojoarraywritable类,它的代码是

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.ArrayWritable;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;

public class SubArrayWritable extends ArrayWritable
{
public SubArrayWritable() {
    super(Sub.class);
}
public SubArrayWritable(Sub[] values) {
    super(Sub.class, values);
}
}

sub.class有以下代码-

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Writable;

public class Sub implements Writable {
private IntWritable id = new IntWritable();
private LongWritable pno = new LongWritable();
private DoubleWritable sal = new DoubleWritable();

public IntWritable getid() {
    return id;
}

public void setId(IntWritable id) {
    this.id = id;
}

public LongWritable getPno() {
    return pno;
}

public void setPno(LongWritable pno) {
    this.pno = pno;
}

public DoubleWritable getSal() {
    return sal;
}

public void setSal(DoubleWritable sal) {
    this.sal = sal;
}

@Override
public void readFields(DataInput in) throws IOException {
    id.readFields(in);
    pno.readFields(in);
    sal.readFields(in);

}

@Override
public void write(DataOutput out) throws IOException {
    id.write(out);
    pno.write(out);
    sal.write(out);
}

}

在减速机类

public void reduce(IntWritable key, Iterator<Pojo> values,
        OutputCollector<IntWritable, SubArrayWritable> output, Reporter reporter)

我想把iterator的values对象存储到array,也就是subraywriteable的对象,那么我怎么做呢?因为我不知道iterator的values对象的大小/长度,所以我要创建的subraywriteable对象的长度是多少??
所以基本上我有 Iterator<Pojo> values 我必须把它转换成 array of Pojo

ijxebb2r

ijxebb2r1#

我找到了解决这个问题的办法-

Pojo pojo=new Pojo();
    List<Pojo> list=new ArrayList<Pojo>();

    if (values.hasNext()) {
        pojo = values.next();
        list.add(pojo);
       }

现在我可以使用这个pojo对象值,它包含迭代器对象的值

相关问题