hadoop mapreduce输入键错误

xtfmy6hx  于 2021-06-04  发布在  Hadoop
关注(0)|答案(1)|浏览(350)

我想传递一个浮点数组作为mapper中的键,

Such as [0.6, 0, 10]

我使用stringutils.join将其转换为字符串,然后编写

context.write(new Text(str), value);

但在减速机课上,我发现我得到的是:

"0 0 1"

如何修复?我正在使用hadoop1.0.4

hpcdzsge

hpcdzsge1#

可以使用 ArrayWritable 。我不确定在发射数组中作为键,下面是如何发射相同的作为值。

ArrayWritable does not implement WritableComparable, so it can't currently
be used as a mapper output key - those keys have to be sorted during the
shuffle, and thus the type must be WritableComparable.

通常对于发出键,我们使用定制的可写类。
为此,您应该在driver类public class driver中定义相同的{

public static class DoubleArrayWritable extends ArrayWritable {
    public DoubleArrayWritable() {
        super(DoubleWritable.class);
    }
}

public static void main(String[] args) {

在mapper中

DoubleArrayWritable pointArray = new DoubleArrayWritable();
DoubleWritable[] data = new DoubleWritable[numAttributes];
for (int k = 0; k < numAttributes; k++) {
    data[k] = new DoubleWritable(point[k]);
}
pointArray.set(data);
context.write(new Text("Array"), pointArray);

希望这在某些方面有所帮助。

相关问题