javahadoopmapper如何发送多个值

ldioqlga  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(181)

我的Map程序需要发送以下元组:

<custID,prodID,rate>

我想把custid作为键发送给reducer,把prodid和rate作为值一起发送,因为reduce阶段需要它们。哪种方法最好?

public void map(Object key, Text value, Context context) 
        throws IOException, InterruptedException {

    String[] col = value.toString().split(",");
    custID.set(col[0]);
    data.set(col[1] + "," + col[2]);
    context.write(custID, data);
}

public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {

    for (Text val : values) {
        String[] temp = val.toString().split(",");
        Text rate = new Text(temp[1]);
        result.set(rate);
        context.write(key, result);
    }
}
zhte4eai

zhte4eai1#

最好的方法是编写定制的可写内容
这是双重价值。您可以将其更改为文本或字符串

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.Writable;

/**
 * @author Unmesha SreeVeni U.B
 *
 */
public class TwovalueWritable implements Writable {
    private double first;
    private double second;

    public  TwovalueWritable() {
        set(first, second);
    }
    public  TwovalueWritable(double first, double second) {
        set(first, second);
    }
    public void set(double first, double second) {
        this.first = first;
        this.second = second;
    }
    public double getFirst() {
        return first;
    }
    public double getSecond() {
        return second;
    }
    @Override
    public void write(DataOutput out) throws IOException {
        out.writeDouble(first);
        out.writeDouble(second);
    }
    @Override
    public void readFields(DataInput in) throws IOException {
        first = in.readDouble();
        second = in.readDouble();
    }

    /* (non-Javadoc)
     * @see java.lang.Object#hashCode()
     */
    @Override
    public int hashCode() {
        final int prime = 31;
        int result = 1;
        long temp;
        temp = Double.doubleToLongBits(first);
        result = prime * result + (int) (temp ^ (temp >>> 32));
        temp = Double.doubleToLongBits(second);
        result = prime * result + (int) (temp ^ (temp >>> 32));
        return result;
    }
    /* (non-Javadoc)
     * @see java.lang.Object#equals(java.lang.Object)
     */
    @Override
    public boolean equals(Object obj) {
        if (this == obj) {
            return true;
        }
        if (obj == null) {
            return false;
        }
        if (!(obj instanceof TwovalueWritable)) {
            return false;
        }
        TwovalueWritable other = (TwovalueWritable) obj;
        if (Double.doubleToLongBits(first) != Double
                .doubleToLongBits(other.first)) {
            return false;
        }
        if (Double.doubleToLongBits(second) != Double
                .doubleToLongBits(other.second)) {
            return false;
        }
        return true;
    }
    @Override
    public String toString() {
        return first + "," + second;
    }
}

从mapper你可以发射出

context.write(key,new TwovalueWritable(prodID,rate));

希望这有帮助。

hjqgdpho

hjqgdpho2#

我能想到的最简单的方法就是把它们合并成一个字符串:

output.collect(custID, prodID + "," + rate);

然后,在减速器上分开。
如果你从你的Map器中发布更多的代码,也许我们可以给出一个更好的例子。
更新:也就是说,你要求最好的方法。最正确的方法可能是创建一个单独的类分组 prodID 以及 rate 一起送过去。

相关问题