我在hadoop上运行一个作业。例行公事。
然而,我的工作是停留在Map100%减少67%。map reduce任务状态为“reduce>reduce”。
我的reduce()方法:
public static class MyReducer extends Reducer<DoubleWritable, Text, DoubleWritable, Text> {
public void reduce(DoubleWritable key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
Text res = new Text("");
for(Text value : values){
if(!value.equals("")){
res.set(res.toString() + value.toString() + " ");
}
}
context.write(key, res);
}
}
map()方法:
public static class TokenizerMapper extends Mapper<Object, Text, DoubleWritable, Text>{
public void map(Object key, Text value, Context context) throws IOException, InterruptedException{
StringTokenizer input = new StringTokenizer(value.toString());
input.nextToken();
Text page = new Text(input.nextToken());
DoubleWritable vue = new DoubleWritable(Double.parseDouble(input.nextToken()) );
context.write(vue, page);
}
}
减少任务日志:
2014-10-27 10:15:37,284 DEBUG [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: [ 29830 29825 ]
2014-10-27 10:15:37,285 DEBUG [IPC Parameter Sending Thread #0] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 sending #614
2014-10-27 10:15:37,287 DEBUG [IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 got value #614
2014-10-27 10:15:37,288 DEBUG [communication thread] org.apache.hadoop.ipc.RPC: Call: statusUpdate 3
2014-10-27 10:15:40,296 DEBUG [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: [ 29830 29825 ]
2014-10-27 10:15:40,296 DEBUG [IPC Parameter Sending Thread #0] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 sending #615
2014-10-27 10:15:40,298 DEBUG [IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:51302 from job_1414399407473_0001 got value #615
2014-10-27 10:15:40,298 DEBUG [communication thread] org.apache.hadoop.ipc.RPC: Call: statusUpdate 2
暂无答案!
目前还没有任何答案,快来回答吧!