我正在尝试使用mapreduce将文件中的所有数字相加,该文件包含以空格分隔的数字,并且包含在多行中

rlcwz9us  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(220)

我的输出出错了。输入文件是:
1 2 3 4
5 4 3 2
输出应为key:sum value:24
mapreduce生成的输出:key:sum value:34
我在ubuntu14.04中使用openjdk7来运行jar文件,而jar文件是在eclipsejuna中创建的,java版本是oraclejdk7来编译它。数字河.java
Package 编号sum;

import java.io.*;
//import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
//import org.apache.hadoop.mapreduce.Mapper;
//import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class NumberDriver {

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        // TODO Auto-generated method stub
            Configuration conf=new Configuration();
            String[] otherArgs=new GenericOptionsParser(conf,args).getRemainingArgs();
            if(otherArgs.length!=2)
            {
                System.err.println("Error");
                System.exit(2);
            }
            Job job=new Job(conf, "number sum");
            job.setJarByClass(NumberDriver.class);
            job.setMapperClass(NumberMapper.class);
            job.setReducerClass(NumberReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
            FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
            System.exit(job.waitForCompletion(true)?0:1);
    }

}

数字Map.java

package numbersum;
import java.io.*;
import java.util.StringTokenizer;

//import org.apache.hadoop.conf.Configuration;
//import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
//import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
//import org.apache.hadoop.mapreduce.Reducer;
//import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
//import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
//import org.apache.hadoop.util.GenericOptionsParser;
//import org.hsqldb.Tokenizer;

public class NumberMapper extends Mapper <LongWritable, Text, Text, IntWritable> 
    {
        int sum;
        public void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException
        {
            StringTokenizer itr=new StringTokenizer(value.toString());
            while(itr.hasMoreTokens())
            {
                sum+=Integer.parseInt(itr.nextToken());
            }
            context.write(new Text("sum"),new IntWritable(sum));
        }
    }

数字减速机.java

package numbersum;
import java.io.*;
//import java.util.StringTokenizer;

//import org.apache.hadoop.conf.Configuration;
//import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
//import org.apache.hadoop.mapreduce.Job;
//import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
//import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
//import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
//import org.apache.hadoop.util.GenericOptionsParser;

public class NumberReducer extends Reducer <Text, IntWritable, Text, IntWritable>
    {
        public void reduce(Text key,Iterable<IntWritable> values, Context context)throws IOException, InterruptedException
        {
            int sum=0;
            for(IntWritable value:values)
                {
                    sum+=value.get();
                }
            context.write(key,new IntWritable(sum));
        }
    }
kokeuurv

kokeuurv1#

我想你忘了设置 sum0 开始时 map 功能:

public void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException
{
    sum = 0;
...
yacmzcpb

yacmzcpb2#

我最好的猜测是:

int sum; // <-- Why a class member?
    public void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException
    {
        int sum = 0; //Why not here?
        StringTokenizer itr=new StringTokenizer(value.toString());

猜测推理:第一张Map:1+2+3+4=10第二张Map:(10+2+3+4+5=34
..意味着,之前的值被保留。

相关问题