hadoopmapreduce输出最大

yeotifhr  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(292)

我目前正在使用eclipse和hadoop创建一个Map器和reducer来查找航空公司数据集的最大总成本。
所以总成本是小数,航空公司是文本。
我使用的数据集可以在以下Web链接中找到:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/236265/dft-flights-data-2011.csv
在hadoop中导出jar文件时,得到以下消息:ls:“output”:没有这样的文件或目录。
有人能帮我改正密码吗?
我的代码在下面。
Map器:

package org.myorg;

import java.io.IOException;

import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MaxTotalCostMapper extends Mapper<LongWritable, Text, Text, DoubleWritable>
{
    private final static DoubleWritable totalcostWritable = new DoubleWritable(0);
    private Text AirCarrier = new Text();

    @Override
    public void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException
    {
        String[] line = value.toString().split(",");
        AirCarrier.set(line[8]);
        double totalcost = Double.parseDouble(line[2].trim());
        totalcostWritable.set(totalcost);
        context.write(AirCarrier, totalcostWritable);
    }
}

减速器:

package org.myorg;

import java.io.IOException;
import java.util.ArrayList;

import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MaxTotalCostReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable>
{
    ArrayList<Double> totalcostList = new ArrayList<Double>();

    @Override
    public void reduce(Text key, Iterable<DoubleWritable> values, Context context)
            throws IOException, InterruptedException
    {
        double maxValue=0.0;
        for (DoubleWritable value : values) 
        {
            maxValue = Math.max(maxValue, value.get());
        }
        context.write(key, new DoubleWritable(maxValue));
    }
}

主要内容:

package org.myorg;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class MaxTotalCost
{
    public static void main(String[] args) throws Exception
    {
        Configuration conf = new Configuration();
        if (args.length != 2)
        {
            System.err.println("Usage: MaxTotalCost<input path><output path>");
            System.exit(-1);
        }

        Job job;
        job=Job.getInstance(conf, "Max Total Cost");
        job.setJarByClass(MaxTotalCost.class);

        FileInputFormat.addInputPath(job, new Path(args[1]));
        FileOutputFormat.setOutputPath(job, new Path(args[2]));

        job.setMapperClass(MaxTotalCostMapper.class);
        job.setReducerClass(MaxTotalCostReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(DoubleWritable.class);

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}
cfh9epnr

cfh9epnr1#

ls:“output”:没有这样的文件或目录
您没有hdfs用户目录。你的代码没有进入Map器或缩减器。这种错误通常出现在作业中

FileOutputFormat.setOutputPath(job, new Path(args[2]));

运行 hdfs dfs -ls ,查看是否有错误。如果是,请在下面创建一个目录 /user 与当前用户匹配的。
否则,将输出目录更改为 /tmp/max

相关问题