java—如何在不使用oozie的情况下创建hadoop作业链

x7rlezfr  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(240)

我想创建一个由三个hadoop作业组成的链,其中一个作业的输出作为第二个作业的输入,以此类推。我不想使用oozie。
我编写了以下代码来实现它:-

public class TfIdf {
    public static void main(String args[]) throws IOException, InterruptedException, ClassNotFoundException
    {
        TfIdf tfIdf = new TfIdf();
        tfIdf.runWordCount();
        tfIdf.runDocWordCount();
        tfIdf.TFIDFComputation();
    }

    public void runWordCount() throws IOException, InterruptedException, ClassNotFoundException
    {
        Job job = new Job();

        job.setJarByClass(TfIdf.class);
        job.setJobName("Word Count calculation");

        job.setMapperClass(WordFrequencyMapper.class);
        job.setReducerClass(WordFrequencyReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.setInputPaths(job, new Path("input"));
        FileOutputFormat.setOutputPath(job, new Path("ouput"));

        job.waitForCompletion(true);
    }

    public void runDocWordCount() throws IOException, InterruptedException, ClassNotFoundException
    {
        Job job = new Job();

        job.setJarByClass(TfIdf.class);
        job.setJobName("Word Doc count calculation");

        job.setMapperClass(WordCountDocMapper.class);
        job.setReducerClass(WordCountDocReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        FileInputFormat.setInputPaths(job, new Path("output"));
        FileOutputFormat.setOutputPath(job, new Path("ouput_job2"));

        job.waitForCompletion(true);
    }

    public void TFIDFComputation() throws IOException, InterruptedException, ClassNotFoundException
    {
        Job job = new Job();

        job.setJarByClass(TfIdf.class);
        job.setJobName("TFIDF calculation");

        job.setMapperClass(TFIDFMapper.class);
        job.setReducerClass(TFIDFReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        FileInputFormat.setInputPaths(job, new Path("output_job2"));
        FileOutputFormat.setOutputPath(job, new Path("ouput_job3"));

        job.waitForCompletion(true);
    }
}

但是我得到了一个错误:

Input path does not exist: hdfs://localhost.localdomain:8020/user/cloudera/output

有人能帮我解决这个问题吗?

pieyvz9o

pieyvz9o1#

这个答案来得有点晚,但是。。。你的名字只是简单的输入错误。您已经将第一个作业的输出写入dir“output”,第二个作业正在“output”中查找它。

相关问题