在为自定义输入格式设置作业时,我遇到了某些错误
下面是我的代码
package com.nline_delimiter;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class NL_driver {
public static void main(String [] args) throws IOException, InterruptedException, ClassNotFoundException
{
Configuration conf=new Configuration(true);
Job job_run =new Job(conf);
job_run.setJobName("nline input format each line seperate wth delimiter");
job_run.setJarByClass(NL_driver.class);
job_run.setMapperClass(NL_mapper.class);
job_run.setReducerClass(NL_reducer.class);
job_run.setInputFormatClass(NL_inputformatter.class);;
job_run.setMapOutputKeyClass(Text.class);
job_run.setMapOutputValueClass(IntWritable.class);
job_run.setOutputKeyClass(Text.class);
job_run.setOutputValueClass(IntWritable.class);
FileInputFormat.setInputPaths(job_run,new Path("/home/hduser/input_formatter_usage.txt"));
FileOutputFormat.setOutputPath(job_run, new Path("/home/hduser/input_formatter_usage"));
job_run.waitForCompletion(true);
}
}
线路
job_run.setInputFormatClass(NL_inputformatter.class)
显示错误
nl\u inputformatter是一个自定义inputformatter类,它扩展了fileinputformat
是否需要为setinputformatclass导入某些内容,因为eclipse中的默认错误检查要求我将setinputformatclass更改为setoutformatclass,但不要求任何导入。
nl\ U inputformatter的源代码如下。
package com.nline_delimiter;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapred.InputSplit;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.RecordReader;
import org.apache.hadoop.mapred.Reporter;
public class NL_inputformatter extends FileInputFormat<Text, IntWritable>{
@Override
public RecordReader<Text, IntWritable> getRecordReader(InputSplit input,
JobConf job_run, Reporter reporter) throws IOException {
// TODO Auto-generated method stub
System.out.println("I am Inside the NL_inputformatter class");
reporter.setStatus(input.toString());
return new NL_record_reader(job_run, (FileSplit)input);
}
}
谢谢你的帮助。
1条答案
按热度按时间dw1jzc5e1#
是因为你在用
FileInputFormat
从古老的Hadoop
新的api。您必须更改导入和实现:到