我已经在mac上安装了hadoopv2.7,并且可以启动hadoop守护进程。
我想用eclipse编写mr程序,我需要一些帮助来获取eclipse上的hadoop,我想知道要添加的jar文件和基本设置指南
下面是我的驱动程序类代码,我无法执行它
public class MyJobDriver extends Configured implements Tool {
@Override
public int run(String[] args) throws Exception {
Configuration conf = getConf();
JobConf job = new JobConf(conf, MyJobDriver.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setJobName("Patent");
job.setMapperClass(InverseMapper.class);
//Input Split consist two values separated by ","
//K1 and V1 type is Text
job.setInputFormat(KeyValueTextInputFormat.class);
job.set("key.value.separator.in.input.line",",");//Everything before the separator is the key and after is the value
job.setOutputFormat(TextOutputFormat.class);//Key and value written as string and separated by tab(default)
//when k1 and k2 are od same type and V1 and V2 are of same type
//we can skip job.setMapOutputKeyClass() and job.setMapOutputValueClass()
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
//jobClient communicates with the JobTrackers to start job across clusters
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
MyJobDriver driver = new MyJobDriver();
System.out.println("Calling the run method");
int exitCode = ToolRunner.run(driver, args);
System.exit(exitCode);
}
1条答案
按热度按时间31moq8wy1#
跟踪和检索必要的jar文件(有很多)太麻烦了。而是在eclipse中创建一个maven项目,并添加必要的依赖项,如这里所述https://hadoopi.wordpress.com/2013/05/25/setup-maven-project-for-hadoop-in-5mn/