org.apache.hadoop.tools.DistCp.configureOutputFormat()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(2.4k)|赞(0)|评价(0)|浏览(122)

本文整理了Java中org.apache.hadoop.tools.DistCp.configureOutputFormat()方法的一些代码示例,展示了DistCp.configureOutputFormat()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。DistCp.configureOutputFormat()方法的具体详情如下:
包路径:org.apache.hadoop.tools.DistCp
类名称:DistCp
方法名:configureOutputFormat

DistCp.configureOutputFormat介绍

[英]Setup output format appropriately
[中]适当设置输出格式

代码示例

代码示例来源:origin: org.apache.hadoop/hadoop-distcp

/**
 * Create Job object for submitting it, with all the configuration
 *
 * @return Reference to job object.
 * @throws IOException - Exception if any
 */
private Job createJob() throws IOException {
 String jobName = "distcp";
 String userChosenName = getConf().get(JobContext.JOB_NAME);
 if (userChosenName != null)
  jobName += ": " + userChosenName;
 Job job = Job.getInstance(getConf());
 job.setJobName(jobName);
 job.setInputFormatClass(DistCpUtils.getStrategy(getConf(), context));
 job.setJarByClass(CopyMapper.class);
 configureOutputFormat(job);
 job.setMapperClass(CopyMapper.class);
 job.setNumReduceTasks(0);
 job.setMapOutputKeyClass(Text.class);
 job.setMapOutputValueClass(Text.class);
 job.setOutputFormatClass(CopyOutputFormat.class);
 job.getConfiguration().set(JobContext.MAP_SPECULATIVE, "false");
 job.getConfiguration().set(JobContext.NUM_MAPS,
        String.valueOf(context.getMaxMaps()));
 context.appendToConf(job.getConfiguration());
 return job;
}

代码示例来源:origin: ch.cern.hadoop/hadoop-distcp

/**
 * Create Job object for submitting it, with all the configuration
 *
 * @return Reference to job object.
 * @throws IOException - Exception if any
 */
private Job createJob() throws IOException {
 String jobName = "distcp";
 String userChosenName = getConf().get(JobContext.JOB_NAME);
 if (userChosenName != null)
  jobName += ": " + userChosenName;
 Job job = Job.getInstance(getConf());
 job.setJobName(jobName);
 job.setInputFormatClass(DistCpUtils.getStrategy(getConf(), inputOptions));
 job.setJarByClass(CopyMapper.class);
 configureOutputFormat(job);
 job.setMapperClass(CopyMapper.class);
 job.setNumReduceTasks(0);
 job.setMapOutputKeyClass(Text.class);
 job.setMapOutputValueClass(Text.class);
 job.setOutputFormatClass(CopyOutputFormat.class);
 job.getConfiguration().set(JobContext.MAP_SPECULATIVE, "false");
 job.getConfiguration().set(JobContext.NUM_MAPS,
        String.valueOf(inputOptions.getMaxMaps()));
 if (inputOptions.getSslConfigurationFile() != null) {
  setupSSLConfig(job);
 }
 inputOptions.appendToConf(job.getConfiguration());
 return job;
}

相关文章