org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.addNamedOutput()方法的使用及代码示例

x33g5p2x  于2022-01-25 转载在 其他  
字(12.6k)|赞(0)|评价(0)|浏览(117)

本文整理了Java中org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.addNamedOutput()方法的一些代码示例，展示了MultipleOutputs.addNamedOutput()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。MultipleOutputs.addNamedOutput()方法的具体详情如下：
包路径：org.apache.hadoop.mapreduce.lib.output.MultipleOutputs
类名称：MultipleOutputs
方法名：addNamedOutput

MultipleOutputs.addNamedOutput介绍

[英]Adds a named output for the job.
[中]为作业添加命名输出。

代码示例

代码示例来源：origin: apache/kylin

private void setupReducer(Path output, int numberOfReducers) throws IOException {
  job.setReducerClass(UHCDictionaryReducer.class);
  job.setPartitionerClass(UHCDictionaryPartitioner.class);
  job.setNumReduceTasks(numberOfReducers);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_DICT, SequenceFileOutputFormat.class, NullWritable.class, ArrayPrimitiveWritable.class);
  FileOutputFormat.setOutputPath(job, output);
  job.getConfiguration().set(BatchConstants.CFG_OUTPUT_PATH, output.toString());
  //prevent to create zero-sized default output
  LazyOutputFormat.setOutputFormatClass(job, SequenceFileOutputFormat.class);
  deletePath(job.getConfiguration(), output);
}

代码示例来源：origin: apache/kylin

private void setupReducer(Path output, CubeSegment cubeSeg)
    throws IOException {
  FactDistinctColumnsReducerMapping reducerMapping = new FactDistinctColumnsReducerMapping(cubeSeg.getCubeInstance());
  int numberOfReducers = reducerMapping.getTotalReducerNum();
  logger.info("{} has reducers {}.", this.getClass().getName(), numberOfReducers);
  if (numberOfReducers > 250) {
    throw new IllegalArgumentException(
        "The max reducer number for FactDistinctColumnsJob is 250, but now it is "
            + numberOfReducers
            + ", decrease 'kylin.engine.mr.uhc-reducer-count'");
  }
  job.setReducerClass(FactDistinctColumnsReducer.class);
  job.setPartitionerClass(FactDistinctColumnPartitioner.class);
  job.setNumReduceTasks(numberOfReducers);
  // make each reducer output to respective dir
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_COLUMN, SequenceFileOutputFormat.class, NullWritable.class, Text.class);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_DICT, SequenceFileOutputFormat.class, NullWritable.class, ArrayPrimitiveWritable.class);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_STATISTICS, SequenceFileOutputFormat.class, LongWritable.class, BytesWritable.class);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_PARTITION, TextOutputFormat.class, NullWritable.class, LongWritable.class);
  FileOutputFormat.setOutputPath(job, output);
  job.getConfiguration().set(BatchConstants.CFG_OUTPUT_PATH, output.toString());
  // prevent to create zero-sized default output
  LazyOutputFormat.setOutputFormatClass(job, SequenceFileOutputFormat.class);
  deletePath(job.getConfiguration(), output);
}

代码示例来源：origin: apache/kylin

MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_COLUMN, SequenceFileOutputFormat.class,
    NullWritable.class, Text.class);
MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_DICT, SequenceFileOutputFormat.class,
    NullWritable.class, ArrayPrimitiveWritable.class);
MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_STATISTICS, SequenceFileOutputFormat.class,
    LongWritable.class, BytesWritable.class);
MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_PARTITION, TextOutputFormat.class,
    NullWritable.class, LongWritable.class);

代码示例来源：origin: org.openimaj/core-hadoop

@Override
  public Job stage(Path[] inputs, Path output, Configuration conf) throws Exception {

    final Job job = super.stage(inputs, output, conf);
    job.setOutputFormatClass(NullOutputFormat.class);
    MultipleOutputs.addNamedOutput(job, "text", TextOutputFormat.class, NullWritable.class, Text.class);
    return job;
  }
}

代码示例来源：origin: openimaj/openimaj

@Override
  public Job stage(Path[] inputs, Path output, Configuration conf) throws Exception {

    final Job job = super.stage(inputs, output, conf);
    job.setOutputFormatClass(NullOutputFormat.class);
    MultipleOutputs.addNamedOutput(job, "text", TextOutputFormat.class, NullWritable.class, Text.class);
    return job;
  }
}

代码示例来源：origin: apache/incubator-rya

/**
 * Set up a MapReduce job to output human-readable text.
 */
protected void configureTextOutput(String destination) {
  Path outPath;
  outPath = MRReasoningUtils.getOutputPath(job.getConfiguration(), destination);
  TextOutputFormat.setOutputPath(job, outPath);
  LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);
  MultipleOutputs.addNamedOutput(job, MRReasoningUtils.INTERMEDIATE_OUT,
    TextOutputFormat.class, NullWritable.class, Text.class);
  MultipleOutputs.addNamedOutput(job, MRReasoningUtils.TERMINAL_OUT,
    TextOutputFormat.class, NullWritable.class, Text.class);
  MultipleOutputs.addNamedOutput(job, MRReasoningUtils.SCHEMA_OUT,
    TextOutputFormat.class, NullWritable.class, Text.class);
  MultipleOutputs.addNamedOutput(job, MRReasoningUtils.INCONSISTENT_OUT,
    TextOutputFormat.class, NullWritable.class, Text.class);
  MultipleOutputs.addNamedOutput(job, MRReasoningUtils.DEBUG_OUT,
    TextOutputFormat.class, Text.class, Text.class);
  MultipleOutputs.setCountersEnabled(job, true);
}

代码示例来源：origin: org.apache.kylin/kylin-engine-mr

private void setupReducer(Path output, int numberOfReducers) throws IOException {
  job.setReducerClass(UHCDictionaryReducer.class);
  job.setPartitionerClass(UHCDictionaryPartitioner.class);
  job.setNumReduceTasks(numberOfReducers);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_DICT, SequenceFileOutputFormat.class, NullWritable.class, ArrayPrimitiveWritable.class);
  FileOutputFormat.setOutputPath(job, output);
  job.getConfiguration().set(BatchConstants.CFG_OUTPUT_PATH, output.toString());
  //prevent to create zero-sized default output
  LazyOutputFormat.setOutputFormatClass(job, SequenceFileOutputFormat.class);
  deletePath(job.getConfiguration(), output);
}

代码示例来源：origin: apache/incubator-rya

/**
 * Set up the MapReduce job to output a schema (TBox).
 */
protected void configureSchemaOutput() {
  Path outPath = MRReasoningUtils.getSchemaPath(job.getConfiguration());
  SequenceFileOutputFormat.setOutputPath(job, outPath);
  job.setOutputFormatClass(SequenceFileOutputFormat.class);
  job.setOutputKeyClass(NullWritable.class);
  job.setOutputValueClass(SchemaWritable.class);
  LazyOutputFormat.setOutputFormatClass(job, SequenceFileOutputFormat.class);
  MultipleOutputs.addNamedOutput(job, "schemaobj",
    SequenceFileOutputFormat.class, NullWritable.class, SchemaWritable.class);
  MultipleOutputs.addNamedOutput(job, MRReasoningUtils.DEBUG_OUT,
    TextOutputFormat.class, Text.class, Text.class);
  MultipleOutputs.setCountersEnabled(job, true);
}

代码示例来源：origin: org.apache.kylin/kylin-engine-mr

private void setupReducer(Path output, CubeSegment cubeSeg)
    throws IOException {
  FactDistinctColumnsReducerMapping reducerMapping = new FactDistinctColumnsReducerMapping(cubeSeg.getCubeInstance());
  int numberOfReducers = reducerMapping.getTotalReducerNum();
  logger.info("{} has reducers {}.", this.getClass().getName(), numberOfReducers);
  if (numberOfReducers > 250) {
    throw new IllegalArgumentException(
        "The max reducer number for FactDistinctColumnsJob is 250, but now it is "
            + numberOfReducers
            + ", decrease 'kylin.engine.mr.uhc-reducer-count'");
  }
  job.setReducerClass(FactDistinctColumnsReducer.class);
  job.setPartitionerClass(FactDistinctColumnPartitioner.class);
  job.setNumReduceTasks(numberOfReducers);
  // make each reducer output to respective dir
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_COLUMN, SequenceFileOutputFormat.class, NullWritable.class, Text.class);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_DICT, SequenceFileOutputFormat.class, NullWritable.class, ArrayPrimitiveWritable.class);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_STATISTICS, SequenceFileOutputFormat.class, LongWritable.class, BytesWritable.class);
  MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_PARTITION, TextOutputFormat.class, NullWritable.class, LongWritable.class);
  FileOutputFormat.setOutputPath(job, output);
  job.getConfiguration().set(BatchConstants.CFG_OUTPUT_PATH, output.toString());
  // prevent to create zero-sized default output
  LazyOutputFormat.setOutputFormatClass(job, SequenceFileOutputFormat.class);
  deletePath(job.getConfiguration(), output);
}

代码示例来源：origin: apache/incubator-rya

MultipleOutputs.addNamedOutput(job, MRReasoningUtils.INTERMEDIATE_OUT,
  SequenceFileOutputFormat.class, Fact.class, NullWritable.class);
MultipleOutputs.addNamedOutput(job, MRReasoningUtils.TERMINAL_OUT,
  SequenceFileOutputFormat.class, Fact.class, NullWritable.class);
MultipleOutputs.addNamedOutput(job, MRReasoningUtils.SCHEMA_OUT,
  SequenceFileOutputFormat.class, Fact.class, NullWritable.class);
MultipleOutputs.addNamedOutput(job, MRReasoningUtils.INCONSISTENT_OUT,
  SequenceFileOutputFormat.class, Derivation.class, NullWritable.class);
MultipleOutputs.setCountersEnabled(job, true);
MultipleOutputs.addNamedOutput(job, MRReasoningUtils.DEBUG_OUT,
  TextOutputFormat.class, Text.class, Text.class);

代码示例来源：origin: pl.edu.icm.coansys/coansys-io-input

SequenceFileOutputFormat.setOutputPath(job, new Path(outputDir));
MultipleOutputs.addNamedOutput(job, FAMILY_METADATA_DOCUMENT_QUALIFIER_PROTO, SequenceFileOutputFormat.class, BytesWritable.class, BytesWritable.class);
MultipleOutputs.addNamedOutput(job, FAMILY_CONTENT_QUALIFIER_PROTO, SequenceFileOutputFormat.class, BytesWritable.class, BytesWritable.class);
MultipleOutputs.addNamedOutput(job, "dproto", SequenceFileOutputFormat.class, BytesWritable.class, BytesWritable.class);

代码示例来源：origin: geftimov/hadoop-map-reduce-patterns

MultipleOutputs.addNamedOutput(job, "bins", TextOutputFormat.class,
    Text.class, NullWritable.class);

代码示例来源：origin: openimaj/openimaj

@Override
public int run(String[] args) throws Exception {
  final CmdLineParser parser = new CmdLineParser(this);
  try {
    parser.parseArgument(args);
  } catch (final CmdLineException e) {
    System.err.println(e.getMessage());
    System.err.println("Usage: hadoop jar HadoopImageIndexer.jar [options]");
    parser.printUsage(System.err);
    return -1;
  }
  final Path[] paths = SequenceFileUtility.getFilePaths(input, "part");
  final Path outputPath = new Path(output);
  if (outputPath.getFileSystem(this.getConf()).exists(outputPath) && replace)
    outputPath.getFileSystem(this.getConf()).delete(outputPath, true);
  final Job job = TextBytesJobUtil.createJob(paths, outputPath, null, this.getConf());
  job.setJarByClass(this.getClass());
  job.setMapperClass(PqPcaVladMapper.class);
  job.setNumReduceTasks(0);
  MultipleOutputs.addNamedOutput(job, "pcavlad", SequenceFileOutputFormat.class, Text.class, BytesWritable.class);
  DistributedCache.createSymlink(job.getConfiguration());
  DistributedCache.addCacheFile(new URI(indexerData + "#vlad-data.bin"), job.getConfiguration());
  SequenceFileOutputFormat.setCompressOutput(job, !dontcompress);
  job.waitForCompletion(true);
  return 0;
}

代码示例来源：origin: ShifuML/shifu

MultipleOutputs.addNamedOutput(job, Constants.SHIFU_VARSELECT_SE_OUTPUT_NAME, TextOutputFormat.class,
    Text.class, Text.class);
return job;

代码示例来源：origin: hortonworks/hive-testbench

MultipleOutputs.addNamedOutput(job, "text", 
 TextOutputFormat.class, LongWritable.class, Text.class);

代码示例来源：origin: thinkaurelius/faunus

MultipleOutputs.addNamedOutput(job, Tokens.SIDEEFFECT, this.graph.getSideEffectOutputFormat(), job.getOutputKeyClass(), job.getOutputKeyClass());
  MultipleOutputs.addNamedOutput(job, Tokens.GRAPH, this.graph.getGraphOutputFormat(), NullWritable.class, FaunusVertex.class);
} else {
  LazyOutputFormat.setOutputFormatClass(job, INTERMEDIATE_OUTPUT_FORMAT);
  MultipleOutputs.addNamedOutput(job, Tokens.SIDEEFFECT, this.graph.getSideEffectOutputFormat(), job.getOutputKeyClass(), job.getOutputKeyClass());
  MultipleOutputs.addNamedOutput(job, Tokens.GRAPH, INTERMEDIATE_OUTPUT_FORMAT, NullWritable.class, FaunusVertex.class);

代码示例来源：origin: ShifuML/shifu

new Path(super.modelConfig.getDataSetRawPath())));
MultipleOutputs.addNamedOutput(job, Constants.POST_TRAIN_OUTPUT_SCORE, TextOutputFormat.class,
    NullWritable.class, Text.class);

代码示例来源：origin: com.conversantmedia/mara-core

name = (String)evaluateExpression(name);
if (!configured.contains(name)) {
  MultipleOutputs.addNamedOutput(job, name, namedOut.format(), keyClass, valueClass);
  MultipleOutputs.setCountersEnabled(job, namedOut.countersEnabled());
  configured.add(name);

代码示例来源：origin: org.apache.kylin/kylin-engine-spark

MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_COLUMN, SequenceFileOutputFormat.class,
    NullWritable.class, Text.class);
MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_DICT, SequenceFileOutputFormat.class,
    NullWritable.class, ArrayPrimitiveWritable.class);
MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_STATISTICS, SequenceFileOutputFormat.class,
    LongWritable.class, BytesWritable.class);
MultipleOutputs.addNamedOutput(job, BatchConstants.CFG_OUTPUT_PARTITION, TextOutputFormat.class,
    NullWritable.class, LongWritable.class);

代码示例来源：origin: ch.cern.hadoop/hadoop-mapreduce-client-jobclient

MultipleOutputs.addNamedOutput(job, TEXT, TextOutputFormat.class,
 Long.class, String.class);

内容来源于网络，如有侵权，请联系作者删除！

MultipleOutputs

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com