org.apache.hadoop.mapred.OutputFormat.getRecordWriter()方法的使用及代码示例

x33g5p2x  于2022-01-25 转载在 其他  
字(10.1k)|赞(0)|评价(0)|浏览(84)

本文整理了Java中org.apache.hadoop.mapred.OutputFormat.getRecordWriter()方法的一些代码示例，展示了OutputFormat.getRecordWriter()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。OutputFormat.getRecordWriter()方法的具体详情如下：
包路径：org.apache.hadoop.mapred.OutputFormat
类名称：OutputFormat
方法名：getRecordWriter

OutputFormat.getRecordWriter介绍

[英]Get the RecordWriter for the given job.
[中]获取给定作业的RecordWriter。

代码示例

代码示例来源：origin: apache/hive

@Override
public org.apache.hadoop.mapred.RecordWriter<K, V> getRecordWriter(FileSystem ignored,
   JobConf job, String name, Progressable progress) throws IOException {
 return (RecordWriter<K, V>) actualOutputFormat.getRecordWriter(ignored,
        job, name, progress);
}

代码示例来源：origin: apache/drill

@Override
public org.apache.hadoop.mapred.RecordWriter<K, V> getRecordWriter(FileSystem ignored,
   JobConf job, String name, Progressable progress) throws IOException {
 return (RecordWriter<K, V>) actualOutputFormat.getRecordWriter(ignored,
        job, name, progress);
}

代码示例来源：origin: elastic/elasticsearch-hadoop

@Override
public org.apache.hadoop.mapred.RecordWriter getRecordWriter(FileSystem ignored, JobConf job, String name, Progressable progress)
    throws IOException {
  List<org.apache.hadoop.mapred.OutputFormat> formats = getOldApiFormats(job);
  List<org.apache.hadoop.mapred.RecordWriter> writers = new ArrayList<org.apache.hadoop.mapred.RecordWriter>();
  for (org.apache.hadoop.mapred.OutputFormat format : formats) {
    writers.add(format.getRecordWriter(ignored, job, name, progress));
  }
  return new MultiOldRecordWriter(writers);
}

代码示例来源：origin: apache/hive

@Override
 public org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter getHiveRecordWriter(
   JobConf jc, Path finalOutPath, Class<? extends Writable> valueClass, boolean isCompressed,
   Properties tableProperties, Progressable progress) throws IOException {
  if (actualOutputFormat instanceof HiveOutputFormat) {
   return ((HiveOutputFormat<K, V>) actualOutputFormat).getHiveRecordWriter(jc,
      finalOutPath, valueClass, isCompressed, tableProperties, progress);
  }
  FileSystem fs = finalOutPath.getFileSystem(jc);
  RecordWriter<?, ?> recordWriter = actualOutputFormat.getRecordWriter(fs, jc, null, progress);
  return new HivePassThroughRecordWriter(recordWriter);
 }
}

代码示例来源：origin: apache/drill

@Override
 public org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter getHiveRecordWriter(
   JobConf jc, Path finalOutPath, Class<? extends Writable> valueClass, boolean isCompressed,
   Properties tableProperties, Progressable progress) throws IOException {
  if (actualOutputFormat instanceof HiveOutputFormat) {
   return ((HiveOutputFormat<K, V>) actualOutputFormat).getHiveRecordWriter(jc,
      finalOutPath, valueClass, isCompressed, tableProperties, progress);
  }
  FileSystem fs = finalOutPath.getFileSystem(jc);
  RecordWriter<?, ?> recordWriter = actualOutputFormat.getRecordWriter(fs, jc, null, progress);
  return new HivePassThroughRecordWriter(recordWriter);
 }
}

代码示例来源：origin: apache/avro

private synchronized RecordWriter getRecordWriter(String namedOutput,
                         String baseFileName,
                         final Reporter reporter,Schema schema)
 throws IOException {
 RecordWriter writer = recordWriters.get(baseFileName);
 if (writer == null) {
  if (countersEnabled && reporter == null) {
   throw new IllegalArgumentException(
    "Counters are enabled, Reporter cannot be NULL");
  }
  if(schema!=null)
   conf.set(MO_PREFIX+namedOutput+".schema",schema.toString());
  JobConf jobConf = new JobConf(conf);
  jobConf.set(InternalFileOutputFormat.CONFIG_NAMED_OUTPUT, namedOutput);
  FileSystem fs = FileSystem.get(conf);
  writer = outputFormat.getRecordWriter(fs, jobConf, baseFileName, reporter);
  if (countersEnabled) {
   if (reporter == null) {
    throw new IllegalArgumentException(
     "Counters are enabled, Reporter cannot be NULL");
   }
   writer = new RecordWriterWithCounter(writer, baseFileName, reporter);
  }
  recordWriters.put(baseFileName, writer);
 }
 return writer;
}

代码示例来源：origin: apache/ignite

/**
 * @param jobConf Job configuration.
 * @param taskCtx Task context.
 * @param directWrite Direct write flag.
 * @param fileName File name.
 * @throws IOException In case of IO exception.
 */
HadoopV1OutputCollector(JobConf jobConf, HadoopTaskContext taskCtx, boolean directWrite,
  @Nullable String fileName, TaskAttemptID attempt) throws IOException {
  this.jobConf = jobConf;
  this.taskCtx = taskCtx;
  this.attempt = attempt;
  if (directWrite) {
    jobConf.set("mapreduce.task.attempt.id", attempt.toString());
    OutputFormat outFormat = jobConf.getOutputFormat();
    writer = outFormat.getRecordWriter(null, jobConf, fileName, Reporter.NULL);
  }
  else
    writer = null;
}

代码示例来源：origin: apache/hive

/**
 * Get the record writer for the job. Uses the storagehandler's OutputFormat
 * to get the record writer.
 * @param context the information about the current task.
 * @return a RecordWriter to write the output for the job.
 * @throws IOException
 */
@Override
public RecordWriter<WritableComparable<?>, HCatRecord>
getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException {
 String name = getOutputName(context);
 return new DefaultRecordWriterContainer(context,
  getBaseOutputFormat().getRecordWriter(null, new JobConf(context.getConfiguration()), name, InternalUtil.createReporter(context)));
}

代码示例来源：origin: apache/flink

/**
 * create the temporary output file for hadoop RecordWriter.
 * @param taskNumber The number of the parallel instance.
 * @param numTasks The number of parallel tasks.
 * @throws java.io.IOException
 */
@Override
public void open(int taskNumber, int numTasks) throws IOException {
  // enforce sequential open() calls
  synchronized (OPEN_MUTEX) {
    if (Integer.toString(taskNumber + 1).length() > 6) {
      throw new IOException("Task id too large.");
    }
    TaskAttemptID taskAttemptID = TaskAttemptID.forName("attempt__0000_r_"
        + String.format("%" + (6 - Integer.toString(taskNumber + 1).length()) + "s", " ").replace(" ", "0")
        + Integer.toString(taskNumber + 1)
        + "_0");
    this.jobConf.set("mapred.task.id", taskAttemptID.toString());
    this.jobConf.setInt("mapred.task.partition", taskNumber + 1);
    // for hadoop 2.2
    this.jobConf.set("mapreduce.task.attempt.id", taskAttemptID.toString());
    this.jobConf.setInt("mapreduce.task.partition", taskNumber + 1);
    this.context = new TaskAttemptContextImpl(this.jobConf, taskAttemptID);
    this.outputCommitter = this.jobConf.getOutputCommitter();
    JobContext jobContext = new JobContextImpl(this.jobConf, new JobID());
    this.outputCommitter.setupJob(jobContext);
    this.recordWriter = this.mapredOutputFormat.getRecordWriter(null, this.jobConf, Integer.toString(taskNumber + 1), new HadoopDummyProgressable());
  }
}

代码示例来源：origin: apache/flink

@Test
public void testOpen() throws Exception {
  OutputFormat<String, Long> dummyOutputFormat = mock(DummyOutputFormat.class);
  DummyOutputCommitter outputCommitter = mock(DummyOutputCommitter.class);
  JobConf jobConf = Mockito.spy(new JobConf());
  when(jobConf.getOutputCommitter()).thenReturn(outputCommitter);
  HadoopOutputFormat<String, Long> outputFormat = new HadoopOutputFormat<>(dummyOutputFormat, jobConf);
  outputFormat.open(1, 1);
  verify(jobConf, times(2)).getOutputCommitter();
  verify(outputCommitter, times(1)).setupJob(any(JobContext.class));
  verify(dummyOutputFormat, times(1)).getRecordWriter(nullable(FileSystem.class), any(JobConf.class), anyString(), any(Progressable.class));
}

代码示例来源：origin: apache/hive

getBaseOutputFormat().getRecordWriter(
  parentDir.getFileSystem(context.getConfiguration()),
  new JobConf(context.getConfiguration()),

代码示例来源：origin: Qihoo360/XLearning

jobConf.set("mapred.job.id", jobID.toString());
amClient.reportMapedTaskID(containerId, taId.toString());
RecordWriter writer = outputFormat.getRecordWriter(dfs, jobConf, "part-r", Reporter.NULL);
String xlearningStreamResultLine;
while ((xlearningStreamResultLine = reader.readLine()) != null) {

代码示例来源：origin: apache/avro

@SuppressWarnings({"unchecked", "deprecation"})
 public RecordWriter<Object, Object> getRecordWriter(FileSystem fs,JobConf job, String baseFileName, Progressable arg3) throws IOException {
 String nameOutput = job.get(CONFIG_NAMED_OUTPUT, null);
 String fileName = getUniqueName(job, baseFileName);
 Schema schema = null;
 String schemastr = job.get(MO_PREFIX+nameOutput+".schema",null);
 if (schemastr!=null)
  schema = Schema.parse(schemastr);
 JobConf outputConf = new JobConf(job);
 outputConf.setOutputFormat(getNamedOutputFormatClass(job, nameOutput));
 boolean isMapOnly = job.getNumReduceTasks() == 0;
 if (schema != null) {
  if (isMapOnly)
   AvroJob.setMapOutputSchema(outputConf, schema);
  else
   AvroJob.setOutputSchema(outputConf, schema);
 }
 OutputFormat outputFormat = outputConf.getOutputFormat();
 return outputFormat.getRecordWriter(fs, outputConf, fileName, arg3);
 }
}

代码示例来源：origin: apache/hive

baseOF.getRecordWriter(parentDir.getFileSystem(currTaskContext.getConfiguration()),
  currTaskContext.getJobConf(), childPath.toString(),
  InternalUtil.createReporter(currTaskContext));

代码示例来源：origin: apache/hive

conf.setInt("mapred.max.split.size", 50);
RecordWriter writer =
  outFormat.getRecordWriter(fs, conf, testFilePath.toString(),
    Reporter.NULL);
writer.write(NullWritable.get(),

代码示例来源：origin: apache/hive

conf.setInt("mapred.max.split.size", 50);
RecordWriter writer =
  outFormat.getRecordWriter(fs, conf, testFilePath.toString(),
    Reporter.NULL);
writer.write(NullWritable.get(),

代码示例来源：origin: apache/hive

OutputFormat<?, ?> outFormat = new OrcOutputFormat();
RecordWriter writer =
  outFormat.getRecordWriter(fs, conf, testFilePath.toString(),
    Reporter.NULL);
writer.write(NullWritable.get(),

代码示例来源：origin: ch.cern.hadoop/hadoop-mapreduce-client-core

@SuppressWarnings("unchecked")
 private void createRecordWriter() throws IOException {
  FileSystem fs = FileSystem.get(job);
  rawWriter = of.getRecordWriter(fs, job, name, progress);
 }  
}

代码示例来源：origin: io.prestosql.hadoop/hadoop-apache

@SuppressWarnings("unchecked")
 private void createRecordWriter() throws IOException {
  FileSystem fs = FileSystem.get(job);
  rawWriter = of.getRecordWriter(fs, job, name, progress);
 }  
}

代码示例来源：origin: org.elasticsearch/elasticsearch-spark

@Override
public org.apache.hadoop.mapred.RecordWriter getRecordWriter(FileSystem ignored, JobConf job, String name, Progressable progress)
    throws IOException {
  List<org.apache.hadoop.mapred.OutputFormat> formats = getOldApiFormats(job);
  List<org.apache.hadoop.mapred.RecordWriter> writers = new ArrayList<org.apache.hadoop.mapred.RecordWriter>();
  for (org.apache.hadoop.mapred.OutputFormat format : formats) {
    writers.add(format.getRecordWriter(ignored, job, name, progress));
  }
  return new MultiOldRecordWriter(writers);
}

内容来源于网络，如有侵权，请联系作者删除！

OutputFormat

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com