org.apache.beam.sdk.Pipeline.begin()方法的使用及代码示例

x33g5p2x  于2022-01-26 转载在 其他  
字(8.7k)|赞(0)|评价(0)|浏览(116)

本文整理了Java中org.apache.beam.sdk.Pipeline.begin()方法的一些代码示例,展示了Pipeline.begin()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Pipeline.begin()方法的具体详情如下:
包路径:org.apache.beam.sdk.Pipeline
类名称:Pipeline
方法名:begin

Pipeline.begin介绍

[英]Returns a PBegin owned by this Pipeline. This serves as the input of a root PTransform such as Read or Create.
[中]返回此管道拥有的PBegin。这用作根pttransform(如Read或Create)的输入。

代码示例

代码示例来源:origin: org.apache.beam/beam-sdks-java-core

/**
 * Like {@link #apply(String, PTransform)} but the transform node in the {@link Pipeline} graph
 * will be named according to {@link PTransform#getName}.
 *
 * @see #apply(String, PTransform)
 */
public <OutputT extends POutput> OutputT apply(PTransform<? super PBegin, OutputT> root) {
 return begin().apply(root);
}

代码示例来源:origin: org.apache.beam/beam-sdks-java-core

/**
 * Adds a root {@link PTransform}, such as {@link Read} or {@link Create}, to this {@link
 * Pipeline}.
 *
 * <p>The node in the {@link Pipeline} graph will use the provided {@code name}. This name is used
 * in various places, including the monitoring UI, logging, and to stably identify this node in
 * the {@link Pipeline} graph upon update.
 *
 * <p>Alias for {@code begin().apply(name, root)}.
 */
public <OutputT extends POutput> OutputT apply(
  String name, PTransform<? super PBegin, OutputT> root) {
 return begin().apply(name, root);
}

代码示例来源:origin: org.apache.beam/beam-runners-direct-java

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, PTransform<PBegin, PCollection<T>>> transform) {
 try {
  return PTransformReplacement.of(
    transform.getPipeline().begin(),
    new DirectTestStream<>(runner, TestStreamTranslation.getTestStream(transform)));
 } catch (IOException exc) {
  throw new RuntimeException(
    String.format(
      "Transform could not be converted to %s", TestStream.class.getSimpleName()),
    exc);
 }
}

代码示例来源:origin: org.apache.beam/beam-sdks-java-extensions-sql

@Override
 public PCollection<Row> expand(PCollectionList<Row> input) {
  checkArgument(
    input.size() == 0,
    "Should not have received input for %s: %s",
    BeamIOSourceRel.class.getSimpleName(),
    input);
  return sqlTable.buildIOReader(input.getPipeline().begin());
 }
}

代码示例来源:origin: com.google.cloud.genomics/google-genomics-dataflow

private static PCollection<Read> getReadsFromAPI() {
 final PCollection<Read> reads = p.begin()
   .apply(Create.of(Collections.singletonList(pipelineOptions.getReadGroupSetId())))
   .apply(new ReadGroupStreamer(auth, ShardBoundary.Requirement.STRICT, READ_FIELDS, SexChromosomeFilter.INCLUDE_XY));
 return reads;
}

代码示例来源:origin: googlegenomics/dataflow-java

private static PCollection<Read> getReadsFromAPI() {
 final PCollection<Read> reads = p.begin()
   .apply(Create.of(Collections.singletonList(pipelineOptions.getReadGroupSetId())))
   .apply(new ReadGroupStreamer(auth, ShardBoundary.Requirement.STRICT, READ_FIELDS, SexChromosomeFilter.INCLUDE_XY));
 return reads;
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, Read.Unbounded<T>> transform) {
 return PTransformReplacement.of(
   transform.getPipeline().begin(), new StreamingUnboundedRead<>(transform.getTransform()));
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Override
public PTransformReplacement<PBegin, PCollection<PubsubMessage>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<PubsubMessage>, PubsubUnboundedSource> transform) {
 return PTransformReplacement.of(
   transform.getPipeline().begin(), new StreamingPubsubIORead(transform.getTransform()));
}

代码示例来源:origin: org.apache.beam/beam-sdks-java-extensions-sql

@Override
 public PCollection<Row> expand(PCollectionList<Row> pinput) {
  checkArgument(
    pinput.size() == 0,
    "Should not have received input for %s: %s",
    BeamValuesRel.class.getSimpleName(),
    pinput);
  if (tuples.isEmpty()) {
   throw new IllegalStateException("Values with empty tuples!");
  }
  Schema schema = CalciteUtils.toSchema(getRowType());
  List<Row> rows = tuples.stream().map(tuple -> tupleToRow(schema, tuple)).collect(toList());
  return pinput.getPipeline().begin().apply(Create.of(rows)).setRowSchema(schema);
 }
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, Read.Bounded<T>> transform) {
 return PTransformReplacement.of(
   transform.getPipeline().begin(), new StreamingBoundedRead<>(transform.getTransform()));
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, Create.Values<T>> transform) {
 Create.Values<T> original = transform.getTransform();
 PCollection<T> output =
   (PCollection) Iterables.getOnlyElement(transform.getOutputs().values());
 return PTransformReplacement.of(
   transform.getPipeline().begin(), new StreamingFnApiCreate<>(original, output));
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, Read.Bounded<T>> transform) {
 return PTransformReplacement.of(
   transform.getPipeline().begin(),
   JavaReadViaImpulse.bounded(transform.getTransform().getSource()));
}

代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, Values<T>> transform) {
 return PTransformReplacement.of(
   transform.getPipeline().begin(),
   new PrimitiveCreate<>(
     transform.getTransform(),
     ((PCollection<T>) Iterables.getOnlyElement(transform.getOutputs().values()))
       .getCoder()));
}

代码示例来源:origin: org.apache.beam/beam-sdks-java-core

@Override
public PTransformReplacement<PBegin, PCollection<T>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<T>, Create.Values<T>> transform) {
 return PTransformReplacement.of(transform.getPipeline().begin(), new EmptyFlatten<T>());
}

代码示例来源:origin: org.apache.beam/beam-sdks-java-io-google-cloud-platform

@Override
public PCollection<PubsubMessage> expand(PBegin input) {
 return input
   .getPipeline()
   .begin()
   .apply(Read.from(new PubsubSource(this)))
   .apply(
     "PubsubUnboundedSource.Stats",
     ParDo.of(
       new StatsFn(pubsubFactory, subscription, topic, timestampAttribute, idAttribute)));
}

代码示例来源:origin: org.apache.beam/beam-sdks-java-core

@Override
public PTransformReplacement<PBegin, PCollection<Long>> getReplacementTransform(
  AppliedPTransform<PBegin, PCollection<Long>, GenerateSequence> transform) {
 return PTransformReplacement.of(transform.getPipeline().begin(), Create.of(0L));
}

代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java

private static AppliedPTransform<?, ?, ?> generateSequence(Pipeline pipeline) {
 GenerateSequence sequence = GenerateSequence.from(0);
 PCollection<Long> pcollection = pipeline.apply(sequence);
 return AppliedPTransform.of(
   "Count", pipeline.begin().expand(), pcollection.expand(), sequence, pipeline);
}

代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java

private static AppliedPTransform<?, ?, ?> read(Pipeline pipeline) {
 Read.Unbounded<Long> transform = Read.from(CountingSource.unbounded());
 PCollection<Long> pcollection = pipeline.apply(transform);
 return AppliedPTransform.of(
   "ReadTheCount", pipeline.begin().expand(), pcollection.expand(), transform, pipeline);
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Test
public void testPartiallyBoundFailure() throws IOException {
 Pipeline p = Pipeline.create(buildPipelineOptions());
 PCollection<Integer> input = p.begin().apply(Create.of(1, 2, 3));
 thrown.expect(IllegalArgumentException.class);
 input.apply(new PartiallyBoundOutputCreator());
 Assert.fail("Failure expected from use of partially bound output");
}

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

@Test
public void testMultiGraphPipelineSerialization() throws Exception {
 DataflowPipelineOptions options = buildPipelineOptions();
 Pipeline p = Pipeline.create(options);
 PCollection<Integer> input = p.begin().apply(Create.of(1, 2, 3));
 input.apply(new UnrelatedOutputCreator());
 input.apply(new UnboundOutputCreator());
 DataflowPipelineTranslator t =
   DataflowPipelineTranslator.fromOptions(
     PipelineOptionsFactory.as(DataflowPipelineOptions.class));
 // Check that translation doesn't fail.
 JobSpecification jobSpecification =
   t.translate(p, DataflowRunner.fromOptions(options), Collections.emptyList());
 assertAllStepOutputsHaveUniqueIds(jobSpecification.getJob());
}

相关文章