org.apache.gobblin.configuration.State.getPropertyNames()方法的使用及代码示例

x33g5p2x  于2022-01-30 转载在 其他  
字(11.8k)|赞(0)|评价(0)|浏览(103)

本文整理了Java中org.apache.gobblin.configuration.State.getPropertyNames()方法的一些代码示例，展示了State.getPropertyNames()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。State.getPropertyNames()方法的具体详情如下：
包路径：org.apache.gobblin.configuration.State
类名称：State
方法名：getPropertyNames

State.getPropertyNames介绍

[英]Get the names of all the properties set in a Set.
[中]获取集合中所有属性集的名称。

代码示例

代码示例来源：origin: apache/incubator-gobblin

@Override
public Set<String> getPropertyNames() {
 Set<String> set = Sets.newHashSet(super.getPropertyNames());
 set.addAll(this.workUnit.getPropertyNames());
 set.addAll(this.jobState.getPropertyNames());
 return set;
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Put all configuration properties in a given {@link State} object into a given
 * {@link Configuration} object.
 *
 * @param state the given {@link State} object
 * @param configuration the given {@link Configuration} object
 */
public static void putStateIntoConfiguration(State state, Configuration configuration) {
 for (String key : state.getPropertyNames()) {
  configuration.set(key, state.getProp(key));
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Add a set of properties that will overwrite properties in the {@link WorkUnitState}.
 * @param state Properties to override.
 */
public void addOverwriteProperties(State state) {
 Map<String, String> propsMap = Maps.newHashMap();
 for (String key : state.getPropertyNames()) {
  propsMap.put(key, state.getProp(key));
 }
 addOverwriteProperties(propsMap);
}

代码示例来源：origin: apache/incubator-gobblin

private static Map<String, String> getParameters(State props) {
 Map<String, String> parameters = Maps.newHashMap();
 if (props.contains(RUNTIME_PROPS)) {
  String runtimePropsString = props.getProp(RUNTIME_PROPS);
  for (String propValue : LIST_SPLITTER_COMMA.splitToList(runtimePropsString)) {
   List<String> tokens = LIST_SPLITTER_COLON.splitToList(propValue);
   Preconditions.checkState(tokens.size() == 2,
     propValue + " is not a valid Hive table/partition property");
   parameters.put(tokens.get(0), tokens.get(1));
  }
 }
 for (String propKey : props.getPropertyNames()) {
  if (!propKey.equals(RUNTIME_PROPS)) {
   parameters.put(propKey, props.getProp(propKey));
  }
 }
 return parameters;
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Set serde parameters for a table/partition.
 *
 * <p>
 *   When using {@link org.apache.gobblin.hive.metastore.HiveMetaStoreBasedRegister}, since it internally use
 *   {@link org.apache.hadoop.hive.metastore.api.Table} and {@link org.apache.hadoop.hive.metastore.api.Partition}
 *   which distinguishes between table/partition parameters, storage descriptor parameters, and serde parameters,
 *   one may need to distinguish them when constructing a {@link HiveRegistrationUnit} by using
 *   {@link #setProps(State)}, {@link #setStorageProps(State)} and
 *   {@link #setSerDeProps(State)}. When using query-based Hive registration, they do not need to be
 *   distinguished since all parameters will be passed via TBLPROPERTIES.
 * </p>
 */
public void setSerDeProps(State serdeProps) {
 for (String propKey : serdeProps.getPropertyNames()) {
  setSerDeProp(propKey, serdeProps.getProp(propKey));
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Set table/partition parameters.
 *
 * <p>
 *   When using {@link org.apache.gobblin.hive.metastore.HiveMetaStoreBasedRegister}, since it internally use
 *   {@link org.apache.hadoop.hive.metastore.api.Table} and {@link org.apache.hadoop.hive.metastore.api.Partition}
 *   which distinguishes between table/partition parameters, storage descriptor parameters, and serde parameters,
 *   one may need to distinguish them when constructing a {@link HiveRegistrationUnit} by using
 *   {@link #setProps(State)}, {@link #setStorageProps(State)} and
 *   {@link #setSerDeProps(State)}. When using query-based Hive registration, they do not need to be
 *   distinguished since all parameters will be passed via TBLPROPERTIES.
 * </p>
 */
public void setProps(State props) {
 for (String propKey : props.getPropertyNames()) {
  setProp(propKey, props.getProp(propKey));
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Set storage parameters for a table/partition.
 *
 * <p>
 *   When using {@link org.apache.gobblin.hive.metastore.HiveMetaStoreBasedRegister}, since it internally use
 *   {@link org.apache.hadoop.hive.metastore.api.Table} and {@link org.apache.hadoop.hive.metastore.api.Partition}
 *   which distinguishes between table/partition parameters, storage descriptor parameters, and serde parameters,
 *   one may need to distinguish them when constructing a {@link HiveRegistrationUnit} by using
 *   {@link #setProps(State)}, {@link #setStorageProps(State)} and
 *   {@link #setSerDeProps(State)}. When using query-based Hive registration, they do not need to be
 *   distinguished since all parameters will be passed via TBLPROPERTIES.
 * </p>
 */
public void setStorageProps(State storageProps) {
 for (String propKey : storageProps.getPropertyNames()) {
  setStorageProp(propKey, storageProps.getProp(propKey));
 }
}

代码示例来源：origin: apache/incubator-gobblin

@Test
 public void testRemovePropsWithPrefix() {
  final State state = new State();
  final String prefix = "prefix";
  for (int i = 0; i < 10; i++) {
   state.setProp("prefix." + i, i);
  }
  Assert.assertTrue(state.getPropertyNames().size() == 10);
  state.removePropsWithPrefix(prefix);
  Assert.assertTrue(state.getPropertyNames().size() == 0);
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Adds all properties from {@link org.apache.gobblin.configuration.State} to this {@link org.apache.gobblin.configuration.WorkUnitState}.
 *
 * <p>
 *   A property with name "property" will be added to this object with the key
 *   "{@link #FINAL_CONSTRUCT_STATE_PREFIX}[.<infix>].property"
 * </p>
 *
 * @param infix Optional infix used for the name of the property in the {@link org.apache.gobblin.configuration.WorkUnitState}.
 * @param finalConstructState {@link org.apache.gobblin.configuration.State} for which all properties should be added to this
 *                                                               object.
 */
public void addFinalConstructState(String infix, State finalConstructState) {
 for (String property : finalConstructState.getPropertyNames()) {
  if (Strings.isNullOrEmpty(infix)) {
   setProp(FINAL_CONSTRUCT_STATE_PREFIX + property, finalConstructState.getProp(property));
  } else {
   setProp(FINAL_CONSTRUCT_STATE_PREFIX + infix + "." + property, finalConstructState.getProp(property));
  }
 }
}

代码示例来源：origin: apache/incubator-gobblin

@Override
public State getFinalState() {
 State state = new State();
 try {
  for (Map.Entry<GenericRecord, DataWriter<D>> entry : this.partitionWriters.asMap().entrySet()) {
   if (entry.getValue() instanceof FinalState) {
    State partitionFinalState = ((FinalState) entry.getValue()).getFinalState();
    if (this.shouldPartition) {
     for (String key : partitionFinalState.getPropertyNames()) {
      // Prevent overwriting final state across writers
      partitionFinalState.setProp(key + "_" + AvroUtils.serializeAsPath(entry.getKey(), false, true),
        partitionFinalState.getProp(key));
     }
    }
    state.addAll(partitionFinalState);
   }
  }
  state.setProp("RecordsWritten", recordsWritten());
  state.setProp("BytesWritten", bytesWritten());
 } catch (Exception exception) {
  log.warn("Failed to get final state." + exception.getMessage());
  // If Writer fails to return bytesWritten, it might not be implemented, or implemented incorrectly.
  // Omit property instead of failing.
 }
 return state;
}

代码示例来源：origin: apache/incubator-gobblin

for (String key : this.getState().getPropertyNames()) {
 conf.set(key, this.getState().getProp(key));

代码示例来源：origin: apache/incubator-gobblin

@Test
 public void testMethods() {
  State state = new State();
  state.setProp("foo", "bar");
  Destination destination = Destination.of(Destination.DestinationType.HDFS, state);
  Assert.assertEquals(destination.getType(), Destination.DestinationType.HDFS);
  Assert.assertEquals(destination.getProperties().getPropertyNames().size(), 1);
  Assert.assertEquals(destination.getProperties().getProp("foo"), "bar");
 }
}

代码示例来源：origin: apache/incubator-gobblin

Configuration conf = new Configuration();
for (String key : properties.getPropertyNames()) {
 conf.set(key, properties.getProp(key));

代码示例来源：origin: org.apache.gobblin/gobblin-api

@Override
public Set<String> getPropertyNames() {
 Set<String> set = Sets.newHashSet(super.getPropertyNames());
 set.addAll(this.workUnit.getPropertyNames());
 set.addAll(this.jobState.getPropertyNames());
 return set;
}

代码示例来源：origin: org.apache.gobblin/gobblin-utility

/**
 * Put all configuration properties in a given {@link State} object into a given
 * {@link Configuration} object.
 *
 * @param state the given {@link State} object
 * @param configuration the given {@link Configuration} object
 */
public static void putStateIntoConfiguration(State state, Configuration configuration) {
 for (String key : state.getPropertyNames()) {
  configuration.set(key, state.getProp(key));
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

/**
 * Add a set of properties that will overwrite properties in the {@link WorkUnitState}.
 * @param state Properties to override.
 */
public void addOverwriteProperties(State state) {
 Map<String, String> propsMap = Maps.newHashMap();
 for (String key : state.getPropertyNames()) {
  propsMap.put(key, state.getProp(key));
 }
 addOverwriteProperties(propsMap);
}

代码示例来源：origin: org.apache.gobblin/gobblin-hive-registration

private static Map<String, String> getParameters(State props) {
 Map<String, String> parameters = Maps.newHashMap();
 if (props.contains(RUNTIME_PROPS)) {
  String runtimePropsString = props.getProp(RUNTIME_PROPS);
  for (String propValue : LIST_SPLITTER_COMMA.splitToList(runtimePropsString)) {
   List<String> tokens = LIST_SPLITTER_COLON.splitToList(propValue);
   Preconditions.checkState(tokens.size() == 2,
     propValue + " is not a valid Hive table/partition property");
   parameters.put(tokens.get(0), tokens.get(1));
  }
 }
 for (String propKey : props.getPropertyNames()) {
  if (!propKey.equals(RUNTIME_PROPS)) {
   parameters.put(propKey, props.getProp(propKey));
  }
 }
 return parameters;
}

代码示例来源：origin: org.apache.gobblin/gobblin-hive-registration

/**
 * Set serde parameters for a table/partition.
 *
 * <p>
 *   When using {@link org.apache.gobblin.hive.metastore.HiveMetaStoreBasedRegister}, since it internally use
 *   {@link org.apache.hadoop.hive.metastore.api.Table} and {@link org.apache.hadoop.hive.metastore.api.Partition}
 *   which distinguishes between table/partition parameters, storage descriptor parameters, and serde parameters,
 *   one may need to distinguish them when constructing a {@link HiveRegistrationUnit} by using
 *   {@link #setProps(State)}, {@link #setStorageProps(State)} and
 *   {@link #setSerDeProps(State)}. When using query-based Hive registration, they do not need to be
 *   distinguished since all parameters will be passed via TBLPROPERTIES.
 * </p>
 */
public void setSerDeProps(State serdeProps) {
 for (String propKey : serdeProps.getPropertyNames()) {
  setSerDeProp(propKey, serdeProps.getProp(propKey));
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-hive-registration

/**
 * Set table/partition parameters.
 *
 * <p>
 *   When using {@link org.apache.gobblin.hive.metastore.HiveMetaStoreBasedRegister}, since it internally use
 *   {@link org.apache.hadoop.hive.metastore.api.Table} and {@link org.apache.hadoop.hive.metastore.api.Partition}
 *   which distinguishes between table/partition parameters, storage descriptor parameters, and serde parameters,
 *   one may need to distinguish them when constructing a {@link HiveRegistrationUnit} by using
 *   {@link #setProps(State)}, {@link #setStorageProps(State)} and
 *   {@link #setSerDeProps(State)}. When using query-based Hive registration, they do not need to be
 *   distinguished since all parameters will be passed via TBLPROPERTIES.
 * </p>
 */
public void setProps(State props) {
 for (String propKey : props.getPropertyNames()) {
  setProp(propKey, props.getProp(propKey));
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-api

/**
 * Adds all properties from {@link org.apache.gobblin.configuration.State} to this {@link org.apache.gobblin.configuration.WorkUnitState}.
 *
 * <p>
 *   A property with name "property" will be added to this object with the key
 *   "{@link #FINAL_CONSTRUCT_STATE_PREFIX}[.<infix>].property"
 * </p>
 *
 * @param infix Optional infix used for the name of the property in the {@link org.apache.gobblin.configuration.WorkUnitState}.
 * @param finalConstructState {@link org.apache.gobblin.configuration.State} for which all properties should be added to this
 *                                                               object.
 */
public void addFinalConstructState(String infix, State finalConstructState) {
 for (String property : finalConstructState.getPropertyNames()) {
  if (Strings.isNullOrEmpty(infix)) {
   setProp(FINAL_CONSTRUCT_STATE_PREFIX + property, finalConstructState.getProp(property));
  } else {
   setProp(FINAL_CONSTRUCT_STATE_PREFIX + infix + "." + property, finalConstructState.getProp(property));
  }
 }
}

内容来源于网络，如有侵权，请联系作者删除！

state

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com