org.apache.spark.sql.DataFrame.collectAsList()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(2.0k)|赞(0)|评价(0)|浏览(407)

本文整理了Java中org.apache.spark.sql.DataFrame.collectAsList()方法的一些代码示例,展示了DataFrame.collectAsList()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。DataFrame.collectAsList()方法的具体详情如下:
包路径:org.apache.spark.sql.DataFrame
类名称:DataFrame
方法名:collectAsList

DataFrame.collectAsList介绍

暂无

代码示例

代码示例来源:origin: Impetus/Kundera

/**
 * Populate entity objects list.
 * 
 * @param dataFrame
 *            the data frame
 * @param m
 *            the m
 * @return the list
 */
private List<?> populateEntityObjectsList(DataFrame dataFrame, EntityMetadata m)
{
  List results = new ArrayList();
  String[] columns = dataFrame.columns();
  Map<String, Integer> map = createMapOfColumnIndex(columns);
  for (Row row : dataFrame.collectAsList())
  {
    Object entity = populateEntityFromDataFrame(m, map, row);
    results.add(entity);
  }
  return results;
}

代码示例来源:origin: Impetus/Kundera

/**
 * Load data and populate results.
 * 
 * @param dataFrame
 *            the data frame
 * @param m
 *            the m
 * @param kunderaQuery
 *            the kundera query
 * @return the list
 */
public List<?> loadDataAndPopulateResults(DataFrame dataFrame, EntityMetadata m, KunderaQuery kunderaQuery)
{
  if (kunderaQuery != null && kunderaQuery.isAggregated())
  {
    return dataFrame.collectAsList();
  }
  // TODO: handle the case of specific field selection
  else
  {
    return populateEntityObjectsList(dataFrame, m);
  }
}

代码示例来源:origin: sujee81/SparkApps

public static void main(String[] args) {
    //Data source options
    Map<String, String> options = new HashMap<>();
    options.put("driver", MYSQL_DRIVER);
    options.put("url", MYSQL_CONNECTION_URL);
    options.put("dbtable",
          "(select emp_no, concat_ws(' ', first_name, last_name) as full_name from employees) as employees_name");
    options.put("partitionColumn", "emp_no");
    options.put("lowerBound", "10001");
    options.put("upperBound", "499999");
    options.put("numPartitions", "10");

    //Load MySQL query result as DataFrame
    DataFrame jdbcDF = sqlContext.load("jdbc", options);

    List<Row> employeeFullNameRows = jdbcDF.collectAsList();

    for (Row employeeFullNameRow : employeeFullNameRows) {
      LOGGER.info(employeeFullNameRow);
    }
  }
}

相关文章