本文整理了Java中org.apache.spark.sql.DataFrame.collectAsList()
方法的一些代码示例,展示了DataFrame.collectAsList()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。DataFrame.collectAsList()
方法的具体详情如下:
包路径:org.apache.spark.sql.DataFrame
类名称:DataFrame
方法名:collectAsList
暂无
代码示例来源:origin: Impetus/Kundera
/**
* Populate entity objects list.
*
* @param dataFrame
* the data frame
* @param m
* the m
* @return the list
*/
private List<?> populateEntityObjectsList(DataFrame dataFrame, EntityMetadata m)
{
List results = new ArrayList();
String[] columns = dataFrame.columns();
Map<String, Integer> map = createMapOfColumnIndex(columns);
for (Row row : dataFrame.collectAsList())
{
Object entity = populateEntityFromDataFrame(m, map, row);
results.add(entity);
}
return results;
}
代码示例来源:origin: Impetus/Kundera
/**
* Load data and populate results.
*
* @param dataFrame
* the data frame
* @param m
* the m
* @param kunderaQuery
* the kundera query
* @return the list
*/
public List<?> loadDataAndPopulateResults(DataFrame dataFrame, EntityMetadata m, KunderaQuery kunderaQuery)
{
if (kunderaQuery != null && kunderaQuery.isAggregated())
{
return dataFrame.collectAsList();
}
// TODO: handle the case of specific field selection
else
{
return populateEntityObjectsList(dataFrame, m);
}
}
代码示例来源:origin: sujee81/SparkApps
public static void main(String[] args) {
//Data source options
Map<String, String> options = new HashMap<>();
options.put("driver", MYSQL_DRIVER);
options.put("url", MYSQL_CONNECTION_URL);
options.put("dbtable",
"(select emp_no, concat_ws(' ', first_name, last_name) as full_name from employees) as employees_name");
options.put("partitionColumn", "emp_no");
options.put("lowerBound", "10001");
options.put("upperBound", "499999");
options.put("numPartitions", "10");
//Load MySQL query result as DataFrame
DataFrame jdbcDF = sqlContext.load("jdbc", options);
List<Row> employeeFullNameRows = jdbcDF.collectAsList();
for (Row employeeFullNameRow : employeeFullNameRows) {
LOGGER.info(employeeFullNameRow);
}
}
}
内容来源于网络,如有侵权,请联系作者删除!