将sql查询转换为等效的spark查询

uurity8g  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(231)

我正在使用 spark-sql-2.4.1v 使用java8。
我有下面这样的场景/片段

Dataset<Row> df =//loaded data from a csv file
// this has columns like "code1","code2","code3","code4","code5","code6", and "class"

 df.createOrReplaceTempView("temp_tab");

List<String> codesList = Arrays.asList("code1","code5"); // codes of interest to be calculated.

 codesList.stream().forEach( code -> {

 String query = "select " 
                                  + " avg(" + code + ") as mean, "
                                  + "percentile(" + code +",0.25) as p25" 
                                  + "from " + temp_tab                      
                                  + " group by class";

  Dataset<Row> resultDs  = sparkSession.sql(query);
 });

如何使用functions.expr()和functions.agg()编写?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题