java代码中spark的聚合函数

6ss1mwsb  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(226)

我在java代码中有一个特定的例子要处理spark,但我不知道该怎么做:我想要最后一个带有partition ptf的false标记(这里的记录在desc中为列integrationdatetime排序)以及标记为true时后面的所有记录。在sql中,它看起来像:

with full_query as (
select distinct t.portfolio, t.commandUId, max(t.isPartialRelaunch) over (partition by t.commandName, t.commandUId, t.portfolio) as isPartialRelaunchPerPTF
    , (DENSE_RANK() over (partition by t.portfolio, t.commandUId) order by t.i2rIntegrationTime desc) as commandUIDRank
from varFumModel t)
)
select t.*, (DENSE_RANK() over (partition by t.portfolio, t.commandUId) order by t.i2rIntegrationTime desc) as commandUIDRank from varFumModel t 
where commandUIDRank <= nvl((select min(d.commandUIDRank) from full_query d where d.isPartialRelaunchPerPTF = false and d.portfolio = t.portfolio and d.commandUId = t.commandUId), 10000)

化学文摘社1

+------+------------+-------+------+---------------+-----------------------+
|PTF1  | Position1  | TRUE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | TRUE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+-----------------------+

+------+------------+-------+------+---------------+-----------------------+
|PTF1  | Position1  | FALSE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+-----------------------+

结果

+------+------------+-------+------+---------------+-----------------------+
|PTF1  | Position1  | TRUE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | TRUE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position1  | FALSE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+-----------------------+

化学文摘社2

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+

结果

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+

化学文摘社3

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+

结果

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | FALSE  | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+

化学文摘社4

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+

结果

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+

化学文摘社5

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+

+------+------------+-------+------+---------------+-------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_-1| integrationdatetime_t-1|
+------+------------+-------+------+---------------+-------------------------+

结果

+------+------------+-------+------+---------------+------------------------+
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position2  | TRUE   | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1  | Position1  | TRUE   | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
|PTF1  | Position1  | FALSE  | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题