groupreducefunction并行运行apache flink java有替代方法吗?

mznpcxlj  于 2021-06-24  发布在  Flink
关注(0)|答案(0)|浏览(176)

下面的代码在本地运行,但不在集群上运行。它挂起在groupreducefunction上,即使在几个小时后也不会终止(对于大数据,本地计算大约需要9分钟)。日志中的最后一条消息: GroupReduce (GroupReduce at main(MyClass.java:80)) (1/1) (...) switched from DEPLOYING to RUNNING. 代码片段:

DataSet<MyData1> myData1 = env.createInput(new UserDefinedFunctions.MyData1Set());
DataSet<MyData2> myData2 = DataSetUtils.sampleWithSize(myData1, false, 8, Long.MAX_VALUE)
                .reduceGroup(new GroupReduceFunction<MyData1, MyData2>() {
                    @Override
                    public void reduce(Iterable<MyData1> itrbl, Collector<MyData2> clctr) throws Exception {
                        int id = 0;
                        for (MyData1 myData1 : itrbl) {
                            clctr.collect(new MyData2(id++, myData1));
                        }
                    }
                });

有没有办法让这一部分并行运行?提前谢谢!

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题