Apache·Flink有水痕的Windows

5f0d552i  于 2021-06-25  发布在  Flink
关注(0)|答案(1)|浏览(405)

我正在尝试聚合60秒的数据,按分钟时间戳键控,最大延迟为30秒。

DataStream<OHLChelp> ohlcAggStream = stockStream.assignTimestampsAndWatermarks(new TimestampExtractor(Time.seconds(30))).map(new mapStockToOhlcHelp()).keyBy((KeySelector<OHLChelp, Long>) o -> o.getMinTime())
            .timeWindow(Time.seconds(60))
            .reduce(new aggregateOHLC());
//map complex object to simpler one
DataStream<OHLCmodel> ohlcStremAggregated = ohlcAggStream.map(new mapOHLCredToOHLCfin());
//log ohlc stream
ohlcStreamAggregated.writeAsText(outLogPath);

我收到数据了。正在设置水印和时间戳。似乎,聚合数据从未发送到ohlcstreamaggregated,因此它们不会被记录。

public TimestampExtractor(Time maxDelayInterval) {
        if (maxDelayInterval.toMilliseconds() < 0) {
            throw new RuntimeException("This parameter must be positive or 0.);
        }
        this.maxDelayInterval = maxDelayInterval.toMilliseconds() / 1000;
        this.currentMaxTimestamp = Long.MIN_VALUE + this.maxDelayInterval;
    }

@Override
public final Watermark getCurrentWatermark() {
        // set maximum delay 30 seconds
        long potentialWM = currentMaxTimestamp - maxDelayInterval;
        if (potentialWM > lastEmittedWM) {
            lastEmittedWM = potentialWM;
        }
        return new Watermark(lastEmittedWM);
    }
@Override
public final long extractTimestamp(StockTrade stockTrade, long previousElementTimestamp) {
        BigDecimal bd = new BigDecimal(stockTrade.getTime());
        long timestamp = bd.longValue();
        //set the maximum seen timestamp so far
        if (timestamp > currentMaxTimestamp) {
            currentMaxTimestamp = timestamp;
        }
        return timestamp;
    }

我用这个例子作为模板。

dgjrabp2

dgjrabp21#

如果您可以分享整个过程(可能是要点),那么诊断应用程序会更容易,但是,您是否:
将时间特性设置为事件时间(docs)?
在流执行环境中调用execute?
另外,时间戳提取器可能会简单一些。更像这样:

public static class TimestampExtractor extends BoundedOutOfOrdernessTimestampExtractor<StockTrade> {
    public TimestampExtractor() {
        super(Time.seconds(30));
    }

    @Override
    public long extractTimestamp(StockTrade trade) {
        return trade.getTime();
    }
}

相关问题