我设置了一个合流的s3接收器连接,它将.avro文件存储在s3中。
我转储了那些文件,发现它们只是消息本身,我不知道在哪里可以找到消息密钥,知道吗?
配置如下:
{
"name": "s3-sink-test",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"tasks.max": "1",
"topics": "book",
"s3.region": "eu-central-1",
"s3.bucket.name": "kafka",
"s3.part.size": "5242880",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
"schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
"partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
"locale": "US",
"timezone": "UTC",
"partition.duration.ms": "3600000",
"timestamp.extractor": "RecordField",
"timestamp.field": "local_timestamp",
"flush.size": "2",
"schema.compatibility": "NONE"
}
}
1条答案
按热度按时间57hvy0tb1#
Kafka消息密钥不是由任何存储Kafka连接器通过汇合出盒来保持的
请尝试编译并设置存档转换,可以使用连接配置中的这些属性设置存档转换
有关Kafka连接中SMT的更多信息,请参阅此博客文章