为什么gcs连接器在gcs中创建Kafka分区目录,但不写入Kafka主题数据

piv4azn7  于 2021-06-05  发布在  Kafka
关注(0)|答案(0)|浏览(312)

我是一个新手与汇合地面军事系统连接器。我遵循了中概述的文档https://docs.confluent.io/4.1.2/connect/kafka-connect-gcs/gcs_connector.html 一切都按计划进行,除了Kafka的主题数据没有出现在gcs桶里。目录结构类似于Kafka主题名,但没有实际的主题记录。我使用上述文档中包含的默认属性文件作为avro格式模式,刷新大小为3。我下载了凭据文件,并且可以从vm示例看到它。我还可以将本地文件从我的googlevm示例复制到bucket,这样我就知道我有正确的权限。

{"name": "gcs",
  "config": {
    "connector.class": "io.confluent.connect.gcs.GcsSinkConnector",
    "tasks.max": "1",
    "topics": "gcs_topic",
    "gcs.bucket.name": "#bucket-name",
    "gcs.part.size": "5242880",
    "flush.size": "3",
    "gcs.credentials.path": "#/path/to/credentials/keys.json",
    "storage.class": "io.confluent.connect.gcs.storage.GcsStorage",
    "format.class": "io.confluent.connect.gcs.format.avro.AvroFormat",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "schema.compatibility": "NONE",
    "confluent.topic.bootstrap.servers": "localhost:9092",
    "confluent.topic.replication.factor": "1",
    "name": "gcs"
  },
  "tasks": [],
  "type": null
}

连接属性文件如下:

bootstrap.servers=localhost:9092
group.id=connect-cluster
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-statuses
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
Plugin.path=~/confluent/share/java,~/confluent/share/confluent-hub-components,~/confluent/etc,~/confluent/etc/
rest.extension.classes=io.confluent.connect.replicator.monitoring.ReplicatorMonitoringExtension
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题