kafka连接器找不到s3目标，该目标是minio url

我有Kafka和Kafka连接器集群安装斯特里姆齐在Kubernetes。
这是我的s3/minio连接器配置：

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: "minio-connector"
  labels:
    strimzi.io/cluster: mssql-minio-connect-cluster
spec:
  class: io.confluent.connect.s3.S3SinkConnector
  config:
    storage.class: io.confluent.connect.s3.storage.S3Storage  
    format.class: io.confluent.connect.s3.format.avro.AvroFormat
    schema.generator.class: io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
    partitioner.class: io.confluent.connect.storage.partitioner.DefaultPartitioner
    tasks.max: '1'
    topics: filesql1.dbo.files
    s3.bucket.name: dosyalar
    s3.part.size: '5242880'
    flush.size: '3'
    format: binary
    schema.compatibility: NONE
    max.request.size: "536870912"
    store.url: http://minio.dev-kik.io

这是我自定义的kafka connect cluster docker container dockerfile配置：

FROM strimzi/kafka:latest-kafka-2.5.0
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
USER 1001
ENV MINIO_VOLUMES=http://minio.dev-kik.io
ENV AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
ENV AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

我的配置有问题。我想我的Kafka连接吊舱找不到或无法访问我的minio:

{                                                                                                                                                                                                                  
      "name": "minio-connector",                                                                                                                                                                                       
      "connector": {                                                                                                                                                                                                   
        "state": "RUNNING",                                                                                                                                                                                            
        "worker_id": "10.244.1.183:8083"                                                                                                                                                                               
      },                                                                                                                                                                                                               
      "tasks": [                                                                                                                                                                                                       
        {                                                                                                                                                                                                              
          "id": 0,                                                                                                                                                                                                     
          "state": "FAILED",                                                                                                                                                                                           
          "worker_id": "10.244.1.183:8083",                                                                                                                                                                            
          "trace": "org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io\n\tat io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java
    :121)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:305)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)\n\tat org.apache.kaf
    ka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n
    \tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Thread
    PoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io\n\tat com.amazonaws.http.AmazonHttpClient$Request
    Executor.handleRetryableException(AmazonHttpClient.java:1201)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)\n\tat com.amazonaws.http.AmazonHttpClient$Request
    Executor.doExecute(AmazonHttpClient.java:796)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execu
    te(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonH
    ttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.Amazo
    nS3Client.invoke(AmazonS3Client.java:5052)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)\n\tat com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1413)\n\ta
    t com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1349)\n\tat io.confluent.connect.s3.storage.S3Storage.bucketExists(S3Storage.java:196)\n\tat io.confluent.connect.s3.S3SinkTask.star
    t(S3SinkTask.java:108)\n\t... 9 more\nCaused by: java.net.UnknownHostException: minio.dev-kik.io\n\tat java.net.InetAddress.getAllByName0(InetAddress.java:1281)\n\tat java.net.InetAddress.getAllByName(InetAddres
    s.java:1193)\n\tat java.net.InetAddress.getAllByName(InetAddress.java:1127)\n\tat com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)\n\tat com.amazonaws.http.DelegatingDnsResolver.r
    esolve(DelegatingDnsResolver.java:38)\n\tat org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)\n\tat org.apache.http.impl.conn.PoolingHttpClientCo
    nnectionManager.connect(PoolingHttpClientConnectionManager.java:374)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
    ava:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.amazonaws.http.conn.ClientConnectionManagerFac
    tory$Handler.invoke(ClientConnectionManagerFactory.java:76)\n\tat com.amazonaws.http.conn.$Proxy37.connect(Unknown Source)\n\tat org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:3
    93)\n\tat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)\n\tat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)\n\tat org.apache.http.impl.client.Interna
    lHttpClient.doExecute(InternalHttpClient.java:185)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(Closeab
    leHttpClient.java:56)\n\tat com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:13
    23)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)\n\t... 22 more\n"                                                                                          
        }                                                                                                                                                                                                              
      ],                                                                                                                                                                                                               
      "type": "sink"                                                                                                                                                                                                   
    }

另外，请看输出：

Name:         minio-connector
Namespace:    kafka
Labels:       strimzi.io/cluster=mssql-minio-connect-cluster
Annotations:  API Version:  kafka.strimzi.io/v1alpha1
Kind:         KafkaConnector
Metadata:
  Creation Timestamp:  2020-05-17T07:47:58Z
  Generation:          2
  Managed Fields:
    API Version:  kafka.strimzi.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:connectorStatus:
          f:connector:
            f:worker_id:
          f:tasks:
    Manager:         okhttp
    Operation:       Update
    Time:            2020-05-17T19:22:12Z
  Resource Version:  5349641
  Self Link:         /apis/kafka.strimzi.io/v1alpha1/namespaces/kafka/kafkaconnectors/minio-connector
  UID:               4f4a080d-580e-4243-b187-c8b485461285
Spec:
  Class:  io.confluent.connect.s3.S3SinkConnector
  Config:
    flush.size:              3
    Format:                  binary
    format.class:            io.confluent.connect.s3.format.avro.AvroFormat
    max.request.size:        536870912
    partitioner.class:       io.confluent.connect.storage.partitioner.DefaultPartitioner
    s3.bucket.name:          dosyalar
    s3.part.size:            5242880
    schema.compatibility:    NONE
    schema.generator.class:  io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
    storage.class:           io.confluent.connect.s3.storage.S3Storage
    store.url:               http://minio.dev-kik.io
    tasks.max:               1
    Topics:                  s3-topic
Status:
  Conditions:
    Last Transition Time:  2020-05-17T19:22:12.526Z
    Status:                True
    Type:                  Ready
  Connector Status:
    Connector:
      State:      RUNNING
      worker_id:  10.244.1.183:8083
    Name:         minio-connector
    Tasks:
      Id:     0
      State:  FAILED
      Trace:  org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
              at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:121)
              at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:305)
              at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
              at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
              at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)
  at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1413)
  at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1349)
  at io.confluent.connect.s3.storage.S3Storage.bucketExists(S3Storage.java:196)
  at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:108)
  ... 9 more
Caused by: java.net.UnknownHostException: minio.dev-kik.io
  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
  at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
  at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
  at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
  at com.amazonaws.http.conn.$Proxy37.connect(Unknown Source)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
  at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
  ... 22 more

      worker_id:        10.244.1.183:8083
    Type:               sink
  Observed Generation:  2
Events:                 <none>

当我刷新所有日志时，我看到：

devadmin@vdi-mk2-ubn:~$ kubectl logs mssql-minio-connect-cluster-connect-5bf6d6bbcc-ggh5g -n kafka | grep minio.dev-kik.io
        store.url = http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
Caused by: java.net.UnknownHostException: minio.dev-kik.io
        store.url = http://minio.dev-kik.io
devadmin@vdi-mk2-ubn:~$ kubectl logs mssql-minio-connect-cluster-connect-5bf6d6bbcc-ggh5g -n kafka | grep http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
devadmin@vdi-mk2-ubn:~$ kubectl logs mssql-minio-connect-cluster-connect-5bf6d6bbcc-ggh5g -n kafka | grep Unable
org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io

看起来s3连接器在aws上查找minio/s3，而我的minio不在aws上，它在本地。当i nslookup minio url（minio.kik-dev.io）时，connect cluster pod可以解析ip，但ping命令不能。我不确定这和dns解析有关。
这个问题快把我逼疯了好几天了，你能告诉我遗漏了什么吗？

kafka连接器找不到s3目标，该目标是minio url

暂无答案！

相关问题

热门标签

最新问答