kafka jdbc接收器连接器：在创建或更新一个连接器时重新创建所有其他连接器

nimxete2  于 2021-06-06  发布在  Kafka
关注(0)|答案(1)|浏览(392)

我有大约500个连接器，每次创建一个新的连接器时，我都要等很长时间才能重新创建以前的连接器。太慢了。
sink.properties如下

{
    "name": "saas-saas_order-worder_fee_price-sink",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
        "errors.log.include.messages": "true",
        "tasks.max": "1",
        "topics": "saasosvc-worder_fee_price",
        "transforms": "unwrap",
        "auto.evolve": "true",
        "name": "saas-saas_order-worder_fee_price-sink",
        "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
        "auto.create": "true",
        "connection.url": "jdbc:postgresql://postgres:5432/saas?user=postgresuser&password=postgrespw",
        "errors.log.enable": "true",
        "insert.mode": "upsert",
        "pk.mode": "record_value",
        "pk.fields": "id"
    }
}

一些日志如下

> 2019-06-12 05:44:42,242 INFO   || 
> WorkerSourceTask{id=saas-com_dyrs_mtsp_furnituresaleservice-source-0}
> flushing 0 outstanding messages for offset commit  
> [org.apache.kafka.connect.runtime.WorkerSourceTask]
>     2019-06-12 05:44:45,950 INFO   ||  WorkerSinkTask{id=saas-appconstruction-appconstruction_requestrecord-sink-0}
> Committing offsets asynchronously using sequence number 20376:
> {appssvc-appconstruction_requestrecord-0=OffsetAndMetadata{offset=2340,
> leaderEpoch=null, metadata=''}}  
> [org.apache.kafka.connect.runtime.WorkerSinkTask]
>     2019-06-12 05:45:18,242 INFO   ||  Connector saas-saas_order-worder_fee_price-sink config updated  
> [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
>     2019-06-12 05:45:18,745 INFO   ||  Rebalance started   [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
>     2019-06-12 05:45:18,747 INFO   ||  Stopping connector saas-order_base-product_vs_install-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,747 INFO   ||  Stopping connector saas-dyrs_complainservice-complain_type_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-appcustomerself-appcustomerself_appversionuser-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-com_dyrs_mtsp_changeservice-replace_product-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-com_dyrs_mtsp_designservice-version_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-dyrs_settlementservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-appcustomerself-appcustomerself_advice-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-im-im_accesstoken-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,761 INFO   ||  172.31.206.219 - - [12/Jun/2019:05:45:18 +0000] "POST /connectors/ HTTP/1.1" 201 604  549
> [org.apache.kafka.connect.runtime.rest.RestServer]
>     2019-06-12 05:45:18,775 INFO   ||  Stopped connector saas-im-im_accesstoken-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,776 INFO   ||  Stopping connector saas-saas_order-dd_order-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,778 INFO   ||  Stopped connector saas-dyrs_settlementservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,780 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_info_detail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopped connector saas-com_dyrs_mtsp_designservice-version_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopping connector saas-dyrs_authorityservice-pro_city_area-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopped connector saas-dyrs_complainservice-complain_type_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopping connector saas-constructionconfig-constructionconfig_checkmanagedetailstandard-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,799 INFO   ||  Stopped connector saas-saas_order-dd_order-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,799 INFO   ||  Stopping connector saas-saas_order-goods-sink   [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,857 INFO   ||  Stopped connector saas-saas_order-goods-sink   [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,857 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-business_process_operator-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,860 INFO   ||  Stopped connector saas-appcustomerself-appcustomerself_appversionuser-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,860 INFO   ||  Stopping connector saas-com_dyrs_mtsp_changeservice-custom_wood_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,862 INFO   ||  Stopped connector saas-order_base-product_vs_install-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,863 INFO   ||  Stopping connector saas-com_dyrs_mtsp_quotelistservice-pre_quote_detail_tab_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,865 INFO   ||  Stopped connector saas-com_dyrs_mtsp_changeservice-replace_product-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,868 INFO   ||  Stopping connector saas-com_dyrs_mtsp_quoteservice-program_template_tab-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,870 INFO   ||  Stopped connector saas-dyrs_settlementservice-balance_info_detail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,870 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_config-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,866 INFO   ||  Stopped connector saas-appcustomerself-appcustomerself_advice-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,873 INFO   ||  Stopping connector saas-dyrs_authorityservice-account_login_fail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,872 INFO   ||  Stopped connector saas-dyrs_authorityservice-pro_city_area-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,873 INFO   ||  Stopping connector saas-com_dyrs_mtsp_dataexpansionservice-dynamic_tabpk-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,875 INFO   ||  Stopped connector saas-constructionconfig-constructionconfig_checkmanagedetailstandard-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,875 INFO   ||  Stopping connector saas-com_dyrs_mtsp_promotionservice-offer_content-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,884 INFO   ||  Stopped connector saas-com_dyrs_mtsp_businessopportunityservice-business_process_operator-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,885 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-personal_clue_limit-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,891 INFO   ||  Stopped connector saas-com_dyrs_mtsp_changeservice-custom_wood_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,891 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_operation-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,902 INFO   ||  Stopped connector saas-com_dyrs_mtsp_quotelistservice-pre_quote_detail_tab_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,902 INFO   ||  Stopping connector saas-order_base-unit_vs_unit-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,904 INFO   ||  Stopped connector saas-com_dyrs_mtsp_dataexpansionservice-dynamic_tabpk-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,905 INFO   ||  Stopped connector saas-com_dyrs_mtsp_quoteservice-program_template_tab-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,907 INFO   ||  Stopping connector saas-com_dyrs_mtsp_fieldassociationservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,904 INFO   ||  Stopped connector saas-dyrs_authorityservice-account_login_fail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,907 INFO   ||  Stopping connector saas-customerself-customerself_msgsendrecorddetail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,908 INFO   ||  Stopping connector saas-finereport-constructionprocess_constructioninfoprocesscheck-sink 
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,913 INFO   ||  Stopped connector saas-dyrs_settlementservice-balance_config-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,134 INFO   ||  Stopped connector saas-order_base-unit_vs_unit-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,136 INFO   ||  Stopping connector saas-com_dyrs_mtsp_furnituresaleservice-fur_sale_contract_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,139 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-call_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,154 INFO   ||  Stopped connector saas-com_dyrs_mtsp_furnituresaleservice-fur_sale_contract_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,154 INFO   ||  Stopping connector saas-com_dyrs_mtsp_fieldassociationservice-pre_filed-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,166 INFO   ||  Stopped connector saas-com_dyrs_mtsp_promotionservice-offer_content-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,166 INFO   ||  Stopping connector saas-runmonitoring-runmonitoring_browsinghistory-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,167 INFO   ||  Stopped connector saas-com_dyrs_mtsp_fieldassociationservice-pre_filed-sink  
> [org.apache.kafka.connect.runtime.Worker]

谢谢

apache-kafkaapache-kafka-connectconfluent-platform

1条答案

按热度按时间
nafvub8i

nafvub8i1#

“停止世界”的再平衡是Kafka连接已知的问题。好消息是，在ApacheKafka2.3中发布的kip-415中，有一个新的增量再平衡特性，应该会让事情变得更好。
同时，唯一的其他选择是将kafka connect worker分区并拥有单独的集群，将500个连接器拆分到它们上面（例如，按函数、类型或其他任意因素）。

赞(0）分享  回复(0）举报  2021-06-06
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com