我是apache环境中的新手，目前我正在尝试从filebeat producer向kafka代理发送日志数据。
环境：
Kafka2.11（通过ambari安装）
filebeat 7.4.2（windows）
我试图将日志从filebeat发送到ambari，我启动了kafka服务器并创建了名为“test”的主题，它被列在--list上。我对Kafka的港口相当困惑。在一些教程中，我看到他们使用9092而不是2181。那么现在，我应该使用哪个端口从filebeat发送日志呢？
这是我的filebeat.conf
filebeat.inputs:
- type: log
paths:
- C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
hosts: ["x.x.x.x:9092"]
topic: "test"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000结果
2020-06-10T09:00:32.214+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:32.214+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)
2020-06-10T09:00:32.466+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:34.475+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:34.475+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:34.477+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:34.477+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:34.478+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)
2020-06-10T09:00:34.729+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:36.737+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:36.737+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)
2020-06-10T09:00:36.989+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:39.002+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:39.002+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092它让我怀疑我是否真的得到了9092端口。所以我查看了server.properties.some，我最关心的是：
port=6667
listeners=PLAINTEXT://x.x.x.x:6667然后我再次尝试执行filebeat.conf并将端口9092更改为6667，结果如下
2020-06-10T09:18:01.448+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667
2020-06-10T09:18:01.450+0700 INFO kafka/log.go:53 producer/broker/1001 starting up
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 producer/broker/1001 state change to [open] on test/0
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 producer/leader/test/0 selected broker 1001
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.452+0700 INFO kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.453+0700 DEBUG [kafka] kafka/client.go:264 finished kafka batch
2020-06-10T09:18:01.453+0700 DEBUG [kafka] kafka/client.go:278 Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [normal]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 abandoning broker 1001
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/broker/1001 shut down问题
发生什么事了？应该使用哪个端口？每个端口有什么用？
任何回应都将不胜感激。谢谢您
更新
根据这个来源，正确的来源是6667，因为Kafka是通过Ambari安装
1条答案
按热度按时间sgtfey8w1#
可使用的端口没有限制，仅取决于可用性。在第一种情况下，如您所说，代理可以在6667上启动，因此9092上没有运行任何进程。
2020-06-10t09:18:01.451+0700 info kafka/log.go:53无法连接到代理x.x.x.x:6667:拨号tcp:查找x.x.x.x:没有这样的主机
接下来，当你提到
advertised.listeners您应该确保您在advertised.listeners是分配给那台机器的ip。您不能分配1.1.1.1:9092（举个例子）。执行
ifconfig（linux版），ipconfig（windows）并在可从应用程序计算机访问的网络接口上查看计算机的ip。在linux中，主要是eth0必须可以从运行应用程序的计算机访问此ip。因此，运行应用程序的计算机应该能够解析该ip。您可能还需要检查kafka代理和运行应用程序的计算机之间的网络连接。