概述

我正试图让logstash和Kafka合作，使用docker容器。
对于我的kafka容器，我使用spotify构建。我也在使用官方的logstash容器。
我想实现以下目标：

以下是我用于日志存储的配置：
文件\u kafka.conf

input {
    file {
        path => ["/data/*.log"]
        start_position => "beginning"
    }
}

output {
    kafka {
        topic_id => "test"
        bootstrap_servers => "<host>:9092"
    }
}

Kafka标准输出.conf

input {
    kafka {
        topics => ["test"]
        bootstrap_servers => "<host>:9092"
        auto_offset_reset => "latest"
    }
}

output {
    stdout {}
}

对我来说，这种设置似乎完全合理。事实上，一开始还挺管用的。

问题

我启动了相关的容器，logstash正确地在我的 test.log 文件并打印到屏幕上。
但是，我运行了以下命令： echo 'blah' >> test.log 我收到了logstash的输出： 2017-05-11T18:36:49.090Z %{host} 2017-05-11T18:36:49.082Z %{host} 2017-05-11T18:36:49.076Z %{host} 2017-05-11T18:36:49.069Z %{host} 2017-05-11T18:36:49.062Z %{host} 2017-05-11T18:36:49.053Z %{host} 2017-05-11T18:36:49.044Z %{host} 2017-05-11T18:36:49.034Z %{host} 2017-05-11T18:36:49.025Z %{host} 2017-05-11T18:36:49.016Z %{host} 2017-05-11T18:36:49.009Z %{host} 2017-05-11T18:36:49.003Z %{host} 2017-05-11T18:36:48.995Z %{host} 2017-05-11T18:36:48.988Z %{host} 2017-05-11T18:36:48.982Z %{host} 2017-05-11T18:36:48.975Z %{host} 2017-05-11T18:36:48.968Z %{host} 2017-05-11T18:36:48.948Z %{host} 2017-05-11T18:36:48.927Z %{host} 2017-05-11T18:36:48.911Z %{host} 2017-05-11T18:36:48.896Z %{host} 2017-05-11T18:36:48.881Z %{host} 2017-05-11T18:36:48.868Z %{host} 2017-05-11T18:36:48.857Z %{host} 2017-05-11T18:36:48.849Z %{host} 2017-05-11T18:36:48.841Z %{host} 2017-05-11T18:36:48.830Z %{host} 2017-05-11T18:36:48.820Z %{host} 2017-05-11T18:36:48.805Z %{host} 2017-05-11T18:36:48.791Z %{host} 2017-05-11T18:36:48.779Z %{host} 2017-05-11T18:36:48.492Z %{host} 2017-05-11T18:36:48.483Z %{host} 2017-05-11T18:36:48.471Z %{host} 2017-05-11T18:36:48.463Z %{host} 2017-05-11T18:36:48.454Z %{host} 2017-05-11T18:36:48.445Z %{host} 2017-05-11T18:36:48.437Z %{host} 2017-05-11T18:36:48.427Z %{host} 2017-05-11T18:36:48.420Z %{host} 2017-05-11T18:36:48.412Z %{host} 2017-05-11T18:36:48.406Z %{host} 2017-05-11T18:36:48.399Z %{host} 2017-05-11T18:36:48.391Z %{host} 2017-05-11T18:36:48.385Z %{host} 2017-05-11T18:36:48.378Z %{host} 2017-05-11T18:36:48.371Z %{host} 2017-05-11T18:36:48.356Z %{host} 2017-05-11T18:36:48.286Z %{host} 2017-05-11T18:36:48.278Z %{host} 2017-05-11T18:36:48.269Z %{host} 2017-05-11T18:36:48.261Z %{host} 2017-05-11T18:36:48.246Z %{host} 2017-05-11T18:36:48.238Z %{host} 2017-05-11T18:36:48.229Z %{host} 2017-05-11T18:36:48.219Z %{host} 2017-05-11T18:36:48.209Z %{host} 2017-05-11T18:36:48.201Z %{host} 2017-05-11T18:36:48.191Z %{host} 2017-05-11T18:36:48.182Z %{host} 2017-05-11T18:36:48.175Z %{host} 2017-05-11T18:36:48.167Z %{host} 2017-05-11T18:36:48.112Z %{host} 2017-05-11T18:36:48.103Z %{host} 2017-05-11T18:36:48.095Z %{host} 2017-05-11T18:36:48.086Z %{host} 2017-05-11T18:36:48.077Z %{host} 2017-05-11T18:36:48.068Z %{host} 2017-05-11T18:36:48.059Z %{host} 2017-05-11T18:36:48.048Z %{host} 2017-05-11T18:36:48.031Z %{host} 2017-05-11T18:36:48.016Z %{host} 2017-05-11T18:36:48.009Z %{host} 2017-05-11T18:36:48.002Z %{host} 2017-05-11T18:36:47.994Z %{host} 2017-05-11T18:36:47.986Z %{host} 2017-05-11T18:36:47.972Z %{host} 2017-05-11T18:36:47.947Z %{host} 2017-05-11T18:36:47.923Z %{host} 2017-05-11T18:36:47.904Z %{host} 2017-05-11T18:36:47.879Z %{host} 2017-05-11T18:36:47.864Z %{host} 2017-05-11T18:36:47.853Z %{host} 2017-05-11T18:36:47.842Z %{host} 2017-05-11T18:36:47.834Z %{host} 2017-05-11T18:36:47.820Z %{host} 2017-05-11T18:36:47.810Z %{host} 2017-05-11T18:36:47.799Z %{host} 2017-05-11T18:36:47.792Z %{host} 2017-05-11T18:36:47.785Z %{host} 2017-05-11T18:36:47.778Z %{host} 2017-05-11T18:36:47.768Z %{host} 2017-05-11T18:36:47.757Z %{host} 2017-05-11T18:36:47.741Z %{host} 2017-05-11T18:36:47.722Z %{host} 2017-05-11T18:36:47.702Z %{host} 2017-05-11T18:36:47.690Z %{host} 2017-05-11T18:36:47.674Z %{host} 2017-05-11T18:36:47.667Z %{host} 2017-05-11T18:36:47.659Z %{host} 2017-05-11T18:36:47.649Z %{host} 2017-05-11T18:36:47.639Z %{host} 2017-05-11T18:36:47.632Z %{host} 2017-05-11T18:36:47.626Z %{host} 2017-05-11T18:36:47.619Z %{host} 2017-05-11T18:36:47.612Z %{host} 2017-05-11T18:36:47.603Z %{host} 2017-05-11T18:36:47.592Z %{host} 2017-05-11T18:36:47.555Z %{host} 2017-05-11T18:36:47.537Z %{host} 2017-05-11T18:36:47.523Z %{host} 2017-05-11T18:36:47.513Z %{host} 2017-05-11T18:36:47.505Z %{host} 2017-05-11T18:36:47.492Z %{host} 2017-05-11T18:36:47.484Z %{host} 2017-05-11T18:36:47.476Z %{host} 2017-05-11T18:36:47.468Z %{host} 2017-05-11T18:36:47.456Z %{host} 2017-05-11T18:36:47.447Z %{host} 2017-05-11T18:36:47.437Z %{host} 2017-05-11T18:36:47.425Z %{host} 2017-05-11T18:36:47.416Z %{host} 2017-05-11T18:36:47.403Z %{host} 2017-05-11T18:36:47.395Z %{host} 2017-05-11T18:36:47.388Z %{host} 2017-05-11T18:36:47.379Z %{host} 2017-05-11T18:36:47.372Z %{host} 2017-05-11T18:36:47.364Z %{host} 2017-05-11T18:36:47.351Z %{host} 2017-05-11T18:36:47.318Z %{host} 2017-05-11T18:36:47.308Z %{host} 2017-05-11T18:36:47.293Z %{host} 2017-05-11T18:36:47.274Z %{host} 2017-05-11T18:36:47.255Z %{host} 2017-05-11T18:36:47.244Z %{host} 2017-05-11T18:36:47.232Z %{host} 2017-05-11T18:36:47.215Z %{host} 2017-05-11T18:36:47.205Z %{host} 2017-05-11T18:36:47.197Z %{host} 2017-05-11T18:36:47.188Z %{host} 2017-05-11T18:36:47.180Z %{host} 2017-05-11T18:36:47.170Z %{host} 2017-05-11T18:36:47.162Z %{host} 2017-05-11T18:36:47.155Z %{host} 2017-05-11T18:36:47.147Z %{host} 2017-05-11T18:36:47.141Z %{host} 2017-05-11T18:36:47.134Z %{host} 2017-05-11T18:36:47.127Z %{host} 2017-05-11T18:36:47.119Z %{host} 2017-05-11T18:36:47.111Z %{host} 2017-05-11T18:36:47.096Z %{host} 2017-05-11T18:36:47.082Z %{host} 2017-05-11T18:36:47.064Z %{host} 2017-05-11T18:36:47.047Z %{host} 2017-05-11T18:36:47.034Z %{host} 2017-05-11T18:36:47.026Z %{host} 2017-05-11T18:36:47.014Z %{host} 2017-05-11T18:36:47.006Z %{host} 2017-05-11T18:36:46.996Z %{host} 2017-05-11T18:36:46.975Z %{host} 2017-05-11T18:36:46.965Z %{host} 2017-05-11T18:36:46.935Z %{host} 2017-05-11T18:36:46.928Z %{host} 2017-05-11T18:36:46.921Z %{host} 2017-05-11T18:36:46.914Z %{host} 2017-05-11T18:36:46.907Z %{host} 2017-05-11T18:36:46.899Z %{host} 2017-05-11T18:36:46.888Z %{host} 2017-05-11T18:36:46.867Z %{host} 2017-05-11T18:36:46.845Z %{host} 2017-05-11T18:36:46.833Z %{host} 2017-05-11T18:36:46.817Z %{host} 2017-05-11T18:36:46.798Z %{host} 2017-05-11T18:36:46.783Z %{host} 2017-05-11T18:36:46.759Z %{host} 2017-05-11T18:36:46.743Z %{host} 2017-05-11T18:36:46.725Z %{host} 2017-05-11T18:36:46.712Z %{host} 2017-05-11T18:36:46.695Z %{host} 2017-05-11T18:36:46.685Z %{host} 2017-05-11T18:36:46.677Z %{host} 2017-05-11T18:36:46.666Z %{host} 2017-05-11T18:36:46.659Z %{host} 2017-05-11T18:36:46.642Z %{host} 2017-05-11T18:36:46.631Z %{host} 2017-05-11T18:36:46.620Z %{host} 2017-05-11T18:36:46.606Z %{host} 2017-05-11T18:36:46.593Z %{host} 2017-05-11T18:36:46.577Z %{host} 2017-05-11T18:36:46.568Z %{host} 2017-05-11T18:36:46.549Z %{host} 2017-05-11T18:36:46.541Z %{host} 2017-05-11T18:36:46.532Z %{host} 2017-05-11T18:36:46.524Z %{host} 2017-05-11T18:36:46.507Z %{host} 2017-05-11T18:36:46.491Z %{host} 2017-05-11T18:36:46.474Z %{host} 2017-05-11T18:36:46.429Z %{host} 2017-05-11T18:36:46.420Z %{host} 2017-05-11T18:36:46.408Z %{host} 2017-05-11T18:36:46.397Z %{host} 2017-05-11T18:36:46.382Z %{host} 2017-05-11T18:36:46.353Z %{host} 2017-05-11T18:36:46.345Z %{host} 2017-05-11T18:36:46.337Z %{host} 2017-05-11T18:36:46.330Z %{host} 2017-05-11T18:36:46.323Z %{host} 2017-05-11T18:36:46.314Z %{host} 2017-05-11T18:36:46.302Z %{host} 2017-05-11T18:36:46.293Z %{host} 2017-05-11T18:36:46.279Z %{host} 2017-05-11T18:36:46.265Z %{host} 2017-05-11T18:36:46.252Z %{host} 2017-05-11T18:36:46.181Z %{host} 2017-05-11T18:36:46.152Z %{host} 2017-05-11T18:36:46.128Z %{host} 2017-05-11T18:36:46.110Z %{host} 2017-05-11T18:36:46.088Z %{host} 2017-05-11T18:36:46.075Z %{host} 2017-05-11T18:36:46.067Z %{host} 2017-05-11T18:36:46.058Z %{host} 2017-05-11T18:36:46.045Z %{host} 2017-05-11T18:36:46.025Z %{host} 2017-05-11T18:36:46.016Z %{host} 2017-05-11T18:36:46.008Z %{host} 2017-05-11T18:36:45.995Z %{host} 2017-05-11T18:36:45.986Z %{host} 2017-05-11T18:36:45.972Z %{host} 2017-05-11T18:36:45.934Z %{host} 2017-05-11T18:36:45.915Z %{host} 2017-05-11T18:36:45.898Z %{host} 2017-05-11T18:36:45.887Z %{host} 2017-05-11T18:36:45.855Z dc6258001c01 blah 这段文字被无限重复，直到我终止了日志存储过程。好像进入了某种非终止循环。我也在Kafka的例子中看到了我的主题。
所以我有两个问题：
为什么洛斯塔什要一遍又一遍地读同一行，永远？
那封信的格式怎么样了？不是应该是吗 timestamp host line ? 最后看起来是正确的，但是 timestamp %{host} 在此之前，这种模式被重复了232次。为什么？

1条答案

按热度按时间

rggaifut1#

Kafka和拉比都遇到过类似的问题。
logstash并不关心您是否将配置拆分为单独的文件，最终这样做的功能是供人使用（使配置更易于查找和修改）。在运行时，所有日志存储配置在管道中合并在一起，并被视为单个配置。虽然我不一定知道顺序是如何应用的，但它会在单个主机上运行：

input {
    file {
        path => ["/data/*.log"]
        start_position => "beginning"
    }
    kafka {
        topics => ["test"]
        bootstrap_servers => "<host>:9092"
        auto_offset_reset => "latest"
    }
}

output {
    kafka {
        topic_id => "test"
        bootstrap_servers => "<host>:9092"
    }
    stdout {}
}

假设您的一个输入被设置为读取上次写入的偏移量，它将只读取该偏移量，不应用过滤器，并将其输出到同一主题。kafka代理不知道消息本身是什么，但是kafka（最终是zookeeper）跟踪发布者编写的消息的偏移量，以及订阅者读取的消息的偏移量。
您需要运行一个单独的logstash示例。为了在不使用第二个配置（使用stdout）的情况下验证输入，我建议使用kafkacat阅读kafka主题（也可以用于发布消息）。

kafkacat -C -b <broker>:9092 -t <topic>

唯一允许您使用单个logstash示例的替代方法（我知道是可行的）是您必须使用 mutate 中的方法 filter 但是，这会使逻辑复杂化，最好使用logstash遵循shipper/indexer模式。

赞(0）回复(0）举报 2021-06-09

通过kafka使用logstash进行无限日志记录

概述

问题

1条答案

相关问题

热门标签

最新问答