fluent位日志不会使用fluentd发送到elasticsearch日志写在一行中

u0njafvf  于 2021-06-10  发布在  ElasticSearch
关注(0)|答案(0)|浏览(327)

java模块编写单行日志:

{"timestamp":"2020-09-29 10:46:18.761","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}{"timestamp":"2020-09-29 10:49:11.338","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}...

每个新事件都粘贴到行的末尾。在fluentd设置中,我使用标准解析器:

[PARSER]
    Name         java-system
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%d %H:%M:%S.%L
    Time_Keep    On

如果我将日志中的事件分成几行,那么代理会将所有内容发送到elastic,而不会出现任何问题。如果所有东西都在一条线上,它就不会发送任何东西。它不会在日志中写入任何信息。

Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.0 tag=gw_system
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.0:es.0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.1 tag=calc-system
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.1:es.3
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.2 tag=push-system
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.2:es.2
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.3 tag=gw_integration
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.3:es.1
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [ info] [sp] stream processor started
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log read=35484 lines=0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log read=336548 lines=0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log read=12302 lines=0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.3] file=/opt/java_services/gateway/log/integration.log promote to TAIL_EVENT
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log promote to TAIL_EVENT

编写了一个单独的解析器

[PARSER]
    Name    java_push_system
    Format  regex
    Regex   (?<date>(?<={"timestamp":")(\d+\-\d+\-\d+\s+\d+:\d+:\d+.\d+))",(?<message>[^}]*)
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Keep    On

在regex101.com站点上,我的字符串被解析,但是代理仍然不发送任何内容,直到它被拆分成单独的行。

/etc/td-agent-bit/conf]# cat push-system*
[FILTER]
    Name   record_modifier
    Match  push-system
    Record hostname ${HOSTNAME}
    Record environment DEV
    Record path /opt/java_services/push/log/system.log
[INPUT]
    Name   tail
    Tag    push-system
    Parser java-system
    Path   /opt/java_services/push/log/system.log
    DB     /etc/td-agent-bit/system-push.db
    Buffer_Max_Size 32MB
    Buffer_Chunk_Size 8MB
[OUTPUT]
    Name        es
    Match       push-system
    Host        10.*
    Port        9200
    HTTP_User   *
    HTTP_Passwd *
    Index       push

请帮助理解!

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题