java模块编写单行日志:
{"timestamp":"2020-09-29 10:46:18.761","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}{"timestamp":"2020-09-29 10:49:11.338","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}...
每个新事件都粘贴到行的末尾。在fluentd设置中,我使用标准解析器:
[PARSER]
Name java-system
Format json
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep On
如果我将日志中的事件分成几行,那么代理会将所有内容发送到elastic,而不会出现任何问题。如果所有东西都在一条线上,它就不会发送任何东西。它不会在日志中写入任何信息。
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.0 tag=gw_system
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.0:es.0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.1 tag=calc-system
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.1:es.3
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.2 tag=push-system
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.2:es.2
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.3 tag=gw_integration
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.3:es.1
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [ info] [sp] stream processor started
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log read=35484 lines=0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log read=336548 lines=0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log read=12302 lines=0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.3] file=/opt/java_services/gateway/log/integration.log promote to TAIL_EVENT
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log promote to TAIL_EVENT
编写了一个单独的解析器
[PARSER]
Name java_push_system
Format regex
Regex (?<date>(?<={"timestamp":")(\d+\-\d+\-\d+\s+\d+:\d+:\d+.\d+))",(?<message>[^}]*)
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep On
在regex101.com站点上,我的字符串被解析,但是代理仍然不发送任何内容,直到它被拆分成单独的行。
/etc/td-agent-bit/conf]# cat push-system*
[FILTER]
Name record_modifier
Match push-system
Record hostname ${HOSTNAME}
Record environment DEV
Record path /opt/java_services/push/log/system.log
[INPUT]
Name tail
Tag push-system
Parser java-system
Path /opt/java_services/push/log/system.log
DB /etc/td-agent-bit/system-push.db
Buffer_Max_Size 32MB
Buffer_Chunk_Size 8MB
[OUTPUT]
Name es
Match push-system
Host 10.*
Port 9200
HTTP_User *
HTTP_Passwd *
Index push
请帮助理解!
暂无答案!
目前还没有任何答案,快来回答吧!