我正在尝试使用dockerizedfluentd通过tcp接收一些日志行。可以在 fluent.conf 中为每个日志行格式设置一个regexpparser块,将其转换为fluentdy time-tag-jsonmsg 格式。
但是,我无法访问日志行本身的创建方式。它们可能看起来像这样:
10:26:30 WARNING [PyClass.method:135]: Some text.
以下是我已经尝试过的:
Dockerfile没有什么特别的:
FROM fluent/fluentd:v1.11-1
# This line is important. Apparently the default user is "fluent". However, root is needed for apk.
# https://github.com/fluent/fluentd-docker-image/issues/21
USER root
RUN apk add --no-cache --update --virtual .build-deps \
sudo build-base ruby-dev \
&& sudo gem install fluent-plugin-elasticsearch \
&& sudo gem sources --clear-all \
&& apk del .build-deps \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
COPY log_fluentd/config/fluent.conf /fluentd/etc/
# COPY log_fluentd/config/entrypoint.sh /bin/ # << Led to trouble but there already is such a file.
# Default port for Forward
EXPOSE 24224
# Default port for TCP
EXPOSE 5170
USER fluent
docker-compose up
-ping该图像(项目名为 paws)通过
server_fluentd:
image: paws_server_fluentd:latest
container_name: server_fluentd
environment:
- "FLUENTD_CONF=fluent.conf"
ports:
- "24224:24224"
- "24224:24224/udp"
- "5170:5170"
networks:
- paws
volumes:
- "paws_log:/fluentd/log"
最后,这是我从文档中派生出来的配置文件的相关部分:
<system>
workers 1
@log_level info
root_dir /fluentd/log
</system>
<source>
@type tcp
@label @mainstream
@id pawc
tag myapp.tcp # required
port 5170 # defaults to 5170
bind 0.0.0.0
# https://docs.fluentd.org/parser/regexp
<parse>
@type regexp
# Example from PAWC log line: "10:26:29 INFO [trifles.config:114]: Some text."
expression /^(?<logtime>[^\s]+) (?<loglvl>[^\s]+) \[(?<file>[^\]:]+):(?<line>\d+)\]: (?<msg>.*)$/
time_key logtime
time_format %H:%M:%S
types line:integer
</parse>
</source>
<filter **>
@type stdout
</filter>
<label @mainstream>
<match myapp.tcp>
@type file
@id output_tcp
path /fluentd/log/tcp.*.log
symlink_path /fluentd/log/tcp.log
</match>
</label>
这(以及隐藏的http内容)导致了这个容器日志:
[info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
[info]: gem 'fluent-plugin-elasticsearch' version '4.1.1'
[info]: gem 'fluentd' version '1.11.1'
[warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: using configuration file: <ROOT> [.. shortened ..]
[info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
[info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
[info]: adding match in @mainstream pattern="docker.**" type="file"
[info]: adding match in @mainstream pattern="myapp.tcp" type="file"
[info]: adding match in @mainstream pattern="myapp.access" type="file"
[info]: adding filter pattern="**" type="stdout"
[info]: adding source type="tcp"
[info]: adding source type="http"
[warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
[info]: #0 fluentd worker is now running worker=0
fluent.info: {"pid":20,"ppid":6,"worker":0,"message":"starting fluentd worker pid=20 ppid=6 worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"
fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"
在我的主机上,我启动我的python3.7说
data_to_send = bytes("10:26:30 WARNING [ServiceBase.subservice:135]: Some text.", 'utf-8')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 5170))
s.sendall(data_to_send)
s.close()
惩罚我却毫无React容器中至少有一个(空)文件树:
/fluentd/log # tree
.
├── http.log -> /fluentd/log/worker0/output_http/buffer/buffer.b5ac49abfabe305660f691eb4d5e782c2.log
└── worker0
[.. shortened ..]
└── output_tcp
└── buffer
13 directories, 3 files [none of them about tcp]
我怀疑这和涂鸦有关。你知道我想念什么吗?
2条答案
按热度按时间hrysbysz1#
其实一切都很简单:正则表达式希望行结束符
\n
与其$
字符匹配。eyh26e7m2#
在我的例子中,改变:
<match **>
致:
<match **-**>
修复了问题中发布的不推荐使用的警告。