配置dockerized fluentd从远程源接收日志行

wbrvyc0a  于 2023-05-22  发布在  Docker
关注(0)|答案(2)|浏览(160)

我正在尝试使用dockerizedfluentd通过tcp接收一些日志行。可以在 fluent.conf 中为每个日志行格式设置一个regexpparser块,将其转换为fluentdy time-tag-jsonmsg 格式。
但是,我无法访问日志行本身的创建方式。它们可能看起来像这样:

10:26:30 WARNING [PyClass.method:135]: Some text.

以下是我已经尝试过的:
Dockerfile没有什么特别的:

FROM fluent/fluentd:v1.11-1

# This line is important. Apparently the default user is "fluent". However, root is needed for apk.
# https://github.com/fluent/fluentd-docker-image/issues/21
USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

COPY log_fluentd/config/fluent.conf /fluentd/etc/
# COPY log_fluentd/config/entrypoint.sh /bin/  # << Led to trouble but there already is such a file.

# Default port for Forward
EXPOSE 24224
# Default port for TCP
EXPOSE 5170

USER fluent

docker-compose up-ping该图像(项目名为 paws)通过

server_fluentd:
    image: paws_server_fluentd:latest
    container_name: server_fluentd
    environment:
      - "FLUENTD_CONF=fluent.conf"
    ports:
      - "24224:24224"
      - "24224:24224/udp"
      - "5170:5170"
    networks:
      - paws
    volumes:
      - "paws_log:/fluentd/log"

最后,这是我从文档中派生出来的配置文件的相关部分:

<system>
  workers 1
  @log_level info
  root_dir /fluentd/log
</system>

<source>
  @type tcp
  @label @mainstream
  @id pawc
  tag myapp.tcp # required
  port 5170 # defaults to 5170
  bind 0.0.0.0
  # https://docs.fluentd.org/parser/regexp
  <parse>
    @type regexp
    # Example from PAWC log line: "10:26:29 INFO [trifles.config:114]: Some text."
    expression /^(?<logtime>[^\s]+) (?<loglvl>[^\s]+) \[(?<file>[^\]:]+):(?<line>\d+)\]: (?<msg>.*)$/
    time_key logtime
    time_format %H:%M:%S
    types line:integer
  </parse>
</source>

<filter **>
  @type stdout
</filter>

<label @mainstream>
  <match myapp.tcp>
    @type file
    @id output_tcp
    path /fluentd/log/tcp.*.log
    symlink_path /fluentd/log/tcp.log
  </match>
</label>

这(以及隐藏的http内容)导致了这个容器日志:

[info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
[info]: gem 'fluent-plugin-elasticsearch' version '4.1.1'
[info]: gem 'fluentd' version '1.11.1'
[warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: using configuration file: <ROOT> [.. shortened ..]
[info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
[info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
[info]: adding match in @mainstream pattern="docker.**" type="file"
[info]: adding match in @mainstream pattern="myapp.tcp" type="file"
[info]: adding match in @mainstream pattern="myapp.access" type="file"
[info]: adding filter pattern="**" type="stdout"
[info]: adding source type="tcp"
[info]: adding source type="http"
[warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
[info]: #0 fluentd worker is now running worker=0
  fluent.info: {"pid":20,"ppid":6,"worker":0,"message":"starting fluentd worker pid=20 ppid=6 worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"
  fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"

在我的主机上,我启动我的python3.7说

data_to_send = bytes("10:26:30 WARNING [ServiceBase.subservice:135]: Some text.", 'utf-8')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 5170))
s.sendall(data_to_send)
s.close()

惩罚我却毫无React容器中至少有一个(空)文件树:

/fluentd/log # tree
.
├── http.log -> /fluentd/log/worker0/output_http/buffer/buffer.b5ac49abfabe305660f691eb4d5e782c2.log
└── worker0
    [.. shortened ..]
    └── output_tcp
        └── buffer

13 directories, 3 files [none of them about tcp]

我怀疑这和涂鸦有关。你知道我想念什么吗?

hrysbysz

hrysbysz1#

其实一切都很简单:正则表达式希望行结束符\n与其$字符匹配。

eyh26e7m

eyh26e7m2#

在我的例子中,改变:
<match **>
致:
<match **-**>
修复了问题中发布的不推荐使用的警告。

相关问题