使用fluentd展平嵌套JSON

g52tjvyc  于 2023-03-09  发布在  其他
关注(0)|答案(1)|浏览(154)

我有一个写结构化日志的程序,下面的例子适用:

{
    "time": "time_val",
    "log": "{
        \"field1\": \"value1\",
        \"field2\": \"value2\",
        \"field3\": \"{
            \"nested_field1\": \"value1\",
            \"nested_field2\": \"value2\",
            \"nested_field3\": \"value3\"
        }\"
    }"
}

我正在使用fluentd跟踪容器的输出,并解析JSON消息,但是,我希望解析嵌套的结构化日志,以便它们在原始消息中被扁平化。例如,我希望fluentd最终将消息视为:

{
    "time": "time_val",
    "field1": "value1",
    "field2": "value2",
    "nested_field1": "value1",
    "nested_field2": "value2",
    "nested_field3": "value3"
}

这是可以使用fluentd配置来完成的吗?在我的情况下,改变原始程序行为不是一个选项。

7nbnzgx9

7nbnzgx91#

您可以使用带有key_name、reserve_data和remove_key_name字段的parser过滤器插件。
示例:

<filter **>
  @type parser
  key_name field3
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

以下是使JSON有效后完整的工作示例,即:

{"field1":"value1","field2":"value2","field3":"{\"nested_field1\":\"value1\",\"nested_field2\":\"value2\",\"nested_field3\":\"value3\"}"}

流畅-展平-json.配置

<source>
  @type forward
</source>

<filter **>
  @type parser
  key_name field3
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

<match **>
  @type stdout
</match>

运行fluentd

fluentd -c ./fluent-flatten-json.conf

从另一个终端运行fluent-cat,输入JSON:

fluent-cat test <<< '{"field1":"value1","field2":"value2","field3":"{\"nested_field1\":\"value1\",\"nested_field2\":\"value2\",\"nested_field3\":\"value3\"}"}'

fluentd日志中的输出:

{"field1":"value1","field2":"value2","nested_field1":"value1","nested_field2":"value2","nested_field3":"value3"}

格式化输出:

{
  "field1": "value1",
  "field2": "value2",
  "nested_field1": "value1",
  "nested_field2": "value2",
  "nested_field3": "value3"
}

相关问题