logstash 从mysql获取时间戳不起作用,将其转换为字符串是否起作用?

bvjxkvbb  于 2022-12-16  发布在  Logstash
关注(0)|答案(2)|浏览(192)

我使用这个conf文件来覆盖ElasticSearch中的@timestamp字段,但是我会自动得到一个_dateparsefailure标志:

input {
    jdbc {
        jdbc_driver_library => "C:/path/to/mariadb-java-client.jar"
        statement => "SELECT '${FIELD}' as field, from ${TABLE_NAME}"
        tracking_column => "timestamp"
        tracking_column_type => "timestamp" 
    }
}

filter {
    grok {
        match => ["timestamp","%{TIMESTAMP_ISO8601}"]
    }
    date {
        match => ["timestamp", "ISO8601"]
    }
}

请注意,无论是否使用grok过滤器,我都得到了相同的结果。
结果是:

{
    "@timestamp" => 2022-12-13T09:16:10.365Z,
    "timestamp" => 2022-11-23T10:36:13.000Z,              
    "@version" => "1",
    "tags" => [
        [0] "_dateparsefailure"
    ],
    "type" => "mytype",
}

但是当我用这个conf提取时间戳时:

input {
    *same input*
}

filter {
    grok {
        match => ["timestamp","%{TIMESTAMP_ISO8601:tmp}"]
        tag_on_failure => [ "_grokparsefailure"]
    }
    date {
        match => ["tmp", "ISO8601"]
    }
}


那么它会给予我预期的结果:

{
    "@timestamp" => 2022-11-23T11:16:36.000Z,
    "@version" => "1",
    "timestamp" => 2022-11-23T11:16:36.000Z,
    "tmp" => "2022-11-23T11:16:36.000Z",
}

有人能解释一下为什么会这样吗?我怎样才能避免产生这个额外的字段呢?谢谢

sdnqo3pr

sdnqo3pr1#

好的,我猜第一次解析字符串,但是timestamp已经有了正确的类型,所以一个副本就足以保存和覆盖@timestamp字段:

filter {
    mutate {
        copy => { "@timestamp" => "insertion_timestamp" }
        copy => { "timestamp" => "@timestamp" }
        remove_field => [ "timestamp" ]
    }
}
jgovgodb

jgovgodb2#

如果数据库列类型是时间戳,那么jdbc输入会自动将字段转换为LogStash::Timestamp对象,而不是字符串。日期过滤器cannot parse转换为Timestamp对象,并添加_dateparsefailure标记。
grok过滤器调用.to_s,在匹配之前将所有内容转换为字符串,因此如果从Timestamp对象中获取时间戳,它将是日期过滤器 * 可以 * 解析的字符串。

相关问题