无法参数化输出Logstash索引- AWS s3输入

pcww981p  于 2022-12-09  发布在  Logstash
关注(0)|答案(1)|浏览(154)

早上好,
我们正在尝试使用Logstash从AWS s3收集日志。下面是Logstash.conf文件:

input {

  tcp {
    port => 5000
    codec => json
  }

  s3 {
    # Keys for user in prod env
    id => "prod" 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

  s3 {
    # Keys for user in preprod env
    id => "preprod"
    access_key_id => "key2" 
    secret_access_key => "secretkey2"
    region => "eu-central-1"
    bucket => "bucket2"
    prefix => "prefix2/"
    include_object_properties => true
  }
  
}

filter {

  # Matching prod logs
  if [id] == "prod" {
    mutate {
      add_field => { "env" => "prod" }
    }
  }
  else if [id] == "preprod" {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }

}

output {
  opensearch {
    hosts       => "https://osb-platform"
      user => "username"
      password => "password"
    index       => "4s-%{[env]}-logs-%{+YYYY.MM.dd}"
    ecs_compatibility => disabled
    ssl_certificate_verification => false
  }
}

正如您所看到的,我有来自两个不同环境的两个输入,我希望将来自不同环境的日志放在不同的索引上。这是通过对输入的id进行过滤来完成的,方法是根据env添加一个新字段。
正确收集日志,但未区分索引:在OpenSearch中,我可以看到一个名为4s-%{[env]}-logs-2022.05.30的索引,它似乎无法解析env变量。
你能帮帮我吗?
谢谢你,
卢卡

o2rvlv0m

o2rvlv0m1#

任何输入中的id选项用于在监视logstash时帮助标识输入,由于文档中不存在该选项,因此不用于过滤。
您需要使用标签或类型选项来执行此操作。
例如,使用type

s3 {
    # Keys for user in prod env
    type => "prod" 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

那么您的筛选器将是:

filter {
  if [type] == "prod" {
    mutate {
      add_field => { "env" => "prod" }
    }
  } else if [type] == "preprod" {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }
}

如果你想使用tags,你需要这样的代码:
第一次

相关问题