elasticsearch 无法发布事件:临时批量发送失败

xuo3flqw  于 2023-01-04  发布在  ElasticSearch
关注(0)|答案(1)|浏览(288)

当我尝试在filebeat.yml中创建多个索引并输出到elasticsearch时,我得到临时的批量发送失败错误。只有当我将ilm设置为disable时才会出现这种情况。有人能帮忙吗
以下是filebeat配置

filebeat.inputs:
- type: filestream
  id: denali
  enabled: true
  paths:
    - /var/log/denali/denali.log
  parsers:
    - multiline:
        type: pattern
        pattern: '^(\d{4}-\d{2}-\d{2})'
        negate: true
        match: after
  fields:
    app_id: denali

- type: filestream
  id: freeswitch
  enabled: true
  paths:
    - /var/log/freeswitch/freeswitch.log
  parsers:
    - multiline:
        type: pattern
        pattern: '^((\d|[a-z]|-)+ \d{4}-\d{2}-\d{2}|\d{4}-\d{2}-\d{2})'
        negate: true
        match: after
  fields:
    app_id: freeswitch

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.enabled: true
setup.ilm.enabled: false
setup.template.overwrite: true
setup.template.name: "index-%{[agent.version]}"
setup.template.pattern: "index-%{[agent.version]}-*"

output.elasticsearch:
  hosts: ["ip:port"]
  index: "index-%{[agent.version]}-%{[fields.app_id]:other}-%{+yyyy.MM.dd}"
  protocol: "http"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.name", "agent.type", "agent.version", "cloud.account.id", "cloud.provider", "cloud.service.name", "container.id", "container.image.name", "container.labels.COMMIT", "container.labels.PIPELINE_URL", "container.labels.PROJECT_NAME", "container.labels.PROJECT_URL", "container.labels.SOURCE_BRANCH", "container.labels.TimeStamp", "container.labels.RELEASEARTIFACT_VERSION", "container.labels.com_docker_compose_config-hash", "container.labels.com_docker_compose_container-number", "container.labels.com_docker_compose_oneoff", "container.labels.com_docker_compose_project", "container.labels.com_docker_compose_project_config_files", "container.labels.com_docker_compose_project_working_dir", "container.labels.com_docker_compose_service", "container.labels.com_docker_compose_version", "ecs.version", "host.architecture", "host.containerized", "host.id", "host.mac", "host.os.codename", "host.os.family", "host.os.kernel", "host.os.name", "host.os.platform", "host.os.type", "host.os.version", "log.offset"]
y1aodyip

y1aodyip1#

@Ramanichandran您能提供filebeat的错误日志吗?另外,当filebeat尝试发送日志进行摄取时,您是否在ES日志上看到任何错误?
我不认为这是由于创建了多个索引,因为您实际上只创建了3个索引。在我的使用案例中,我配置了filebeat来创建大约15个索引,在我的配置与您的配置类似但禁用了ILM的情况下,它工作得很好。
值得尝试为output.elasticsearch设置以下属性:

bulk_max_size: 25
bulk_max_bytes: 104857600

相关问题