如何连接fluentd到opendisto进行ElasticSearch

9w11ddsr  于 2022-12-22  发布在  ElasticSearch
关注(0)|答案(4)|浏览(167)

我已经创建了一个docker文件:

FROM fluentd:v1.14.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.1.2"]
USER fluent

我在合成文件中创建了一个容器:

version: '3'
services:
  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    restart: always
    container_name: fluentd
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - network
  elasticsearch:
    image: docker pull amazon/opendistro-for-elasticsearch:latest
    container_name: elasticsearch
    environment:
      - cluster.name=elasticsearch
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - network
  kibana:
    image: docker pull amazon/opendistro-for-elasticsearch-kibana:latest
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://elasticsearch:9200
      ELASTICSEARCH_HOSTS: https://elasticsearch:9200
    networks:
      - network 
volumes:
  elasticsearch:

networks:
  network:
    driver: bridge
    name: network

这是我的fluent.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    user admin
    password admin
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    scheme https
    ssl_verify false
  </store>

  <store>
    @type stdout
  </store>
</match>

由于某种原因,elasticsearch和kibana启动时没有问题,但fluentd无法连接。这是我得到的错误
一个三个三个一个
我不确定用于fluentd连接到elasticsearch的插件是否也可以连接到OpenDistro进行elasticsearch。如果需要任何其他信息,请询问。任何帮助将不胜感激。

8iwquhpp

8iwquhpp1#

使用稳定的最新版本的插件elasticsearchfluent-plugin-elasticsearch。我工作的Dockerfile例如:
来自流利语言/流利语言:v1.12.0-debian-1.0
用户根
运行["gem","安装","ElasticSearch","--无文档","--版本","7.13.3"]
运行["gem","安装","流利插件ElasticSearch","--无文档","--版本","5.0.5"]
用户流利

s1ag04yj

s1ag04yj2#

问题是,插件工程与较低版本的elasticsearch,重构后,我的docker文件一切工作作为一个魅力:

FROM fluentd:v1.14.0-debian-1.0
USER root
RUN gem install elasticsearch -v 7.13.3 \
&& gem install fluent-plugin-elasticsearch --no-document
USER fluent
hec6srdp

hec6srdp3#

我在测试与elasticsearch7.10.2服务器的连接
我删除了从recentd版本到8.x的fluentd插件,并安装了我正在使用的版本的插件(7)本节帮助我调试该问题

RUN fluent-gem uninstall --force fluent-plugin-elasticsearch
RUN fluent-gem uninstall --force elasticsearch
RUN fluent-gem uninstall --force elastic_ruby_console
RUN fluent-gem uninstall -a --force elasticsearch-api
RUN fluent-gem uninstall -a --force elastic-transport
RUN fluent-gem uninstall -a --force elasticsearch-xpack
RUN fluent-gem install elasticsearch -v 7.10
RUN fluent-gem install fluent-plugin-elasticsearch
dldeef67

dldeef674#

它在工作

version: '3.7'
services:
  fluentd:
    image: cr.fluentbit.io/fluent/fluent-bit

相关问题