Logstash未将内容摄取到ElasticSearch中

7fhtutme 于 2022-11-02 发布在 ElasticSearch

关注(0)|答案(2)|浏览(175)

我已经安装了elasticsearch-8.2.3 logstash-8.2.3和kibana-8.2.3我已经配置了logstash conf文件，以将内容摄取到elasticsearch中，logstash运行时没有任何错误，但它没有摄取内容。
下面是conf文件：

input {
  #stdin {type => "stdin-type" }
  file
  {
    path => "D:/logstash-8.2.3/inspec/*.*"
    type => "file"
    start_position=>"beginning"
    sincedb_path => "NUL"
    ignore_older => 0
  }
}
filter {
    csv
    {
        columns =>
                    [
                      "itemid","itemtitle","rlabel","ayear","rid","rsid","anotatedby","anotatetime","antype","astate","broaderlevel3","broaderlevel2","broaderlevel1","categorylabel","toppreferedlabel"
                    ]
        separator => ","
        remove_field => ["type","host"]
        }

    mutate 
    {
    split => { "antype" => ";" }
    split => { "broaderlevel3" => ";" }
    split => { "broaderlevel2" => ";" }
    split => { "broaderlevel1" => ";" }
    split => { "categorylabel" => ";" }
    split => { "toppreferedlabel" => ";" }

    }
}

output {
  stdout { }
  elasticsearch
   {
        hosts => ["localhost"]

        index => "iet-tv"
   }
}

我没有得到任何错误消息，而运行logstash，但内容没有得到摄取到ElasticSearch。
下面是日志：

[2022-06-29T14:03:03,579][INFO ][logstash.runner          ] Log4j configuration path used is: D:\logstash-8.2.3\config\log4j2.properties
[2022-06-29T14:03:03,595][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-06-29T14:03:03,598][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.2.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [mswin32-x86_64]"}
[2022-06-29T14:03:03,600][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-29T14:03:03,736][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-29T14:03:11,340][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-29T14:03:12,628][INFO ][org.reflections.Reflections] Reflections took 153 ms to scan 1 urls, producing 120 keys and 395 values 
[2022-06-29T14:03:15,580][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-06-29T14:03:15,662][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2022-06-29T14:03:16,210][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2022-06-29T14:03:16,532][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2022-06-29T14:03:16,549][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.2.3) {:es_version=>8}
[2022-06-29T14:03:16,553][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-06-29T14:03:16,627][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-29T14:03:16,627][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-29T14:03:16,632][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-06-29T14:03:16,652][INFO ][logstash.filters.csv     ][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2022-06-29T14:03:16,694][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-06-29T14:03:16,762][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/logstash-8.2.3/conf/inspec.conf"], :thread=>"#<Thread:0x48e38277 run>"}
[2022-06-29T14:03:18,017][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.25}
[2022-06-29T14:03:18,102][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-29T14:03:18,171][INFO ][filewatch.observingtail  ][main][2c845ee5978dc5ed1bf8d0f617965d2013df9d31461210f0e7c2b799e02f6bb8] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-29T14:03:18,220][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

如有任何建议，不胜感激。
谢谢
达曼德拉·库马尔·辛格

elasticsearch

来源：https://stackoverflow.com/questions/72800973/logstash-not-ingesting-content-into-elasticsearch

2条答案

按热度按时间

kuuvgm7e1#

在filebeat中，ignore_older => 0关闭了基于年龄的过滤，在logstash文件输入中，它告诉过滤器忽略任何超过零秒的文件，由于文件输入在周期性轮询新文件之间休眠，这意味着它忽略所有文件，即使它们正在更新。

赞(0）回复(0）举报 2022-11-02

kpbwa7wx2#

在我的案例中（Windows 10，Logstash 8.1.0），带有反斜杠的文件路径（C：\path\to\csv\etc.CSV）导致了同样的问题，将反斜杠改为正斜杠解决了这个问题。
下面是一个有效的logstash配置：

input {
    file {
        path => "C:/path/to/csv/file.csv"
        type => "file"
        start_position => "beginning"
        sincedb_path => "NUL"
    }
}
filter {
  csv {
    columns => 
        [
          "WID","LID","IID","Product","QTY","TID"
        ]
    separator => ","
  }
  mutate {        
        rename => { 
            "WID" => "w_id"
            "LID" => "l_id"
            "IID" => "i_id"
            "Product" => "product"
            "QTY" => "quantity" 
        }
        convert => {
            "w_id" => "integer"
            "l_id" => "integer"
            "i_id" => "integer"
            "quantity" => "float"
        }
        remove_field => [
            "@timestamp", 
            "@version", 
            "host",
            "message", 
            "type",
            "path", 
            "event",
            "log",
            "TID"
        ]
    }
}

output {
  elasticsearch {
    action => "index"
    hosts => ["https://127.0.0.1:9200"]
    index => "product_inline"
  }
  stdout { }
}

赞(0）回复(0）举报 2022-11-02

我来回答

Logstash未将内容摄取到ElasticSearch中

2条答案

相关问题

热门标签

最新问答