elasticsearch 为Amazon OpenSearch设置Serilog

jgzswidk  于 2022-11-22  发布在  ElasticSearch
关注(0)|答案(2)|浏览(128)

我正在尝试设置Serilog,以便将日志从ASP.NET Core WebAPI发送到Amazon OpenSearch的本地示例。我在控制台上看到日志,但OpenSearch中没有显示任何内容。
已安装的第三方库:

  • 串行接口AspNetCore(6.0.0-开发-00265)
  • 系列富集器环境(2.2.1-dev-00787)
  • 系列日志.汇.ElasticSearch(9.0.0-beta7)

OpenSearch通过Development Docker Compose运行(无安全插件):
https://opensearch.org/docs/2.0/opensearch/install/docker/#sample-docker-compose-file-for-development
Program.cs

var logger = new LoggerConfiguration()
      .WriteTo.Console()
      .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
      {
          AutoRegisterTemplate = true,
          MinimumLogEventLevel = LogEventLevel.Information,  
          FailureCallback = FailureCallback,
          EmitEventFailure = EmitEventFailureHandling.RaiseCallback | EmitEventFailureHandling.ThrowException
      })
      .CreateLogger();

builder.Logging.ClearProviders();
builder.Logging.AddSerilog(logger);

控制器类别:

_logger.LogWarning("Example warning");
_logger.LogError("Example error");

FailureCallback为空OpenSearch控制台未显示任何问题
可能出了什么问题?

lb3vh1jj

lb3vh1jj1#

我已经尝试了您的设置,下面是一些结果(注意仅使用稳定版本的软件):

  • .NET核心v6.0(非测试版)
  • Serilog.Flume.ElasticSearchv 8.4.1
  • 串行接口AspNetCore 5.0.0

使用的停靠合成:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:2.0.1
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.0.1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

Program.cs

using Serilog;
using Serilog.Events;
using Serilog.Sinks.Elasticsearch;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddRazorPages();

builder.Logging.ClearProviders();

Serilog.Debugging.SelfLog.Enable(msg => Console.WriteLine(msg));

ServicePointManager.ServerCertificateValidationCallback =
    (source, certificate, chain, sslPolicyErrors) => true;

var logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        AutoRegisterTemplate = true,
        MinimumLogEventLevel = LogEventLevel.Information,
        FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
        EmitEventFailure = EmitEventFailureHandling.RaiseCallback | EmitEventFailureHandling.ThrowException,
        TypeName =  "_doc",
        InlineFields = false

    })
    .CreateLogger();

builder.Logging.ClearProviders();
builder.Logging.AddSerilog(logger);

    

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

有几件事应该有助于故障排除

  1. Serilog.Debugging.SelfLog.Enable(msg => Console.WriteLine(msg));-将显示来自Serilog的真实的错误(很可能会看到SSL)
  2. ServicePointManager.ServerCertificateValidationCallback = (source, certificate, chain, sslPolicyErrors) => true;-暂时停止任何SSL问题(您可以稍后修复它们)
    1.我发现的下一件事是Serilog生成的[_type]字段的问题,它不被Elastic〉v8.2接受,这很可能是因为您的缓冲区保留了旧记录。
    1.虽然Serilog的最新测试版采用了TypeName="_doc”,但AWS Opensearch 2.0.1在“compatibility.override_main_response_version=true”设置(请参阅此处的详细信息)https://github.com/opensearch-project/OpenSearch/pull/3530方面存在另一个漏洞-基本上,我建议将AWS OpenSearch回滚到v2。
    之后,希望它应该工作:)
n1bvdmb6

n1bvdmb62#

对我来说,问题是我没有提供IndexFormat属性的值(在ElasticSearchSinkOptions对象中)。我把它放在端点中,就像通过REST插入数据时应该做的那样。总而言之,下面的代码为我解决了这个问题:

var jsonFormatter = new CompactJsonFormatter();

var loggerConfig = new LoggerConfiguration()
  .Enrich.FromLogContext()
  .WriteTo.Map("Name", "**error**", (name, writeTo) =>
  {
    var currentYear = DateTime.Today.Year;
    var currentWeek = calendar.GetWeekOfYear(DateTime.Now, 
                                CalendarWeekRule.FirstDay, 
                                DayOfWeek.Monday);
    writeTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("<opensearch endpoint>"))
    {
      CustomFormatter = jsonFormatter,
      TypeName = "_doc",
      IndexFormat = $"my-index-{currentYear}-{currentWeek}",
      MinimumLogEventLevel = LogEventLevel.Information,
      EmitEventFailure = EmitEventFailureHandling.RaiseCallback | 
                          EmitEventFailureHandling.ThrowException,
      FailureCallback = e =>
        Console.WriteLine(
          "An error occured in Serilog ElasticSearch sink: " +
          $"{e.Exception.Message} | {e.Exception.InnerException?.Message}")
    });
  });
Log.Logger = loggerConfig.CreateLogger();

当然,您还需要正确地设置OpenSearch,以便它可以自动将策略应用到您的索引等。

相关问题