kubernetes fluentd with OpenSearch -@timestamp字段从何而来？

pbwdgjma 于 2023-06-21 发布在 Kubernetes

关注(0)|答案(1)|浏览(186)

我在Kubernetes集群中作为DaemonSet运行fluentd。fluentd将日志条目写入OpenSearch。查看https://github.com/fluent/fluentd-kubernetes-daemonset
在我的问题之前，我必须提供一些背景：它的工作方式是Kubernetes pod写入stdout，容器运行时将其写入某个位置，即/var/logs/pods/<pod_specific_location>。这些日志文件的格式如下：

31-12-23T12:00:00.123456Z    stdout    F     my great log message

现在，fluentd被配置为从那里选择它，并使用cri parser plugin将其转换为：

{
"time": "31-12-23T12:00:00.123456Z",
"stream": "stdout",
"logtag": "F",
"message": "my great log message"
}

现在，假设我在集群中运行一个pod，它写入以下日志消息：

hello

接下来，使用kubernetes metadata plugin，fluentd用kubernetes元数据丰富了这个记录，比如命名空间名称、pod名称等。等等，所以它看起来像这样：

{
"stream":"stdout",
"logtag":"F",
"time":"31-12-23T12:00:00.123456Z",
"message": "my great log message"
"docker":
{"container_id":"9077644273956d3f3e9d171240f412b3b6e959984a5fd99adfcb77f9b998a370"},
"kubernetes":
{"container_name":"demo-app",
"namespace_name":"foo",
"pod_name":"foo-ns-app",
"container_image":"docker.io/yoavklein3/net-tools:latest",
"container_image_id":"docker.io/yoavklein3/net-tools@sha256:3fd9646a14d97ecc2d236a5bebd88faf617bc6045f1e4f32c49409f1c930879a",
"pod_id":"a69fb942-c0ab-457d-b752-ffa3fa27e574",
"pod_ip":"10.0.2.224",
"host":"ip-10-0-2-5.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"6bdf5fe9-9a5a-4501-ab6c-deddd241e071",
"namespace_labels":{"kubernetes.io/metadata.name":"foo"}}}

现在，使用opensearch plugin将其发送到Opensearch。
现在，当我打开Opensearch Dashboards时，我可以看到一个名为@timestamp的字段，我只是不知道这个字段来自哪里：
这是OpenSearch中的一个文档（抱歉没有完全坚持上面的示例，但概念保持不变）：

{
  "_index": "logstash-2023.06.06",
  "_type": "_doc",
  "_id": "sVHjj4gByMQm1Wc45hv2",
  "_version": 1,
  "_score": null,
  "_source": {
    "stream": "stdout",
    "logtag": "F",
    "time": "2023-06-06T08:47:35.874884092Z",
    "docker": {
      "container_id": "9077644273956d3f3e9d171240f412b3b6e959984a5fd99adfcb77f9b998a370"
    },
    "kubernetes": {
      "container_name": "demo-app",
      "namespace_name": "foo",
      "pod_name": "foo-ns-app",
      "container_image": "docker.io/yoavklein3/net-tools:latest",
      "container_image_id": "docker.io/yoavklein3/net-tools@sha256:3fd9646a14d97ecc2d236a5bebd88faf617bc6045f1e4f32c49409f1c930879a",
      "pod_id": "a69fb942-c0ab-457d-b752-ffa3fa27e574",
      "pod_ip": "10.0.2.224",
      "host": "ip-10-0-2-5.ec2.internal",
      "master_url": "https://172.20.0.1:443/api",
      "namespace_id": "6bdf5fe9-9a5a-4501-ab6c-deddd241e071",
      "namespace_labels": {
        "kubernetes.io/metadata.name": "foo"
      }
    },
    "data": "This is from FOO namespace",
    "@timestamp": "2023-06-06T08:47:35.882677347+00:00",
    "tag": "kubernetes.var.log.containers.foo-ns-app_foo_demo-app-9077644273956d3f3e9d171240f412b3b6e959984a5fd99adfcb77f9b998a370.log"
  },
  "fields": {
    "@timestamp": [
      "2023-06-06T08:47:35.882Z"
    ],
    "time": [
      "2023-06-06T08:47:35.874Z"
    ]
  },
  "sort": [
    1686041255882
  ]
}

注意：缺少message字段，而是有data字段。这是由于将message字段解析为JSON。你可以忽略这一点，它是完全不相关的，只是注意到，如果你感到困惑。