kubernetes 如何调试PodMonitor生成的规则?

nwlqm0z1  于 2023-08-03  发布在  Kubernetes
关注(0)|答案(1)|浏览(165)

我有一个terraform1#prometheus-operator仓库,以前用普通的prometheus(没有operator)让它工作,现在测试prometheus-operator,所以如果pod水平扩展,它仍然可以正确地抓取所有pod的所有指标,而不仅仅是从一个pod。
这个article一步一步的如何运行minikube和terraform。
展开后,显示正确:

k get pods -n pf1ns -w                                                                    1 ↵
NAME                                               READY   STATUS    RESTARTS   AGE
keda-admission-webhooks-76cd6c4b59-6b59r           1/1     Running   0          20h
keda-operator-5bb494667b-bb2bl                     1/1     Running   0          20h
keda-operator-metrics-apiserver-68d9f78869-d65vj   1/1     Running   0          20h
prometheus-pf1prom-0                               2/2     Running   0          3m9s
promfiberdeploy-868697d555-2jfgv                   1/1     Running   0          20h

字符串
prometheus-pf1prom-0上没有错误:

ts=2023-07-04T17:37:42.315Z caller=main.go:1257 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=554.419µs db_storage=1.312µs remote_storage=1.232µs web_handler=410ns query_engine=851ns scrape=36.078µs scrape_sd=187.441µs notify=681ns notify_sd=1.453µs rules=922ns tracing=2.856µs


PodMonitor生成的规则中缺少什么配置?

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: pf1ns/pf1prom
    prometheus_replica: prometheus-pf1prom-0
scrape_configs:
- job_name: podMonitor/pf1ns/pf1prompodmonitor/0
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_labelpresent_app]
    separator: ;
    regex: (promfiber);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: "3000"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: pf1ns/pf1prompodmonitor
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: "3000"
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: pod
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - pf1ns


规则:

resource "kubernetes_manifest" "pf1prompodmonitor" {
  manifest = {
    "apiVersion" = "monitoring.coreos.com/v1"
    "kind"       = "PodMonitor"
    "metadata"   = {
      "name"      = "pf1prompodmonitor"
      "namespace" = var.nsname
      "labels"    = {
        "name" = "pf1podmonitor"
      }
    }
    "spec" = {
      "selector" = {
        "matchLabels" = {
          "app" = kubernetes_deployment_v1.promfiberdeploy.spec.0.selector.0.match_labels.app # -> promfiber
        }
      }
      "namespaceSelector" = {
        "matchNames" = [
          var.nsname
        ]
      }
      "podMetricsEndpoints" = [
        {
          "interval" = "5s"
          "port"     = kubernetes_deployment_v1.promfiberdeploy.spec.0.template.0.spec.0.container.0.port.0.container_port # -> 3000
        }
      ]
    }
  }
}


需要刮擦的pod展开:

resource "kubernetes_deployment_v1" "promfiberdeploy" {
  metadata {
    name      = "promfiberdeploy"
    namespace = var.nsname
  }
  spec {
    selector {
      match_labels = {
        app = "promfiber"
      }
    }
    replicas = "1"
    template {
      metadata {
        labels = {
          app = "promfiber"
        }
        annotations = {
          "prometheus.io/path"   = "/metrics"
          "prometheus.io/scrape" = "true"
          "prometheus.io/port"   = 3000
        }
      }
      spec {
        container {
          name  = "pf1"
          image = "kokizzu/pf1:v0001" # from promfiber.go
          port {
            container_port = 3000
          }
        }
      }
    }
  }
}


在prometheus上没有任何错误,但它也不会抓取任何东西,GUI上的目标是空的。
服务发现存在,但所有服务都处于“已删除”状态
x1c 0d1x的数据
有一个匹配“promfiber”,但状态也被丢弃


__address__="10.244.0.45:3000"
__meta_kubernetes_namespace="pf1ns"
__meta_kubernetes_pod_annotation_prometheus_io_path="/metrics"
__meta_kubernetes_pod_annotation_prometheus_io_port="3000"
__meta_kubernetes_pod_annotation_prometheus_io_scrape="true"
__meta_kubernetes_pod_annotationpresent_prometheus_io_path="true"
__meta_kubernetes_pod_annotationpresent_prometheus_io_port="true"
__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape="true"
__meta_kubernetes_pod_container_id="docker://abbc5c17e873b730fdc6e490917b72ccb4ec8a45f4a250d6fb5f8be4ce95b59b"
__meta_kubernetes_pod_container_image="kokizzu/pf1:v0001"
__meta_kubernetes_pod_container_init="false"
__meta_kubernetes_pod_container_name="pf1"
__meta_kubernetes_pod_container_port_number="3000"
__meta_kubernetes_pod_container_port_protocol="TCP"
__meta_kubernetes_pod_controller_kind="ReplicaSet"
__meta_kubernetes_pod_controller_name="promfiberdeploy-868697d555"
__meta_kubernetes_pod_host_ip="240.1.0.2"
__meta_kubernetes_pod_ip="10.244.0.45"
__meta_kubernetes_pod_label_app="promfiber"
__meta_kubernetes_pod_label_pod_template_hash="868697d555"
__meta_kubernetes_pod_labelpresent_app="true"
__meta_kubernetes_pod_labelpresent_pod_template_hash="true"
__meta_kubernetes_pod_name="promfiberdeploy-868697d555-2jfgv"
__meta_kubernetes_pod_node_name="minikube"
__meta_kubernetes_pod_phase="Running"
__meta_kubernetes_pod_ready="true"
__meta_kubernetes_pod_uid="ad529d61-67f0-4867-b2a6-5c9915870b20"
__metrics_path__="/metrics"
__scheme__="http"
__scrape_interval__="5s"
__scrape_timeout__="5s"
job="podMonitor/pf1ns/pf1prompodmonitor/0"

laximzn5

laximzn51#

"port" = 3000引起

"podMetricsEndpoints" = [
   {
     "interval" = "5s"
     #"port"     = kubernetes_deployment_v1.promfiberdeploy.spec.0.template.0.spec.0.container.0.port.0.container_port
   }
]

字符串
添加该行会导致应用过滤器__meta_kubernetes_pod_container_port_name regex应该匹配字符串"3000"
删除这条线导致没有任何东西被丢弃。
还有一个问题,你不能匹配包含点.或斜线/的标签,它永远不会匹配。

相关问题