Kafka 无法使用缩放对象将KEDA连接到AWS MSK

zqdjd7g9  于 11个月前  发布在  Apache
关注(0)|答案(1)|浏览(101)

我已经创建了一个有3个代理的AWS MSK集群。集群的身份验证是使用IAM role-based authenticationSASL/SCRAM authentication。我已经在我的微服务中使用了MSK,并且我在代理中发送和接收事件。现在我想根据Kafka主题的事件计数来扩展我的Kubernetes pod。
我有一个包含多个微服务的EKS集群,这些微服务的名称空间称为default
我已经在一个名为keda的新名称空间中安装了KEDA。我还向KEDA添加了一个TriggerAuthentication,并将其安装在default名称空间中。我使用SASL/SCRAM authentication通过MSK对KEDA进行了身份验证:

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: {{ .Release.Namespace }}
data:
  username: "{{ $.Values.kafka.username | b64enc }}"
  password: "{{ $.Values.kafka.password | b64enc }}"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-conn
  namespace: {{ .Release.Namespace }}
spec:
  secretTargetRef:
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password

字符串
我将ScaledObject添加到了我的微服务的Helm Charts中:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: user-service-scaledobject
spec:
  scaleTargetRef:
    kind: Deployment
    name: {{ $.Chart.Name }}
  pollingInterval: 30 # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  idleReplicaCount: 0 # Optional. Default: 0
  minReplicaCount: 0 # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: kafka
    authenticationRef:
        name: keda-trigger-auth-kafka-conn
    metadata:
        bootstrapServers: {{ .Values.kafka.bootstrapServers }}
        consumerGroup: {{ .Values.kafka.consumerGroup }}
        topic: {{ .Values.kafka.topic }}
        lagThreshold: "500"
        version: 2.8.1
        sasl: scram_sha512


这是微服务的Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ $.Chart.Name }}
  labels:
    app: {{ $.Chart.Name }}
    component: {{ $.Chart.Name }}
    release: {{ $.Release.Name }}
spec:
  replicas: {{ .Values.deployment.replicas }}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      {{- if .Values.deployment.rollingUpdate.enabled }}
      maxSurge: {{ $.Values.deployment.rollingUpdate.maxSurge }}
      maxUnavailable: {{ $.Values.deployment.rollingUpdate.maxUnavailable }}
      {{- end }}
  selector:
    matchLabels:
      app: {{ $.Chart.Name }}
      release: {{ $.Release.Name }}
  template:
    metadata:
      labels:
        app: {{ $.Chart.Name }}
        version: {{ $.Chart.Version }}
        release: {{ $.Release.Name }}
    spec:
      imagePullSecrets:
      - name: {{ $.Values.deployment.image.imagepullsecret }}
      containers:
      - name: {{ $.Chart.Name }}
        image: "{{ .Values.deployment.image.repository }}:{{ .Values.deployment.image.tag }}"
        ports:
        - name: {{ $.Chart.Name }}
          containerPort: {{ $.Values.deployment.ports.http }}
          protocol: {{ $.Values.service.protocol }}
        imagePullPolicy: {{ $.Values.deployment.image.pullPolicy | quote }}
        resources:
          requests:
            memory: "{{ .Values.deployment.resources.requests.memory }}"
            cpu: "{{ .Values.deployment.resources.requests.cpu }}"
          limits:
            memory: "{{ .Values.deployment.resources.limits.memory }}"
            cpu: "{{ .Values.deployment.resources.limits.cpu }}"
        {{- if $.Values.deployment.volumes.enabled }}
        volumeMounts:
        - name: volume
          mountPath: /var/volume/myapp
        {{- end }}
        {{- if $.Values.deployment.env }}
        env:
          {{- range .Values.deployment.env }}
          - name: {{ .name }}
            value: "{{ .value }}"
          {{- end }}
        {{- end }}
      {{- if $.Values.deployment.volumes.enabled }}
      volumes:
      - name: volume
        emptyDir: {}
      {{- end }}
      {{- if $.Values.deployment.nodeSelector }}
      nodeSelector:
        {{- toYaml $.Values.deployment.nodeSelector | nindent 8 }}
      {{- end }}


当我尝试将ScaledObject添加到微服务名称空间(default)时,我在KEDA操作员窗格中收到以下错误:

ERROR   scale_handler   error resolving auth params
ERROR   Error getting scalers
ERROR   Failed to create new HPA resource
Warning  KEDAScalerFailed         13m (x82 over 18h)  keda-operator  error creating kafka client: kafka: client has run out of available brokers to talk to
Warning  ScaledObjectCheckFailed  13m (x82 over 18h)  keda-operator  failed to ensure HPA is correctly created for ScaledObject


有什么问题吗?
谢谢你,谢谢

sh7euo9m

sh7euo9m1#

这可能有点晚,但根据KEDA的文档,实际上有两个Kafka缩放器:

只有Experimental flavor支持AWS MSK IAM身份验证。如果您查看两者的触发器规范,您会注意到type的不同。apache-kafka是新的实验版本。

triggers:
- - type: kafka
+ - type: apache-kafka

字符串
请参阅PR了解更多详情:https://github.com/kedacore/keda/pull/4801
也许这种微妙的差异就是两个版本的定标器之间的差异。

相关问题