我的目标是我有一个带有avro序列化属性的spring boot应用程序kafka producer,我把推送到相应主题的消息排除在外,该主题应该通过汇合接收器连接器访问并插入mysql/oracle数据库表中,我能够产生avro序列化和spring-boot消费者可以avro反序列化,但我的接收器连接器不工作,我无法捕捉什么样的有效负载接收器连接器是例外,应该如何编写spring-boot生产者推送消息这样一种方式,接收器连接器可以cop起来的属性
提前感谢:)
这是spring boot应用程序中的application.yml
server:
port: 9000
spring.kafka:
bootstrap-servers: "localhost:9092"
properties:
schema.registry.url: "http://localhost:8081"
specific.avro.reader: true
producer:
key-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
value-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
app:
topic: event_pay2
这是从spring启动应用程序创建模式的有效负载
{
"schema": {
"type": "struct",
"fields": [
{
"type": "string",
"optional": false,
"field": "userid"
},
{
"type": "string",
"optional": false,
"field": "regionid"
},
{
"type": "string",
"optional": false,
"field": "gender"
}
],
"optional": false,
"name": "oracle.user"
},
"payload": {
"userid": "User_1",
"regionid": "Region_5",
"gender": "FEMALE"
}
}
pom.xml文件
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.kafka</groupId>
<artifactId>kafka-producer-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-producer-example</name>
<description>Demo project for Spring Boot</description>
<repositories>
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven</url>
</repository>
</repositories>
<properties>
<java.version>1.8</java.version>
<confluent.version>4.0.0</confluent.version>
</properties>
<dependencies>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>${confluent.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.8.2</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
<outputDirectory>${project.build.directory}/generated/avro</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
这是我的休息电话,我是如何把梅斯盖推向Kafka的主题的
@PostMapping("/publish/avrodata")
public String sendMessage(@RequestBody String request) {
sender.send(request);
return "Published successfully";
}
最后是我的Flume接头
"name": "JDBC_Sink_EVENT_PAY",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "event_pay2",
"connection.url": "jdbc:mysql://localhost:3306/user",
"connection.user": "****",
"connection.password": "****",
"auto.create": "true",
"auto.evolve":"true",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://localhost:8081",
"key.converter.schemas.enable": "true",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081",
"value.converter.schemas.enable": "true"
1条答案
按热度按时间nuypyhwy1#
总是,总是在设置连接器之前调试主题。使用
kafka-avro-console-consumer
为了这个。如果不起作用,请连接+AvroConverter
根据我的经验,很可能也行不通,你可以缩小问题空间。如果我正确地阅读了您的代码,那么您发送的是字符串,而不是avro对象
相反,您需要将您的输入请求解析为一个对象,该对象是作为应用程序的一部分创建的
/src/main/resources/avro
模式数据,而不仅仅是作为字符串通过传入请求转发。avsc文件可能看起来像
这将创造一个
oracle.user.User
对象,使你的KafkaTemplate
需要像这样KafkaTemplate<String, oracle.user.User> sender