我正在使用confluent 3.3.0。我的意图是使用kafka connect将kafka主题中的值插入到oracle表中。我的连接器与我使用avro控制台生产商制作的avro唱片配合良好,如下所示:
./kafka-avro-console-producer --broker-list 192.168.0.1:9092 --topic topic6 --property value.schema='{"type":"record","name":"flights3","fields":[{"name":"flight_id","type":"string"},{"name":"flight_to", "type": "string"}, {"name":"flight_from", "type": "string"}]}'
我插入如下值:
{"flight_id":"1","flight_to":"QWE","flight_from":"RTY"}
我试图实现的是使用java应用程序插入相同的数据,使用对象。以下是我的生产商代码:
public class Sender {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.0.1:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "serializers.custom.FlightSerializer");
props.put("schema.registry.url", "http://192.168.0.1:8081");
Producer<String, Flight> producer = new KafkaProducer<String, Flight>(props);
Flight myflight = new Flight("testflight1","QWE","RTY");
ProducerRecord<String, Flight> record = new ProducerRecord<String, Flight>("topic5","key",myflight);
try {
producer.send(record).get();
} catch (Exception e) {
e.printStackTrace();
}
}
}
以下是航班vo:
package vo;
public class Flight {
String flight_id,flight_to,flight_from;
public Flight(String flight_id, String flight_to, String flight_from) {
this.flight_id = flight_id;
this.flight_to = flight_to;
this.flight_from = flight_from;
}
public Flight(){
}
public String getFlight_id() {
return flight_id;
}
public void setFlight_id(String flight_id) {
this.flight_id = flight_id;
}
public String getFlight_to() {
return flight_to;
}
public void setFlight_to(String flight_to) {
this.flight_to = flight_to;
}
public String getFlight_from() {
return flight_from;
}
public void setFlight_from(String flight_from) {
this.flight_from = flight_from;
}
}
最后,序列化程序:
package serializers.custom;
import java.util.Map;
import org.apache.kafka.common.serialization.Serializer;
import vo.Flight;
import com.fasterxml.jackson.databind.ObjectMapper;
public class FlightSerializer implements Serializer<Flight> {
@Override
public void close() {
}
@Override
public void configure(Map<String, ?> arg0, boolean arg1) {
}
@Override
public byte[] serialize(String arg0, Flight arg1) {
byte[] retVal = null;
ObjectMapper objectMapper = new ObjectMapper();
try {
retVal = objectMapper.writeValueAsString(arg1).getBytes();
} catch (Exception e) {
e.printStackTrace();
}
return retVal;
}
}
但我所理解的是,需要定义一些类似schema的东西,并使用一些avro序列化程序来获取确切的数据,就像我使用avro控制台console consumer那样。我看过一些示例代码,但没有一个对我有用。
编辑
我尝试了以下代码。但没有什么是来在avro控制台消费者。
package producer.serialized.avro;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import vo.Flight;
import java.util.Properties;
public class Sender {
public static void main(String[] args) {
String flightSchema = "{\"type\":\"record\"," + "\"name\":\"flights\","
+ "\"fields\":[{\"name\":\"flight_id\",\"type\":\"string\"},{\"name\":\"flight_to\",\"type\":\"string\"},{\"name\":\"flight_from\",\"type\":\"string\"}]}";
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.0.1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://192.168.0.1:8081");
KafkaProducer producer = new KafkaProducer(props);
Schema.Parser parser = new Schema.Parser();
Schema schema = parser.parse(flightSchema);
GenericRecord avroRecord = new GenericData.Record(schema);
avroRecord.put("flight_id", "1");
avroRecord.put("flight_to", "QWE");
avroRecord.put("flight_from", "RTY");
ProducerRecord<String, GenericRecord> record = new ProducerRecord<>("topic6", avroRecord);
try {
producer.send(record);
} catch (Exception e) {
e.printStackTrace();
}
}
}
2条答案
按热度按时间o2g1uqev1#
架构没有定义,所以
KafkaAvroSerializer
必须联系架构注册表才能提交该架构—它将不具有该架构。你必须为你的对象创建一个模式
Flight
下面的file.avdl(avro扩展名文件之一)示例可以:见avro idl文件
在编译时,当您使用
avro-maven-plugin
,上面的avro模式将生成您的javaFlight
类,因此必须删除以前创建的。当涉及到主类时,必须设置如下两个属性:
和你的制作人,你可以使用你生成的,特定的avro类
希望有帮助:-)
e4yzc0pl2#
准确的数据就像我使用avro控制台一样
你可以看看它的源代码
假设你想使用通用记录,这一切都是正确的,
但你错过了一个电话
producer.flush()
以及producer.close()
最后实际发送该批记录