kafka配置

ekqde3dh  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(383)

我正在现有的cdh5.5.2集群上安装kafka-2.0,下面是我所遵循的步骤
从cm添加服务
选定的Kafka(在此之前,我下载并分发并激活了所有节点上的Kafka包裹)
为kafkabroker选择1个节点,为kafka mirrormaker选择4个节点
然后,我用一个mirror maker节点更新了目标代理列表(bootstrap.servers)属性,并用同一个节点更新了源代理列表(source.bootstrap.servers)
下面是我得到的错误(日志文件)

Fatal error during KafkaServerStartable startup. Prepare to shutdown 
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:43)
    at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:186)
    at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
    at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.Range.foreach(Range.scala:166)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at kafka.log.LogCleaner.<init>(LogCleaner.scala:83)
    at kafka.log.LogManager.<init>(LogManager.scala:64)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:601)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
    at kafka.Kafka$.main(Kafka.scala:67)
    at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
    at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
wqsoz72f

wqsoz72f1#

堆栈跟踪显示“java.lang.outofmemoryerror:java heap space”——vm heap的空间不足。通过设置

export KAFKA_HEAP_OPTS="-Xmx1G -Xms512M"

在/bin/kafka-server-start.sh中。

wr98u20j

wr98u20j2#

你需要增加 broker_max_heap_size 值至少为1gb,并从cloudera manager重新启动kafka服务。如果您仍然面临相同的问题,请尝试根据您的群集配置增加

相关问题