内存耗尽问题和Kafka经纪人死亡

性能测试kafka，端到端管线，kafka data producer->kafka->spark streaming->hdfs--stream1 kafka data producer->kafka->flume->hdfs--stream2
stream1Kafka配置：
主题数：10
分区数：所有主题20个
stream2Kafka配置：
主题数：10
分区数：所有主题20个
一些重要的Kafka配置：

"BROKER_MEM": "32768"(32GB)
"BROKER_JAVA_HEAP": "16384"(16GB)
"BROKER_COUNT": "3"
"KAFKA_MESSAGE_MAX_BYTES": "1000012"(1MB)
"KAFKA_REPLICA_FETCH_MAX_BYTES": "1048576"(1MB)
"KAFKA_NUM_PARTITIONS": "20"
"BROKER_DISK_SIZE": "5000" (5GB)
"KAFKA_LOG_SEGMENT_BYTES": "50000000",(50MB)
"KAFKA_LOG_RETENTION_BYTES": "5000000000"(5GB)

数据生产者到Kafka吞吐量：
消息速率：所有3个代理和主题/分区的消息速率约为500000条/秒。消息大小：大约300到400字节。
此配置中观察到的问题：
问题1：
堆栈跟踪：

[2017-11-03 00:56:28,484] FATAL [Replica Manager on Broker 0]: Halting due to unrecoverable I/O error while handling produce request:  (kafka.server.ReplicaManager)
kafka.common.KafkaStorageException: I/O exception in append to log 'store_sales-16'
    at kafka.log.Log.append(Log.scala:349)
    at kafka.cluster.Partition$$anonfun$10.apply(Partition.scala:443)
    at kafka.cluster.Partition$$anonfun$10.apply(Partition.scala:429)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
    at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:240)
    at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:429)
    at kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:407)
    at kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:393)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:393)
    at kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:330)
    at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:425)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:78)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
    at kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:116)
    at kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:106)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
    at kafka.log.AbstractIndex.resize(AbstractIndex.scala:106)
    at kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(AbstractIndex.scala:160)
    at kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:160)
    at kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:160)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
    at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:159)
    at kafka.log.Log.roll(Log.scala:771)
    at kafka.log.Log.maybeRoll(Log.scala:742)
    at kafka.log.Log.append(Log.scala:405)
    ... 22 more
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
    ... 34 more

问题2：
堆栈跟踪：

[2017-11-02 23:55:49,602] FATAL [ReplicaFetcherThread-0-0], Disk error while replicating data for catalog_sales-3 (kafka.server.ReplicaFetcherThread)
kafka.common.KafkaStorageException: I/O exception in append to log 'catalog_sales-3'
    at kafka.log.Log.append(Log.scala:349)
    at kafka.server.ReplicaFelakhstcherThread.processPartitionData(ReplicaFetcherThread.scala:130)
    at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:159)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:141)
    at scala.Option.foreach(Option.scala:257)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:141)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:138)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:138)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:138)
    at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:138)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:136)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
    at kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:116)
    at kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:106)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
    at kafka.log.AbstractIndex.resize(AbstractIndex.scala:106)
    at kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(AbstractIndex.scala:160)
    at kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:160)
    at kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:160)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
    at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:159)
    at kafka.log.Log.roll(Log.scala:771)
    at kafka.log.Log.maybeRoll(Log.scala:742)
    at kafka.log.Log.append(Log.scala:405)
    ... 16 more
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
    ... 28 more

这两个异常在所有3个具有相同kafka配置的代理中连续发生。

内存耗尽问题和Kafka经纪人死亡

暂无答案！

相关问题

热门标签

最新问答