Log directory offline on AWS EBS

Ops
1

Hello,

We are encountering such issues on AWS EBS based system, nothing in AWS logs.

Is it memory or IO issue, as it states in one line

in dir /srv/kafka/disk1 due to IOException (kafka.server.LogDirFailureChannel)
java.io.IOException: Map failed

and in second

Caused by: java.lang.OutOfMemoryError: Map failed

[2022-02-07 21:35:43,046] WARN [ReplicaManager broker=77] Unable to describe replica dirs for /srv/kafka/disk1 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Log directory /srv/kafka/disk1 is offline


[2022-02-07 21:34:54,355] ERROR Failed to clean up log for invent_points_of_interest-39 in dir /srv/kafka/disk1 due to IOException (kafka.server.LogDirFailureChannel)
java.io.IOException: Map failed
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938)
	at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:124)
	at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:54)
	at kafka.log.LazyIndex$.$anonfun$forOffset$1(LazyIndex.scala:106)
	at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex.scala:63)
	at kafka.log.LazyIndex.get(LazyIndex.scala:60)
	at kafka.log.LogSegment.offsetIndex(LogSegment.scala:64)
	at kafka.log.Cleaner.buildOffsetMapForSegment(LogCleaner.scala:932)
	at kafka.log.Cleaner.$anonfun$buildOffsetMap$5(LogCleaner.scala:908)
	at kafka.log.Cleaner.$anonfun$buildOffsetMap$5$adapted(LogCleaner.scala:904)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
	at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
	at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:904)
	at kafka.log.Cleaner.doClean(LogCleaner.scala:523)
	at kafka.log.Cleaner.clean(LogCleaner.scala:511)
	at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:380)
	at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:352)
	at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332)
	at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:321)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Caused by: java.lang.OutOfMemoryError: Map failed
	at sun.nio.ch.FileChannelImpl.map0(Native Method)
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935)
	... 21 more
[2022-02-07 21:34:54,356] WARN [ReplicaManager broker=77] Stopping serving replicas in dir /srv/kafka/disk1 (kafka.server.ReplicaManager)
-02-07 21:34:54,588] WARN [ReplicaManager broker=77] While recording the replica LEO, the partition multibooks-250 hasn't been created. (kafka.server.ReplicaManager)
[2022-02-07 21:34:54,696] WARN Stopping serving logs in dir /srv/kafka/disk1 (kafka.log.LogManager)