kafka关于ERROR Uncaught exception in scheduled task ‘kafka-log-retention‘问题解决

骑猪看日落 2022-09-10 10:28 224阅读 0赞

\[2021-09-07 10:27:07,763\] WARN \[ReplicaManager broker=0\] Stopping serving replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)  
\[2021-09-07 10:27:07,764\] ERROR Uncaught exception in scheduled task ‘kafka-log-retention’ (kafka.utils.KafkaScheduler)  
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for rescue-0 in dir /tmp/kafka-logs  
Caused by: java.nio.file.NoSuchFileException: /tmp/kafka-logs/rescue-0/00000000000000000000.log  
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)  
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)  
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)  
at java.nio.file.Files.move(Files.java:1395)  
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:904)  
at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:248)  
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:494)  
at kafka.log.Log. a n o n f u n anonfun anonfundeleteSegmentFiles 1 ( L o g . s c a l a : 2339 ) a t k a f k a . l o g . L o g . 1(Log.scala:2339) at kafka.log.Log. 1(Log.scala:2339)atkafka.log.Log.anonfun$deleteSegmentFiles 1 1 1adapted(Log.scala:2339)  
at scala.collection.immutable.List.foreach(List.scala:333)  
at kafka.log.Log.deleteSegmentFiles(Log.scala:2339)  
at kafka.log.Log.removeAndDeleteSegments(Log.scala:2323)  
at kafka.log.Log. a n o n f u n anonfun anonfundeleteSegments 2 ( L o g . s c a l a : 1834 ) a t k a f k a . l o g . L o g . d e l e t e S e g m e n t s ( L o g . s c a l a : 2453 ) a t k a f k a . l o g . L o g . d e l e t e R e t e n t i o n M s B r e a c h e d S e g m e n t s ( L o g . s c a l a : 1818 ) a t k a f k a . l o g . L o g . d e l e t e O l d S e g m e n t s ( L o g . s c a l a : 1887 ) a t k a f k a . l o g . L o g M a n a g e r . 2(Log.scala:1834) at kafka.log.Log.deleteSegments(Log.scala:2453) at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1818) at kafka.log.Log.deleteOldSegments(Log.scala:1887) at kafka.log.LogManager. 2(Log.scala:1834)atkafka.log.Log.deleteSegments(Log.scala:2453)atkafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1818)atkafka.log.Log.deleteOldSegments(Log.scala:1887)atkafka.log.LogManager.anonfun$cleanupLogs 3 ( L o g M a n a g e r . s c a l a : 1092 ) a t k a f k a . l o g . L o g M a n a g e r . 3(LogManager.scala:1092) at kafka.log.LogManager. 3(LogManager.scala:1092)atkafka.log.LogManager.anonfun$cleanupLogs 3 3 3adapted(LogManager.scala:1089)  
at scala.collection.immutable.List.foreach(List.scala:333)  
at kafka.log.LogManager.cleanupLogs(LogManager.scala:1089)  
at kafka.log.LogManager. a n o n f u n anonfun anonfunstartupWithConfigOverrides 2 ( L o g M a n a g e r . s c a l a : 429 ) a t k a f k a . u t i l s . K a f k a S c h e d u l e r . 2(LogManager.scala:429) at kafka.utils.KafkaScheduler. 2(LogManager.scala:429)atkafka.utils.KafkaScheduler.anonfun$schedule 2 ( K a f k a S c h e d u l e r . s c a l a : 114 ) a t j a v a . u t i l . c o n c u r r e n t . E x e c u t o r s 2(KafkaScheduler.scala:114) at java.util.concurrent.Executors 2(KafkaScheduler.scala:114)atjava.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:511)  
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access 301 ( S c h e d u l e d T h r e a d P o o l E x e c u t o r . j a v a : 180 ) a t j a v a . u t i l . c o n c u r r e n t . S c h e d u l e d T h r e a d P o o l E x e c u t o r 301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor 301(ScheduledThreadPoolExecutor.java:180)atjava.util.concurrent.ScheduledThreadPoolExecutorScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
at java.lang.Thread.run(Thread.java:748)  
Suppressed: java.nio.file.NoSuchFileException: /tmp/kafka-logs/rescue-0/00000000000000000000.log -> /tmp/kafka-logs/rescue-0/00000000000000000000.log.deleted  
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)  
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)  
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)  
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)  
at java.nio.file.Files.move(Files.java:1395)  
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:901)  
… 24 more

分析：我设置的log目录地址是tmp下，到那时liunx系统会定期清除该目录下的文件，导致kafka找不到数据。  
解决办法：更改kafka配置中的logs目录