flume采集kafka数据到hdfs报如下错误

ゞ浴缸里的玫瑰 2022-01-25 23:25 318阅读 0赞

运行Flume没多久就报下面的异常：

2016-08-24 17:35:58,927 (Flume Thrift IPC Thread 8) \[ERROR - org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:196)\] Error while writing to required channel: org.apache.flume.channel.MemoryChannel\{name: memoryChannel\}  
2016-08-24 17:35:59,332 (SinkRunner-PollingRunner-DefaultSinkProcessor) \[ERROR - kafka.utils.Logging$class.error(Logging.scala:97)\] Failed to collate messages by topic, partition due to: GC overhead limit exceeded  
2016-08-24 17:35:59,332 (Flume Thrift IPC Thread 8) \[ERROR - org.apache.thrift.ProcessFunction.process(ProcessFunction.java:41)\] Internal error processing appendBatch  
java.lang.OutOfMemoryError: GC overhead limit exceeded  
    at java.util.Arrays.copyOf(Arrays.java:3332)  
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)  
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)  
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)  
    at java.lang.StringBuffer.append(StringBuffer.java:272)  
    at java.io.StringWriter.write(StringWriter.java:112)  
    at java.io.PrintWriter.write(PrintWriter.java:456)  
    at java.io.PrintWriter.write(PrintWriter.java:473)  
    at java.io.PrintWriter.print(PrintWriter.java:603)  
    at java.io.PrintWriter.println(PrintWriter.java:756)  
    at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)  
    at java.lang.Throwable.printStackTrace(Throwable.java:658)  
    at java.lang.Throwable.printStackTrace(Throwable.java:721)  
    at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)  
    at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)  
    at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)  
    at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)  
    at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)  
    at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)  
    at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)  
    at org.apache.log4j.Category.callAppenders(Category.java:206)  
    at org.apache.log4j.Category.forcedLog(Category.java:391)  
    at org.apache.log4j.Category.log(Category.java:856)  
    at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)  
    at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:196)  
    at org.apache.flume.source.ThriftSource$ThriftSourceHandler.appendBatch(ThriftSource.java:457)  
    at org.apache.flume.thrift.ThriftSourceProtocol$Processor$appendBatch.getResult(ThriftSourceProtocol.java:259)  
    at org.apache.flume.thrift.ThriftSourceProtocol$Processor$appendBatch.getResult(ThriftSourceProtocol.java:247)  
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)  
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)  
    at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)  
    at org.apache.thrift.server.Invocation.run(Invocation.java:18)

显然是我采集的数据太大了，导致Flume的JVM内存不够用。

用 ps -aux | grep flume查找Flume进程也能看到Flume使用了多少内存。

解决方法：

1、 vim bin/flume-ng

在里面找到JAVA\_OPTS="-Xmx20m"，它默认启动时最大可用内存为20，只要将其调大一点就可以了。

2、 或者在Flume conf目录下找到flume-env.sh.template文件

cp flume-env.sh.template flume-env.sh

vim flume-env.sh

把下面这句配置的注释删掉就可了

\# export JAVA\_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

需要cd /etc/flume-ng/conf

cp flume-env.sh.template flume-env.sh