ELK学习9_ELK数据流传输过程_问题总结2

男娘i 2023-10-17 13:55 37阅读 0赞

目的：使用[Java][]程序把一个.csv文件发送给Kafka，再发送给ELK

实验步骤：见以前的博客

遇到的问题：

一、

直接发送比较大的.csv文件，ELK接到的汉字内容是乱码的

发送比较小的.csv文件，ELK接到的汉字内容正常

猜测：是否是Kafka出错

二、

把大的.csv文件拆分成多个小.csv文件发送，发送过程中，会不定期报错

报错内容如下

**\[html\]** [view plain][] [copy][view plain]

1.  \[2015-12-13 16:02:14,273\] ERROR fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.utils.Utils$:106) 
2.  kafka.common.KafkaException: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed 
3.   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72) 
4.   at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) 
5.   at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67) 
6.   at kafka.utils.Utils$.swallow(Utils.scala:172) 
7.   at kafka.utils.Logging$class.swallowError(Logging.scala:106) 
8.   at kafka.utils.Utils$.swallowError(Utils.scala:45) 
9.   at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67) 
10.  at kafka.producer.Producer.send(Producer.scala:77) 
11.  at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
12.  at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33) 
13.  at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73) 
14.  at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17) 
15. Caused by: java.nio.channels.ClosedChannelException 
16.  at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) 
17.  at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) 
18.  at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) 
19.  at kafka.producer.SyncProducer.send(SyncProducer.scala:113) 
20.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) 
21.  ... 11 more 
22. \[2015-12-13 16:02:14,283\] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.producer.async.DefaultEventHandler:97) 
23. \[2015-12-13 16:02:14,394\] ERROR fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.utils.Utils$:106) 
24. kafka.common.KafkaException: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed 
25.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72) 
26.  at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) 
27.  at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78) 
28.  at kafka.utils.Utils$.swallow(Utils.scala:172) 
29.  at kafka.utils.Logging$class.swallowError(Logging.scala:106) 
30.  at kafka.utils.Utils$.swallowError(Utils.scala:45) 
31.  at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78) 
32.  at kafka.producer.Producer.send(Producer.scala:77) 
33.  at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
34.  at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33) 
35.  at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73) 
36.  at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17) 
37. Caused by: java.nio.channels.ClosedChannelException 
38.  at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) 
39.  at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) 
40.  at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) 
41.  at kafka.producer.SyncProducer.send(SyncProducer.scala:113) 
42.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) 
43.  ... 11 more 
44. \[2015-12-13 16:02:14,404\] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.producer.async.DefaultEventHandler:97) 
45. \[2015-12-13 16:02:14,514\] ERROR fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.utils.Utils$:106) 
46. kafka.common.KafkaException: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed 
47.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72) 
48.  at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) 
49.  at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78) 
50.  at kafka.utils.Utils$.swallow(Utils.scala:172) 
51.  at kafka.utils.Logging$class.swallowError(Logging.scala:106) 
52.  at kafka.utils.Utils$.swallowError(Utils.scala:45) 
53.  at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78) 
54.  at kafka.producer.Producer.send(Producer.scala:77) 
55.  at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
56.  at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33) 
57.  at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73) 
58.  at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17) 
59. Caused by: java.nio.channels.ClosedChannelException 
60.  at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) 
61.  at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) 
62.  at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) 
63.  at kafka.producer.SyncProducer.send(SyncProducer.scala:113) 
64.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) 
65.  ... 11 more 
66. \[2015-12-13 16:02:14,524\] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.producer.async.DefaultEventHandler:97) 
67. \[2015-12-13 16:02:14,644\] ERROR fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.utils.Utils$:106) 
68. kafka.common.KafkaException: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed 
69.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72) 
70.  at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) 
71.  at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78) 
72.  at kafka.utils.Utils$.swallow(Utils.scala:172) 
73.  at kafka.utils.Logging$class.swallowError(Logging.scala:106) 
74.  at kafka.utils.Utils$.swallowError(Utils.scala:45) 
75.  at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78) 
76.  at kafka.producer.Producer.send(Producer.scala:77) 
77.  at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
78.  at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33) 
79.  at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73) 
80.  at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17) 
81. Caused by: java.nio.channels.ClosedChannelException 
82.  at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) 
83.  at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) 
84.  at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) 
85.  at kafka.producer.SyncProducer.send(SyncProducer.scala:113) 
86.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) 
87.  ... 11 more 
88. \[2015-12-13 16:02:14,654\] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.producer.async.DefaultEventHandler:97) 
89. \[2015-12-13 16:02:14,764\] ERROR fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed (kafka.utils.Utils$:106) 
90. kafka.common.KafkaException: fetching topic metadata for topics \[Set(MyPattern2)\] from broker \[ArrayBuffer(id:0,host:10.10.16.14,port:9092)\] failed 
91.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72) 
92.  at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) 
93.  at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78) 
94.  at kafka.utils.Utils$.swallow(Utils.scala:172) 
95.  at kafka.utils.Logging$class.swallowError(Logging.scala:106) 
96.  at kafka.utils.Utils$.swallowError(Utils.scala:45) 
97.  at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78) 
98.  at kafka.producer.Producer.send(Producer.scala:77) 
99.  at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
100.  at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33) 
101.  at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73) 
102.  at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17) 
103. Caused by: java.nio.channels.ClosedChannelException 
104.  at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) 
105.  at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) 
106.  at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) 
107.  at kafka.producer.SyncProducer.send(SyncProducer.scala:113) 
108.  at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) 
109.  ... 11 more 
110. \[2015-12-13 16:02:14,764\] ERROR Failed to send requests for topics MyPattern2 with correlation ids in \[0,8\] (kafka.producer.async.DefaultEventHandler:97) 
111. kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.error 
112. 
113.  at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) 
114.  at kafka.producer.Producer.send(Producer.scala:77) 
115.  at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
116.  at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33) 
117.  at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73) 
118.  at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)

没能找到出错原因

三、使用服务器本地安装的Eclipse执行程序，中间不会报错，但是，接收到的中文内容还是乱码

来源:http://blog.csdn.net/wang\_zhenwei/article/details/50285345

[Java]: http://lib.csdn.net/base/java
[view plain]: http://blog.csdn.net/wang_zhenwei/article/details/50285345#