flume采集kafka数据到hdfs，不会产生大量小文件的配置Name the components on this agentsourcesink

59 阅读 0 评论 39 点赞

我是靠谱客的博主尊敬嚓茶，最近开发中收集的这篇文章主要介绍flume采集kafka数据到hdfs，不会产生大量小文件的配置Name the components on this agentsourcesink，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

Name the components on this agent

a1.sources=r1
a1.channels=c1
a1.sinks=k1

source

a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 5000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = cdh01:9092,cdh02:9092,cdh03:9092
a1.sources.r1.kafka.topics = DayFreezingDataTest

channel1

a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20

#官网说明
#Maximum total bytes of memory allowed as a sum of all events in this channel. The implementation only counts the Event body, which is the reason for providing the byteCapacityBufferPercentage configuration parameter as well. Defaults to a computed value equal to 80% of the maximum memory available to the JVM (i.e. 80% of the -Xmx value passed on the command line). Note that if you have multiple memory channels on a single JVM, and they happen to hold the same physical events (i.e. if you are using a replicating channel selector from a single source) then those event sizes may be double-counted for channel byteCapacity purposes. Setting this value to 0 will cause this value to fall back to a hard internal limit of about 200 GB.
#值太小会引发这个问题
#Cannot commit transaction. Byte capacity allocated to store event body 640000.0reached. Please increase heap space/byte capacity allocated to the channel as the sinks may not be keeping up with the sources

a1.channels.c1.byteCapacity = 800000

sink

a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /event/flume/kafkaToHDFS/test/DayFreezingDataTest/%y-%m-%d/%H
a1.sinks.k1.hdfs.filePrefix = event-