flume消费kafka到hdfs

1.压缩包版本
flume版本1.7.0
kafka版本0.10.2.1
2.flume配置文件
进入conf文件
创建 flume-hdfs.conf文件，配置文件代码如下：

# source alias
agent.sources = source_from_kafka
# channels alias
agent.channels = mem_channel
# sink alias
agent.sinks = hdfs_sink
# define kafka source
agent.sources.source_from_kafka.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.source_from_kafka.channels = mem_channel
agent.sources.source_from_kafka.batchSize = 5000
# set kafka broker address
agent.sources.source_from_kafka.kafka.bootstrap.servers = master:9092,slave1:9092,slave2:9092
# set kafka topic
agent.sources.source_from_kafka.kafka.topics = test
# set kafka groupid
agent.sources.source_from_kafka.kafka.consumer.group.id = test-consumer-group
# defind hdfs sink
agent.sinks.hdfs_sink.type = hdfs
# specify the channel the sink should use
agent.sinks.hdfs_sink.channel = mem_channel
# set store hdfs path
agent.sinks.hdfs_sink.hdfs.path = /data/flume/kafka/%Y%m%d
# set file size to trigger roll
agent.sinks.hdfs_sink.hdfs.rollSize = 0
agent.sinks.hdfs_sink.hdfs.rollCount = 0
agent.sinks.hdfs_sink.hdfs.rollInterval = 3600
agent.sinks.hdfs_sink.hdfs.threadsPoolSize = 30
agent.sinks.hdfs_sink.hdfs.fileType=DataStream
agent.sinks.hdfs_sink.hdfs.writeFormat=Text
# define channel from kafka source to hdfs sink
agent.channels.mem_channel.type = memory
# channel store size
agent.channels.mem_channel.capacity = 100000
# transaction size
agent.channels.mem_channel.transactionCapacity = 10000

3.启动flume命令

./flume-ng agent --conf ../conf/ --name agent --conf-file ../conf/flume-hdfs.conf -Dflume.root.logger=INFO,console

4.启动kafka

./kafka-server-start.sh
-daemon ../config/server.properties

5.开启发送者并发送消息

./kafka-console-producer.sh --broker-list 192.168.88.5:9092 --topic test

6.查看文件是否采集成功

Hadoop fs -cat /data/flume/kafka/20190620/FlumeData.1561088964259.tmp

最后

以上就是温暖皮卡丘为你收集整理的flume消费kafka到hdfs的全部内容，希望文章能够帮你解决flume消费kafka到hdfs所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。