概述
转自：http://blog.csdn.net/morning_pig/article/details/9093149
集群采用3台机器：
- host1 load-balance设置
- host2 机器1
- host3 机器2
其中，host1 机器配置：
- #Define a memory channel called c1 on a1
- a1.channels = c1
- a1.sources = r1
- a1.sinks = k1 k2
- a1.sinkgroups = g1
- a1.sinkgroups.g1.sinks = k1 k2
- a1.sinkgroups.g1.processor.type = load_balance
- a1.sinkgroups.g1.processor.selector = round_robin
- a1.sinkgroups.g1.processor.backoff = true
- a1.channels.c1.type = file
- a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint
- a1.channels.c1.dataDirs = /tmp/flume/loaddata
- a1.sources.r1.channels = c1
- a1.sources.r1.type = avro
- a1.sources.r1.bind = 0.0.0.0
- a1.sources.r1.port = 41415
- a1.sinks.k1.channel = c1
- a1.sinks.k1.type = avro
- a1.sinks.k1.hostname = host2
- a1.sinks.k1.port = 41414
- a1.sinks.k2.channel = c1
- a1.sinks.k2.type = AVRO
- a1.sinks.k2.hostname = host3
- a1.sinks.k2.port = 41414
host2 机器配置：
- a2.channels = c1
- a2.sources = r1
- a2.sinks = k1
- a2.channels.c1.type = FILE
- a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
- a1.channels.c1.dataDirs = /tmp/flume/data
- a2.sources.r1.channels = c1
- a2.sources.r1.type = AVRO
- a2.sources.r1.bind = 0.0.0.0
- a2.sources.r1.port = 41414
- a2.sinks.k1.channel = c1
- a2.sinks.k1.type = file_roll
- a2.sinks.k1.sink.directory = /tmp/load/
- a2.sinks.k1.sink.rollInterval = 0
host3 机器配置：
- a2.channels = c1
- a2.sources = r1
- a2.sinks = k1
- a2.channels.c1.type = FILE
- a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
- a1.channels.c1.dataDirs = /tmp/flume/data
- a2.sources.r1.channels = c1
- a2.sources.r1.type = AVRO
- a2.sources.r1.bind = 0.0.0.0
- a2.sources.r1.port = 41414
- a2.sinks.k1.channel = c1
- a2.sinks.k1.type = file_roll
- a2.sinks.k1.sink.directory = /tmp/load/
- a2.sinks.k1.sink.rollInterval = 0
客户端机器暂时使用flume-ng的agent发送，配置如下：
- # Define a memory channel called c1 on a1
- a1.channels.c1.type = file
- a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
- a1.channels.c1.dataDirs = /tmp/flume/data
- a1.sources.r1.channels = c1
- a1.sources.r1.type = exec
- a1.sources.r1.command = cat /tmp/linux.log
- a1.sinks.k1.type = avro
- a1.sinks.k1.channel = c1
- a1.sinks.k1.hostname = host1
- a1.sinks.k1.port = 41415
- a1.channels = c1
- a1.sources = r1
- a1.sinks = k1
客户端/tmp/linux.log文件3G左右，发送给host1。
启动：
host2：
- bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2
host3：
- bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2
host1：
- bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1
客户端：
- bin/flume-ng agent -c conf -f conf/client.conf -n a1
注意：启动方式最好是从下往上启动。即：先启动host2和host3，然后启动host1，最后启动client。
测试过程中，可以随时将host2或host3停止，过一段时间再启动。
这样，就测试了flume-ng的load-balance和failover功能。
最后
以上就是文静金毛为你收集整理的flume-ng负载均衡load-balance、failover集群搭建的全部内容，希望文章能够帮你解决flume-ng负载均衡load-balance、failover集群搭建所遇到的程序开发问题。
如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。
本图文内容来源于网友提供，作为学习参考使用，或来自网络收集整理，版权属于原作者所有。
发表评论 取消回复