1. 首页
  2. 文章中心
  3. 大数据知识

flume负载均衡，容错，监控负载均衡容错

100 阅读 0 评论
我是靠谱客的博主 正直石头，这篇文章主要介绍flume负载均衡，容错，监控负载均衡容错，现在分享给大家，希望可以做个参考。

文章目录

    • 串联flume
    • exec-avro-agent.conf 客户端
    • avro-logger-agent.conf 服务端
  • 负载均衡
    • 客户端
    • 两个server端
  • 容错
    • 监控

串联flume

在这里插入图片描述用avro串联

exec-avro-agent.conf 客户端

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#agent
exec-avro-agent.sources = exec-source
exec-avro-agent.channels = memory-channel
exec-avro-agent.sinks = avro-sink

#source
exec-avro-agent.sources.exec-source.type = exec
exec-avro-agent.sources.exec-source.command = tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-avro-agent.channels.memory-channel.type = memory

#sink
exec-avro-agent.sinks.avro-sink.type = avro
exec-avro-agent.sinks.avro-sink.hostname = localhost
exec-avro-agent.sinks.avro-sink.port = 55555


#bind source and sink to channel
exec-avro-agent.sources.exec-source.channels = memory-channel
exec-avro-agent.sinks.avro-sink.channel = memory-channel

启动脚本

复制代码
1
2
3
./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-avro-agent.conf --name exec-avro-agent -Dflume.root.logger=INFO,console

avro-logger-agent.conf 服务端

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#agent
avro-logger-agent-1.sources = avro-source
avro-logger-agent-1.channels = memory-channel
avro-logger-agent-1.sinks = logger-sink

#source
avro-logger-agent-1.sources.avro-source.type = avro
avro-logger-agent-1.sources.avro-source.bind = localhost
avro-logger-agent-1.sources.avro-source.port = 55555

#channel
avro-logger-agent-1.channels.memory-channel.type = memory

#sink
avro-logger-agent-1.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-1.sources.avro-source.channels = memory-channel
avro-logger-agent-1.sinks.logger-sink.channel = memory-channel

启动

复制代码
1
2
./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent.conf --name avro-logger-agent-1 -Dflume.root.logger=INFO,console

先启动服务端，再启动客户端
往1.log写东西，发现服务端有数据，客户端没数据
在这里插入图片描述

负载均衡

在这里插入图片描述
在这里插入图片描述

客户端

exec-avro-agent-Load-balance.conf

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#agent
exec-avro-agent-Load-balance.sources = exec-source
exec-avro-agent-Load-balance.channels = memory-channel
exec-avro-agent-Load-balance.sinks = avro-sink-1 avro-sink-2

#负载平衡
exec-avro-agent-Load-balance.sinkgroups = g1
exec-avro-agent-Load-balance.sinkgroups.g1.sinks = avro-sink-1 avro-sink-2
exec-avro-agent-Load-balance.sinkgroups.g1.processor.type =load_balance
exec-avro-agent-Load-balance.sinkgroups.g1.processor.backoff =true
exec-avro-agent-Load-balance.sinkgroups.g1.processor.selector=round_robin

#source
exec-avro-agent-Load-balance.sources.exec-source.type = exec
exec-avro-agent-Load-balance.sources.exec-source.command = tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-avro-agent-Load-balance.channels.memory-channel.type = memory

#sink
exec-avro-agent-Load-balance.sinks.avro-sink-1.type = avro
exec-avro-agent-Load-balance.sinks.avro-sink-1.hostname = localhost
exec-avro-agent-Load-balance.sinks.avro-sink-1.port = 55555

exec-avro-agent-Load-balance.sinks.avro-sink-2.type = avro
exec-avro-agent-Load-balance.sinks.avro-sink-2.hostname = localhost
exec-avro-agent-Load-balance.sinks.avro-sink-2.port = 55556

#bind source and sink to channel
exec-avro-agent-Load-balance.sources.exec-source.channels = memory-channel
exec-avro-agent-Load-balance.sinks.avro-sink-1.channel = memory-channel
exec-avro-agent-Load-balance.sinks.avro-sink-2.channel = memory-channel

启动

复制代码
1
2
./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-avro-agent-Load-balance.conf --name exec-avro-agent-Load-balance -Dflume.root.logger=INFO,console

两个server端

avro-logger-agent-1.conf

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#agent
avro-logger-agent-1.sources = avro-source
avro-logger-agent-1.channels = memory-channel
avro-logger-agent-1.sinks = logger-sink

#source
avro-logger-agent-1.sources.avro-source.type = avro
avro-logger-agent-1.sources.avro-source.bind = localhost
avro-logger-agent-1.sources.avro-source.port = 55555

#channel
avro-logger-agent-1.channels.memory-channel.type = memory

#sink
avro-logger-agent-1.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-1.sources.avro-source.channels = memory-channel
avro-logger-agent-1.sinks.logger-sink.channel = memory-channel

启动

复制代码
1
2
./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent-1.conf --name avro-logger-agent-1 -Dflume.root.logger=INFO,console

avro-logger-agent-2.conf

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#agent
avro-logger-agent-2.sources = avro-source
avro-logger-agent-2.channels = memory-channel
avro-logger-agent-2.sinks = logger-sink

#source
avro-logger-agent-2.sources.avro-source.type = avro
avro-logger-agent-2.sources.avro-source.bind = localhost
avro-logger-agent-2.sources.avro-source.port = 55556

#channel
avro-logger-agent-2.channels.memory-channel.type = memory

#sink
avro-logger-agent-2.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-2.sources.avro-source.channels = memory-channel
avro-logger-agent-2.sinks.logger-sink.channel = memory-channel

启动

复制代码
1
2
./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent-2.conf --name avro-logger-agent-2 -Dflume.root.logger=INFO,console

先起两个服务端，再起客户端

然后可以看到两个服务端都可以接受数据，总数为log文件的内容，数据分配的策略，就是我们配置的round_robin算法

容错

在这里插入图片描述
在这里插入图片描述 这里的10，5是优先级，越大越优先，maxpenalty是冷却时间上限

可参考http://www.aboutyun.com/thread-9170-1-1.html

监控

在server端启动脚本最后面加上

复制代码
1
2
-Dflume.monitoring.type=http -Dflume.monitoring.port=34545

启动然后可以在网页上34545端看到数据

最后

以上就是正直石头最近收集整理的关于flume负载均衡，容错，监控负载均衡容错的全部内容，更多相关flume负载均衡内容请搜索靠谱客的其他文章。

本图文内容来源于网友提供，作为学习参考使用，或来自网络收集整理，版权属于原作者所有。
点赞(66)

相关文章

flume高并发优化——（1）load_balance
flume高并发优化——（1）load_balance
flume容错（故障转移）
flume容错（故障转移）
flume load_balance
flume load_balance
flume-ng负载均衡load-balance、failover集群搭建
flume-ng负载均衡load-balance、failover集群搭建
flume负载均衡，容错，监控负载均衡容错
flume负载均衡，容错，监控负载均衡容错
Flume的容错性测试和双层Flume架构一.Failover Sink Processor测试二.双层的Flume架构三.单source多channel多sink
Flume的容错性测试和双层Flume架构一.Failover Sink Processor测试二.双层的Flume架构三.单source多channel多sink
4.flume负载均衡（loadbalance）
4.flume负载均衡（loadbalance）
flume的负载均衡load balancer
flume的负载均衡load balancer

评论列表共有 0 条评论

立即
投稿
返回
顶部