我是靠谱客的博主 欢喜电源，最近开发中收集的这篇文章主要介绍flume安装和使用环境安装使用1.在apache-flume-1.9.0-bin\conf目录下创建一个example.conf配置文件然后把官文档中的案例内容复制到example.conf文件中，如下内容： ，觉得挺不错的，现在分享给大家，希望可以做个参考。
概述
环境
Centos7+jdk11+flume1.9+telnet客户端
注：最新版flume1.8需要jdk1.8及以上支持
安装
jdk的安装不多赘述，flume1.9的安装。
进入官网：http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
然后找到1.9版本下载放到centos系统解压即可
使用
1.在apache-flume-1.9.0-binconf目录下创建一个example.conf配置文件
然后把官文档中的案例内容复制到example.conf文件中，如下内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
2.进入到解压flume目录中执行命令
文档中的执行命令如下：[root@mini3 apache-flume-1.9.0-bin]# bin/flume-ng agent --conf conf --conf-file conf/example.conf –name a1 -Dflume.root.logger=INFO,console参数的简要说明：指明conf文件路径、指明conf文件、指定agent、指明log打印信息级别和位置
更加详细的参数说明可查看文档中的介绍
执行效果：
Info: Including Hive libraries found via () for Hive access
- exec /usr/local/jdk1.7.0_65/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp ‘/usr/local/apache-flume-1.7.0-bin/conf:/usr/local/apache-flume-1.7.0-bin/lib/:/lib/’ -Djava.library.path= org.apache.flume.node.Application --conf-file example.conf –name a1
2018-01-31 18:14:04,361 (main) [ERROR - org.apache.flume.node.Application.main(Application.java:348)] A fatal error occurred while running. Exception follows.
org.apache.commons.cli.ParseException: The specified configuration file does not exist: /usr/local/apache-flume-1.9.0-bin/example.conf
at org.apache.flume.node.Application.main(Application.java:316)
[root@mini3 apache-flume-1.9.0-bin]# bin/flume-ng agent --conf conf --conf-file conf/example.conf –name a1 -Dflume.root.logger=INFO,console
Info: Including Hive libraries found via () for Hive access - exec /usr/local/jdk1.11.0_65/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp ‘/usr/local/apache-flume-1.9.0-bin/conf:/usr/local/apache-flume-1.9.0-bin/lib/:/lib/’ -Djava.library.path= org.apache.flume.node.Application --conf-file conf/example.conf –name a1
2018-01-31 18:14:45,241 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
2018-01-31 18:14:45,247 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider F i l e W a t c h e r R u n n a b l e . r u n ( P o l l i n g P r o p e r t i e s F i l e C o n f i g u r a t i o n P r o v i d e r . j a v a : < s p a n c l a s s = " h l j s − n u m b e r " > 134 < / s p a n > ) ] R e l o a d i n g c o n f i g u r a t i o n < s p a n c l a s s = " h l j s − k e y w o r d " > f i l e < / s p a n > : c o n f / e x a m p l e . c o n f < s p a n c l a s s = " h l j s − n u m b e r " > 2018 < / s p a n > − < s p a n c l a s s = " h l j s − n u m b e r " > 01 < / s p a n > − < s p a n c l a s s = " h l j s − n u m b e r " > 31 < / s p a n > < s p a n c l a s s = " h l j s − n u m b e r " > 18 < / s p a n > : < s p a n c l a s s = " h l j s − n u m b e r " > 14 < / s p a n > : < s p a n c l a s s = " h l j s − n u m b e r " > 45 < / s p a n > , < s p a n c l a s s = " h l j s − n u m b e r " > 254 < / s p a n > ( c o n f − < s p a n c l a s s = " h l j s − k e y w o r d " > f i l e < / s p a n > − p o l l e r − < s p a n c l a s s = " h l j s − n u m b e r " > 0 < / s p a n > ) [ I N F O − o r g . a p a c h e . f l u m e . c o n f . F l u m e C o n f i g u r a t i o n FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:<span class="hljs-number">134</span>)] Reloading configuration <span class="hljs-keyword">file</span>:conf/example.conf <span class="hljs-number">2018</span>-<span class="hljs-number">01</span>-<span class="hljs-number">31</span> <span class="hljs-number">18</span>:<span class="hljs-number">14</span>:<span class="hljs-number">45</span>,<span class="hljs-number">254</span> (conf-<span class="hljs-keyword">file</span>-poller-<span class="hljs-number">0</span>) [INFO - org.apache.flume.conf.FlumeConfiguration FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:<spanclass="hljs−number">134</span>)]Reloadingconfiguration<spanclass="hljs−keyword">file</span>:conf/example.conf<spanclass="hljs−number">2018</span>−<spanclass="hljs−number">01</span>−<spanclass="hljs−number">31</span><spanclass="hljs−number">18</span>:<spanclass="hljs−number">14</span>:<spanclass="hljs−number">45</span>,<spanclass="hljs−number">254</span>(conf−<spanclass="hljs−keyword">file</span>−poller−<spanclass="hljs−number">0</span>)[INFO−org.apache.flume.conf.FlumeConfigurationAgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2018-01-31 18:14:45,254 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration A g e n t C o n f i g u r a t i o n . a d d P r o p e r t y ( F l u m e C o n f i g u r a t i o n . j a v a : < s p a n c l a s s = " h l j s − n u m b e r " > 1016 < / s p a n > ) ] P r o c e s s i n g : k 1 < s p a n c l a s s = " h l j s − n u m b e r " > 2018 < / s p a n > − < s p a n c l a s s = " h l j s − n u m b e r " > 01 < / s p a n > − < s p a n c l a s s = " h l j s − n u m b e r " > 31 < / s p a n > < s p a n c l a s s = " h l j s − n u m b e r " > 18 < / s p a n > : < s p a n c l a s s = " h l j s − n u m b e r " > 14 < / s p a n > : < s p a n c l a s s = " h l j s − n u m b e r " > 45 < / s p a n > , < s p a n c l a s s = " h l j s − n u m b e r " > 254 < / s p a n > ( c o n f − < s p a n c l a s s = " h l j s − k e y w o r d " > f i l e < / s p a n > − p o l l e r − < s p a n c l a s s = " h l j s − n u m b e r " > 0 < / s p a n > ) [ I N F O − o r g . a p a c h e . f l u m e . c o n f . F l u m e C o n f i g u r a t i o n AgentConfiguration.addProperty(FlumeConfiguration.java:<span class="hljs-number">1016</span>)] Processing:k1 <span class="hljs-number">2018</span>-<span class="hljs-number">01</span>-<span class="hljs-number">31</span> <span class="hljs-number">18</span>:<span class="hljs-number">14</span>:<span class="hljs-number">45</span>,<span class="hljs-number">254</span> (conf-<span class="hljs-keyword">file</span>-poller-<span class="hljs-number">0</span>) [INFO - org.apache.flume.conf.FlumeConfiguration AgentConfiguration.addProperty(FlumeConfiguration.java:<spanclass="hljs−number">1016</span>)]Processing:k1<spanclass="hljs−number">2018</span>−<spanclass="hljs−number">01</span>−<spanclass="hljs−number">31</span><spanclass="hljs−number">18</span>:<spanclass="hljs−number">14</span>:<spanclass="hljs−number">45</span>,<spanclass="hljs−number">254</span>(conf−<spanclass="hljs−keyword">file</span>−poller−<spanclass="hljs−number">0</span>)[INFO−org.apache.flume.conf.FlumeConfigurationAgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2018-01-31 18:14:45,254 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfigurationKaTeX parse error: Expected 'EOF', got '&' at position 555: …eConfiguration.&̲lt;init>(Flu…AgentConfiguration.isValid(FlumeConfiguration.java:320)] Agent configuration for ‘mple’ does not contain any channels. Marking it as invalid.
2018-01-31 18:14:45,255 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127)] Agent configuration invalid for agent ‘mple’. It will be removed.
2018-01-31 18:14:45,271 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2018-01-31 18:14:45,316 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels
2018-01-31 18:14:45,322 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory
2018-01-31 18:14:45,329 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel c1
2018-01-31 18:14:45,329 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat
2018-01-31 18:14:45,337 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger
2018-01-31 18:14:45,345 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel c1 connected to [r1, k1]
2018-01-31 18:14:45,354 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@12bd971 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2018-01-31 18:14:45,365 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel c1
2018-01-31 18:14:45,368 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:159)] Waiting for channel: c1 to start. Sleeping for 500 ms
2018-01-31 18:14:45,467 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2018-01-31 18:14:45,468 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2018-01-31 18:14:45,869 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1
2018-01-31 18:14:45,870 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1
2018-01-31 18:14:45,870 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting
2018-01-31 18:14:45,894 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:169)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]如上打印日志，启动成功！
3.通过telnet客户端进行测试
另开一个终端命令行：输入命令
连接成功后即可进行模拟通信，即经过44444端口发送消息，让flume监听到。执行结果如下：telnet localhost 44444flume后台监听打印：
2018-01-31 18:15:48,913 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 6E 69 68 61 6F 0D nihao. }注：如果不能使用telnet，通过yum安装即可
最后
以上就是欢喜电源为你收集整理的flume安装和使用环境安装使用1.在apache-flume-1.9.0-bin\conf目录下创建一个example.conf配置文件然后把官文档中的案例内容复制到example.conf文件中，如下内容： 的全部内容，希望文章能够帮你解决flume安装和使用环境安装使用1.在apache-flume-1.9.0-bin\conf目录下创建一个example.conf配置文件然后把官文档中的案例内容复制到example.conf文件中，如下内容： 所遇到的程序开发问题。
如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。
本图文内容来源于网友提供，作为学习参考使用，或来自网络收集整理，版权属于原作者所有。
发表评论 取消回复