功能

自动监控日志变化

核心组件

Source : 完成对日志数据的收集，分成transtion 和 event 打入到channel之中。
Channel : 主要提供一个队列的功能，对source提供中的数据进行简单的缓存。
Sink : 取出Channel中的数据，进行相应的存储文件系统，数据库，或者提交到远程服务器

学习核心：配置个性化-组件的类型

Flume（2）组件概述与列表

Flume 入门–几种不同的Sources

错误:找不到或无法加载主类 org.apache.flume.tools.GetJavaProperty

原因：hbase的配置

解决办法

配置文件示例1-spooldir

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.
See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.
The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.
You may obtain a copy of the License at
#
#
http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.
See the License for the
# specific language governing permissions and limitations
# under the License.
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
agent1.sources = avro-source1
agent1.channels = ch1
agent1.sinks = logger-sink1
# sources
agent1.sources.avro-source1.type = spooldir
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.spoolDir = /home/shuai/logs/
agent1.sources.avro-source1.fileHeader = true
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 4141
# sink
agent1.sinks.logger-sink1.type = logger
agent1.sinks.logger-sink1.channel = ch1
# channel
agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 1000
agent1.channels.ch1.transactionCapacity = 100

启动

参考教程启动[Flume]安装，部署与应用案例

./flume-ng agent –conf ../conf –conf-file ../conf/flume-spool.conf –name agent -Dflume.root.logger=INFO,console

没有提示报错，但是没什么功能

分析日志

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/hadoop/apache-flume-1.8.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2017-10-15 21:44:02,380 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
2017-10-15 21:44:02,382 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:../conf/flume-spool.conf
2017-10-15 21:44:02,385 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:logger-sink1
2017-10-15 21:44:02,386 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: logger-sink1 Agent: agent1
2017-10-15 21:44:02,386 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:logger-sink1
2017-10-15 21:44:02,392 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent1]
2017-10-15 21:44:02,392 (conf-file-poller-0) [WARN - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:135)] No configuration found for this host:agent
2017-10-15 21:44:02,396 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

agent的名字是agent1,改成agent1成功

./flume-ng agent --conf ../conf --conf-file ../conf/flume-spool.conf --name agent1 -Dflume.root.logger=INFO,console

配置文件示例2-spooldir

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.
See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.
The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.
You may obtain a copy of the License at
#
#
http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.
See the License for the
# specific language governing permissions and limitations
# under the License.
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink
# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = spooldir
agent.sources.seqGenSrc.spoolDir = /data/nginx/logs
# The channel can be defined as follows.
agent.sources.seqGenSrc.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.loggerSink.type = logger
#Specify the channel the sink should use
agent.sinks.loggerSink.channel = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity = 100

采用flume提供的默认模板，需要将source 的类型改成spooldir并添加监控的文件夹，此时的名称必须为agent才行
➜ bin ./flume-ng agent –conf ../conf –conf-file ../conf/flume-conf.properties –name agent -Dflume.root.logger=INFO,console

最后

以上就是糊涂雪糕为你收集整理的flume配置启动功能核心组件学习核心：配置个性化-组件的类型错误:找不到或无法加载主类 org.apache.flume.tools.GetJavaProperty配置文件示例1-spooldir启动配置文件示例2-spooldir的全部内容，希望文章能够帮你解决flume配置启动功能核心组件学习核心：配置个性化-组件的类型错误:找不到或无法加载主类 org.apache.flume.tools.GetJavaProperty配置文件示例1-spooldir启动配置文件示例2-spooldir所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。