Logstash拉取Oracle数据到Kafka

51 阅读 0 评论 34 点赞

我是靠谱客的博主活力墨镜，最近开发中收集的这篇文章主要介绍Logstash拉取Oracle数据到Kafka，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

背景

Logstash官方地址：https://www.elastic.co/guide/en/logstash/current/index.html
CDC：Change Data Capture
公司有同步Oracle数据库中记录的需求，传统的做法是用ETL的方式进行异构数据库的定时推送。但弊端比较明显，耦合性较强，且对研发不透明，容易忽略。随采取CDC的抽取，以消息服务的方式解耦，数据两侧间的直接耦合关系。

分析需求：
1、Oracle的数据动态抽取，不像MySQL可以用canal读取binlog，以达到主从秒级数据无差别读取。
2、Oracle商用抽取OGG，价格较高，公司无采购计划
3、程序定时读库并组装后发送到MQ，可控可调，需单独起一个服务。
4、Logstash，input+output方式可以达到动态抽取的轻量级实现

实施计划

数据路线：Oracle -> Logstash ->Kafka

细节

记录logstash不同版本配置的问题

Logstash > output > kafka配置

since 7.3
sasl_jaas_config

[2020-03-23T18:17:29,853][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-03-23T18:17:29,910][INFO ][logstash.runner
] Starting Logstash {"logstash.version"=>"6.6.2"}
[2020-03-23T18:17:46,361][ERROR][logstash.outputs.kafka
] Unknown setting 'sasl_jaas_config' for kafka
[2020-03-23T18:17:46,411][ERROR][logstash.agent
] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/config/mixin.rb:86:in `config_init'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/outputs/base.rb:60:in `initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:224:in `initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in `initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:235:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:181:in `plugin'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "(eval):39:in `<eval>'", "org/jruby/RubyKernel.java:994:in `eval'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/agent.rb:94:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/agent.rb:94:in `exclusive'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "/Users/xxx/Downloads/logstash-6.6.2/logstash-core/lib/logstash/agent.rb:327:in `block in converge_state'"]}
[2020-03-23T18:17:47,085][INFO ][logstash.agent
] Successfully started Logstash API endpoint {:port=>9600}