centos 安装最新elk6.4.0搭建

48 阅读 0 评论 32 点赞

我是靠谱客的博主多情冰棍，最近开发中收集的这篇文章主要介绍centos 安装最新elk6.4.0搭建，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

1.安装jdk8

linux下使用wget下载jdk8:
进到目录/usr/local/software

  wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz"

解压文件：

tar zxvf jdk-8u181-linux-x64.tar.gz

mv jdk-8u181-linux-x64.tar.gz jdk8

增加环境变量，编辑对呀的文件

vim /etc/profile 
#加入
export JAVA_HOME=/usr/local/software/jdk8
export JAVA_BIN=/usr/local/software/jdk8
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH

#启动文件
source /etc/profile
#验证java是否安装成功
java -version

2.安装ELK

1.参考网站：https://www.elastic.co/downloads

2.通过wget命令下载 Elasticsearch/Logstash/Kibaber

1.下载elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
2.下载logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz
3.下载Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz

解压
tar -zxvf elasticsearch-6.4.0.tar.gz
tar -zxvf  logstash-6.4.0.tar.gz
tar -zxvf kibana-6.4.0-linux-x86_64.tar.gz

3.配置并启动Elasticsearch

配置es出现相关问题处理：
       1、问题一
           Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
           #
           # There is insufficient memory for the Java Runtime Environment to continue.
           # Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
           # An error report file with more information is saved as:
           # /usr/local/software/temp/elasticsearch-6.2.2/hs_err_pid1912.log
       解决：内存不够，购买阿里云的机器可以动态增加内存

       2、问题二
           [root@iZwz95j86y235aroi85ht0Z bin]# ./elasticsearch

       解决：用非root用户
           添加用户：useradd -m 用户名然后设置密码 passwd 用户名

       3、问题三
           ./elasticsearch
Exception in thread "main" java.nio.file.AccessDeniedException: /usr/local/software/temp/elasticsearch-6.4.0/config/jvm.options
       解决：权限不够 chmod 777 -R 当前es目录

常见配置问题资料：https://www.jianshu.com/p/c5d6ec0f35e0

使用nohub启动

nohup ./bin/elasticsearch &

在配置文件中增加http外网访问

bootstrap.memory_lock: false
:#增加centos 无法访问
bootstrap.system_call_filter: false

#增加外网访问

http.host: 0.0.0.0

4.配置logstash

input {
  beats {
    port => 5044
  }
}
filter {
   grok {
        match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
        remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[geoip]"
        add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

      }
     mutate {
      convert => [ "[geoip][coordinates]", "float" ]
     }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

启动logstart

./bin/logstash -f config/file-beats.conf

5.配置kibana

修改kibana.yml

server.host="0.0.0.0"

启动kibana

6.下载filebeat-6.3.2

下载地址：https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz
解压文件tar -zxvf filebeat-6.3.2-linux-x86_64.tar.gz

修改配置文件

vi filebeat.yml
文件内容如下：

#------------input部分----------
filebeat.prospectors:
- type: log
  paths:
    - /local/nas/docker/nginx/logs/access.log
  tags: ["nginx-accesslog"]
  document_type: nginx-access
#注意：filebeat在6版本里面，document_type字段好像不起作用

- type: log
  paths:
    - /local/nas/docker/nginx/logs/error.log
  tags: ["nginx-errorlog"]
  document_type: nginx-error

#-------------output部分，将输出到Elasticsearch注释掉，开启输出到logstash----
output.logstash:
  hosts: ["172.17.227.15:5044"]