我是靠谱客的博主 认真薯片,这篇文章主要介绍ES集群搭建ES集群搭建,现在分享给大家,希望可以做个参考。

ES集群搭建

一、安装软件准备

系统:Centos7-2003

软件版本:JDK15.x、ES7.11.X。

官方下载地址:

https://www.elastic.co/cn/downloads/

社区镜像下载

http://dl.elasticsearch.cn/

华为镜像下载

https://mirrors.huaweicloud.com/

Docker容器下载

可选择ES官方镜像或者国内阿里云/网易等镜像

二、操作系统设置

生产环境必须部署在Linux,并需要一些系统配置才可生效。个人环境可以选择Windows,几乎无需配置。无论是单实例还是集群,所有的节点都要以下配置。注意以下配置时,请用root权限账号操作。

2.1 关闭防火墙状态

复制代码
1
2
3
4
#关闭命令 service firewalld stop chkconfig firewalld off

2.2 关闭SELINUX

配置文件:/etc/selinux/config

复制代码
1
2
3
#SELINUX=enforcing SELINUX=disabled

2.3 设置hostname

  • 若是虚拟机,建议配置独立的主机名,便于区分
  • 配置文件:/etc/sysconfig/network

方式一:直接hostnamectl命令

复制代码
1
2
hostnamectl set-hostname elkserver

方式二:配置文件修改

复制代码
1
2
3
NETWORKING=yes HOSTNAME=elkserver

2.4 关闭swap

  • 配置文件 /etc/sysctl.conf
复制代码
1
2
3
4
5
6
[root@elkserver /]# vim /etc/sysctl.conf # 禁用内存与硬盘交换 vm.swappiness=1 # 设置虚拟内存大小 vm.max_map_count=262144
  • 执行/sbin/sysctl -p 立即生效
复制代码
1
2
[root@elkserver /]# /sbin/sysctl -p

2.5 文件句柄设置

配置文件: /etc/security/limits.conf

复制代码
1
2
3
4
5
6
7
8
9
10
11
[root@elkserver /]# vim /etc/security/limits.conf # 进程线程数 * soft nproc 131072 * hard nproc 131072 # 文件句柄数 * soft nofile 131072 * hard nofile 131072 # jvm内存锁定交换 * soft memlock unlimited * hard memlock unlimited

2.6 ES专用账号并授权

  • 创建ES专用账号
复制代码
1
2
3
4
5
useradd es #授权ES程序目录es账号权限 #假设 ES程序目录、数据目录、日志目录都在/es目录下 chown -R es:es /es/*

2.7 JDK配置

将jdk-15.0.2_linux-x64_bin.tar.gz放到路径/usr/local/src/下并解压

复制代码
1
2
[root@elkserver src]# tar -zxvf jdk-15.0.2_linux-x64_bin.tar.gz
  • JDK配置(可选):/etc/profile

ES最新版本自带jdk版本,默认可以不需要配置,建议配置,便于安装其它java程序辅助

本内容中要求安装JDK15.x

复制代码
1
2
3
4
5
6
[root@elkserver src]# vim /etc/profile export JAVA_HOME=/usr/local/src/jdk-15.0.2 export JRE_HOME=$JAVA_HOME/jre export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
  • 保存退出后,使配置生效:
复制代码
1
2
[root@elkserver src]# source /etc/profile
复制代码
1
2
3
4
5
[root@elkserver src]# java -version java version "15.0.2" 2021-01-19 Java(TM) SE Runtime Environment (build 15.0.2+7-27) Java HotSpot(TM) 64-Bit Server VM (build 15.0.2+7-27, mixed mode, sharing)

2.8 JVM临时目录

如在es目录下创建/es/jvm-tmpdir

复制代码
1
2
3
[root@elkserver es]# mkdir jvm_tmpdir [root@elkserver elk]# chown -R es:es /es/*
  • 方式一:修改环境变量
复制代码
1
2
3
4
[root@elkserver src]# vim /etc/profile export ES_TMPDIR=/es/jvm_tmpdir [root@elkserver src]# source /etc/profile
  • 方式二:在jvm.options中设置(原来就有)
复制代码
1
2
-Djava.io.tmpdir=${ES_TMPDIR}

2.9 JNA临时目录

创建ES临时文件目录:jna_tmpdir,仅限制Linux操作系统

如在es目录下创建/es/jna-tmpdir

在启动JVM中增加此参数

在jvm.options中修改

复制代码
1
2
-Djna.tmpdir=/es/jna_tmpdir
复制代码
1
2
3
4
[root@elkserver es]# mkdir jna_tmpdir [root@elkserver es]# ls elasticsearch-7.11.1-linux-x86_64.tar.gz es-9200 jna_tmpdir jvm_tmpdir kibana-7.11.1-linux-x86_64.tar.gz

三、脚本命令

启动ES脚本

Windows系统:elasticsearch.bat

Linux系统:

​ 当前窗口启动: ./elasticsearch

​ 后台进程启动: ./elasticsearch -d

辅助工具

​ 安全、插件、环境、迁移。。。

四、ES配置文件

elasticsearch.yml 核心配置文件、节点实例属性参数

jvm.options 配置堆栈参数等

log4j2.properties 日志常规配置,默认就好

其他

账户安全/动态生成/keystore安全文件

配置elasticsearch.yml

  • 集群名称

    默认无需设置,建议设置

    如:cluster.name: elk01

  • 节点名称

    单实例不用设置,集群下必须区分设置,建议设置

    如:${HOSTNAME}-9200

  • IP地址

    必须设置,默认仅限本地访问

    如:network.host: 192.168.237.127

  • 访问端口

    http端口,默认无需设置

    tcp端口,默认需求设置

    如:http.port: 9200
    transport.port: 9300

  • 集群发现

    discovery.xxx 单节点无需配置

    cluster.xx.xx 单节点默认可无需配置

    单节点如:discovery.seed_hosts: [“192.168.237.127:9300”]

    cluster.initial_master_nodes: [“192.168.237.127:9300”]

    多节点集群如:

    discovery.seed_hosts: [“192.168.237.127:9301”,“192.168.237.127:9302”,“192.168.237.127:9303”]

    cluster.initial_master_nodes: [“192.168.237.127:9301”,“192.168.237.127:9302”,“192.168.237.127:9303”]

  • 数据目录与日志目录

    path.data: /es/es-9200/elk/data

    path.logs: /es/es-9200/elk/logs

  • 内存交换锁定,此处需要操作系统设置才生效

    bootstrap.memory_lock: true

  • 防止批量删除索引
    action.destructive_requires_name: true

  • 设置处理器数量4,默认无需设置,单机器多实例需要设置
    node.processors: 4

  • 设置跨域操作

    http.cors.enabled 是否支持跨域,默认为false http.cors.allow-origin 当设置允许跨域,默认为*,表示支持所有域名,如果我们只是允许某些网站能访问,那么可以使用正则表达式。比如只允许本地地址。

    如:http.cors.enabled: true
    http.cors.allow-origin: “*”

jvm.options

  • JDK 选择

    自主配置或者自带

  • GC选择

    7.8.1已经采用G1、原则JDK14以上采用G1,以下采用CMS、7.11.x

  • 堆栈大小

    默认4GB

    不超过1/2系统内存

    空余1/2闲置内存

    内存上限不超过32G,且不能等于32G

  • GC日志

  • gc.log路径设置

五、Kibana安装配置

Kibana作为ES的开发者工具、管理运维工具(不存储数据)。

  • bin

    启动脚本、常用工具脚本

  • config

    文件配置

  • 其它

    默认,无需关系

kibana.yml

  • ES配置指向 必须配置,默认仅本地

    如:elasticsearch.hosts: [“http://192.168.237.127:9201”]

  • IP+Port设置 IP地址必须配置,默认仅限本机访问

    如:server.port: 6601

    ​ server.host: “192.168.237.127”

  • 元数据索引名称

    默认无需配置,建议配置修改

    如:kibana.index: “.kibana-elk03”

  • 其它设置

    安全、语言、其它

六、单节点实战部署

虚拟机:内存8G及以上,处理器4核,硬盘30G

命令 free -m查看系统资源

之前准备工作创建es文件资源目录 mkdir es,进入目录后pwd命令如下

复制代码
1
2
3
4
5
6
[root@elkserver /]# cd es [root@elkserver es]# pwd /es [root@elkserver es]# hostname elkserver

在es目录下放入es安装包并解压,

复制代码
1
2
[root@elkserver es]# tar -zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz

解压后将解压出的文件夹改名为es-9200

复制代码
1
2
[root@elkserver es]# mv elasticsearch-7.11.1 es-9200
复制代码
1
2
3
4
5
6
[root@elkserver es-9200]# cd elk [root@elkserver elk]# mkdir data [root@elkserver elk]# mkdir logs [root@elkserver elk]# ls data logs

同样解压Kibana,并重命名文件夹

复制代码
1
2
[root@elkserver es]# tar -zxvf kibana-7.11.1-linux-x86_64.tar.gz
复制代码
1
2
[root@elkserver es]# mv kibana-7.11.1-linux-x86_64 kibana-5601

改变目录权限(上面已提到过)

复制代码
1
2
[root@elkserver es]# chown -R es:es /es/*

进入es-9200文件夹,查看es自带jdk版本

复制代码
1
2
3
4
5
6
[root@elkserver es-9200]# cd jdk/bin [root@elkserver bin]# ./java -version openjdk version "15.0.1" 2020-10-20 OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

进入到es-9200/config/文件夹下进行配置文件的配置

复制代码
1
2
[root@elkserver config]# vim jvm.options

增加如下:

-Xms1g
-Xmx1g

其他不建议修改,如修改比如log目录等。保存后退出 :wq

复制代码
1
2
[root@elkserver config]# vim elasticsearch.yml

详细配置见上面 第四章的内容,附录单节点9200的elasticsearch.yml文件内容如下:(注意将临时文件及日志文件路径事先创建好并授权给es用户)

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk01 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9200 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9200/elk/data # # Path to log files: # path.logs: /es/es-9200/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9200 transport.port: 9300 # Lock the memory on startup: # #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # # # #防止批量删除索引 action.destructive_requires_name: true # discovery.type : single-node # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9300"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9300"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true http.cors.enabled: true http.cors.allow-origin: "*"

进入kibana配置路径并修改配置

复制代码
1
2
3
4
5
6
7
8
9
[root@elkserver es-9200]# cd .. [root@elkserver es]# ls elasticsearch-7.11.1-linux-x86_64.tar.gz es-9200 es-9201 es-9202 es-9203 jna-tmpdir kibana-5601 kibana-6601 kibana-7.11.1-linux-x86_64.tar.gz [root@elkserver es]# cd kibana-5601 [root@elkserver kibana-5601]# ls bin config data LICENSE.txt node node_modules NOTICE.txt package.json plugins README.txt src x-pack [root@elkserver kibana-5601]# cd config [root@elkserver config]
复制代码
1
2
[root@elkserver config]# vim kibana.yml

详细配置见上面 第四章的内容,附录单节点5601的kibana.yml文件内容如下:

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
# Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "192.168.237.127" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse proxy. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false # Specifies the public URL at which Kibana is available for end users. If # `server.basePath` is configured this URL should end with the same basePath. #server.publicBaseUrl: "" # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["http://192.168.237.127:9200"] # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. kibana.index: ".kibana-elk01" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. #elasticsearch.username: "kibana_system" #elasticsearch.password: "pass" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files are used to verify the identity of Kibana to Elasticsearch and are required when # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's value to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false # Specifies the path where Kibana creates the process ID file. #pid.file: /run/kibana/kibana.pid # Enables you to specify a file where Kibana stores log output. #logging.dest: stdout # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Specifies locale to be used for all localizable strings, dates and number formats. # Supported languages are the following: English - en , by default , Chinese - zh-CN . #i18n.locale: "en"

启动ES

切换到es用户,并回到es-9200目录下进行命令启动

复制代码
1
2
3
[root@elkserver es-9200]# su es [es@elkserver es-9200]# ./bin/elasticsearch
  • 启动成功后,windows主机浏览器输入es的ip及端口如:http://192.168.237.127:9200/

    显示如下,即ES启动成功

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{ "name" : "elkserver-9200", "cluster_name" : "elk01", "cluster_uuid" : "_o7bbB61RHSVd9kLbTKCiQ", "version" : { "number" : "7.11.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a", "build_date" : "2021-02-15T13:44:09.394032Z", "build_snapshot" : false, "lucene_version" : "8.7.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }
  • 检查节点的健康状态

    http://192.168.237.127:9200/_cat/health

    复制代码
    1
    2
    1621091301 15:08:21 elk01 green 1 1 6 6 0 0 0 0 - 100.0%
  • 查看节点状态

    http://192.168.237.127:9200/_cat/nodes

    复制代码
    1
    2
    192.168.237.127 28 88 4 0.04 0.30 0.27 cdhilmrstw * elkserver-9200
  • xshell中关闭 ctrl +c

启动Kibana

进入到es/kibana-5601文件夹下,执行如下命令启动kibana

复制代码
1
2
[es@elkserver kibana-5601]$ ./bin/kibana
  • 启动成功后,windows主机浏览器访问kibana的ip及端口如: http://192.168.237.127:5601

    • 点击ADD data

    • 点击Sample data 依次添加样例数据(样例数据很重要,后续会使用)

    • 进入左上角菜单 Management——>Stack Monitoring

    • 点击 Or, set up with self monitoring 使用内部手机设置

    • 打开 Turn on monitoring

  • 如看到es节点变黄,则在kibana中的Dev tools中执行如下即可

    复制代码
    1
    2
    PUT /_settings { "number_of_replicas": 0 }

    xshell中关闭 ctrl +c

七、集群环境搭建

ES集群配置

Elastic集群模式必须至少2个实例以上,一般建议3个节点以上,可以保障其中一个节点失效,集群仍然可以服务。集群模式与单实例模式大部分配置上是一样的,仅需修改集群通信差异部分。

集群环境说明

在这里在一台机器但起3个节点服务的集群。原则上与三台机器各起一个节点服务是一样的。

服务器IP:192.168.237.127

服务器hostname:elkserver

ES文件夹目录:es/es-9201、es/es-9202、es/es-9203

三个服务的es端口号:9201、9202、9203

Kibana启动一个节点即可:es kibana-6601 端口号:6601

安装说明

注意需重新解压一个es并命名为es-9201

复制代码
1
2
3
[root@elkserver es]# tar -zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz [root@elkserver es]# mv elasticsearch-7.11.1 es-9201

注意需重新解压一个kibana并命名为 kibana-6601

复制代码
1
2
3
[root@elkserver es]# tar -zxvf kibana-7.11.1-linux-x86_64.tar.gz [root@elkserver es]# mv kibana-7.11.1-linux-x86_64 kibana-6601

ES的9201配置文件内容如下:

jvm.options

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
################################################################ ## ## JVM configuration ## ################################################################ ## ## WARNING: DO NOT EDIT THIS FILE. If you want to override the ## JVM options in this file, or set any additional options, you ## should create one or more files in the jvm.options.d ## directory containing your adjustments. ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html ## for more information. ## ################################################################ ################################################################ ## IMPORTANT: JVM heap size ################################################################ ## ## The heap size is automatically configured by Elasticsearch ## based on the available memory in your system and the roles ## each node is configured to fulfill. If specifying heap is ## required, it should be done through a file in jvm.options.d, ## and the min and max should be set to the same value. For ## example, to set the heap to 4 GB, create a new file in the ## jvm.options.d directory containing these lines: ## ## -Xms4g ## -Xmx4g ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html ## for more information ## ################################################################ -Xms1g -Xmx1g ################################################################ ## Expert settings ################################################################ ## ## All settings below here are considered expert settings. Do ## not adjust them unless you understand what you are doing. Do ## not edit them in this file; instead, create a new file in the ## jvm.options.d directory containing your adjustments. ## ################################################################ ## GC configuration 8-13:-XX:+UseConcMarkSweepGC 8-13:-XX:CMSInitiatingOccupancyFraction=75 8-13:-XX:+UseCMSInitiatingOccupancyOnly ## G1GC Configuration # NOTE: G1 GC is only supported on JDK version 10 or later # to use G1GC, uncomment the next two lines and update the version on the # following three lines to your version of the JDK # 10-13:-XX:-UseConcMarkSweepGC # 10-13:-XX:-UseCMSInitiatingOccupancyOnly 14-:-XX:+UseG1GC ## JVM temporary directory -Djava.io.tmpdir=${ES_TMPDIR} ## heap dumps # generate a heap dump when an allocation from the Java heap fails; heap dumps # are created in the working directory of the JVM unless an alternative path is # specified -XX:+HeapDumpOnOutOfMemoryError # specify an alternative path for heap dumps; ensure the directory exists and # has sufficient space -XX:HeapDumpPath=data # specify an alternative path for JVM fatal error logs -XX:ErrorFile=logs/hs_err_pid%p.log ## JDK 8 GC logging 8:-XX:+PrintGCDetails 8:-XX:+PrintGCDateStamps 8:-XX:+PrintTenuringDistribution 8:-XX:+PrintGCApplicationStoppedTime 8:-Xloggc:logs/gc.log 8:-XX:+UseGCLogFileRotation 8:-XX:NumberOfGCLogFiles=32 8:-XX:GCLogFileSize=64m # JDK 9+ GC logging 9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

附es-9201节点的elasticsearch.yml

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk03 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9201 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9201/elk/data # # Path to log files: # path.logs: /es/es-9201/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # Lock the memory on startup: #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # #防止批量删除索引 action.destructive_requires_name: true # #设置处理器数量4,默认无需设置,单机器多实例需要设置 node.processors: 4 # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9201 transport.port: 9301 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true http.cors.enabled: true http.cors.allow-origin: "*"
  • 创建日志路径及数据路径
复制代码
1
2
3
4
5
6
7
8
9
10
[root@elkserver es-9201]# mkdir elk [root@elkserver es-9201]# ls bin config elk jdk lib LICENSE.txt logs modules NOTICE.txt plugins README.asciidoc [root@elkserver es-9201]# cd elk [root@elkserver elk]# mkdir data [root@elkserver elk]# mkdir logs [root@elkserver elk]# ls data logs [root@elkserver elk]# chown -R es:es /es/*

es-9202、es-9203文件夹的生成方法

  • 其余两个节点,可用第一个节点文件夹复制的形式,再修改配置。
复制代码
1
2
3
[root@elkserver es]# cp -r es-9201/ es-9202 [root@elkserver es]# cp -r es-9201/ es-9203
  • 只需修改两个节点的elasticsearch.yml 以及临时文件和日志文件的名称即可。

附es-9202节点的elasticsearch.yml

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk03 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9202 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9202/elk/data # # Path to log files: # path.logs: /es/es-9202/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # Lock the memory on startup: #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # #防止批量删除索引 action.destructive_requires_name: true # #设置处理器数量4,默认无需设置,单机器多实例需要设置 node.processors: 4 # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9202 transport.port: 9302 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true

附es-9203节点的elasticsearch.yml

复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk03 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9203 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9203/elk/data # # Path to log files: # path.logs: /es/es-9203/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # Lock the memory on startup: #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # #防止批量删除索引 action.destructive_requires_name: true # #设置处理器数量4,默认无需设置,单机器多实例需要设置 node.processors: 4 # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9203 transport.port: 9303 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true
  • 注意:新生成文件夹后需要进行重新授权。chown -R es:es /es/*

启动ES集群环境

  • 切换到es账户下 su - es,分别依次启动三个节点的es服务
复制代码
1
2
3
4
[es@elkserver es-9201]$ ./bin/elasticsearch [es@elkserver es-9202]$ ./bin/elasticsearch [es@elkserver es-9203]$ ./bin/elasticsearch

检查三个节点启动是否成功

  • 依次在windows主机的浏览器中输入三个ES节点的地址信息进行查看

http://192.168.237.127:9201/

http://192.168.237.127:9202/

http://192.168.237.127:9203/

检查三个节点的健康状态

http://192.168.237.127:9201/_cat/health

结果:1621074718 10:31:58 elk03 green 3 3 53 25 0 0 0 0 - 100.0%

http://192.168.237.127:9202/_cat/health

结果:1621074689 10:31:29 elk03 green 3 3 53 25 0 0 0 0 - 100.0%

http://192.168.237.127:9203/_cat/health

结果:1621074658 10:30:58 elk03 green 3 3 53 25 0 0 0 0 - 100.0%

检查集群是否成功启动

-在任意节点看即可

http://192.168.237.127:9202/_cat/nodes

结果:

复制代码
1
2
3
4
192.168.237.127 39 97 7 0.28 1.19 0.88 cdhilmrstw * elkserver-9201 192.168.237.127 26 97 7 0.28 1.19 0.88 cdhilmrstw - elkserver-9203 192.168.237.127 43 97 6 0.28 1.19 0.88 cdhilmrstw - elkserver-9202

配置Kibana

附6601节点Kibana的kibana.yml配置

  • 注elasticsearch.hosts只需配置集群中一个节点即可
复制代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
# Kibana is served by a back end server. This setting specifies the port to use. server.port: 6601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "192.168.237.127" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse proxy. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false # Specifies the public URL at which Kibana is available for end users. If # `server.basePath` is configured this URL should end with the same basePath. #server.publicBaseUrl: "" # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["http://192.168.237.127:9201"] # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. kibana.index: ".kibana-elk03" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. #elasticsearch.username: "kibana_system" #elasticsearch.password: "pass" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files are used to verify the identity of Kibana to Elasticsearch and are required when # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's value to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false # Specifies the path where Kibana creates the process ID file. #pid.file: /run/kibana/kibana.pid # Enables you to specify a file where Kibana stores log output. #logging.dest: stdout # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Specifies locale to be used for all localizable strings, dates and number formats. # Supported languages are the following: English - en , by default , Chinese - zh-CN . #i18n.locale: "en"

启动Kibana环境

复制代码
1
2
[es@elkserver kibana-6601]$ ./bin/kibana

检查Kibana环境启动是否成功

  • windows主机浏览器访问:

http://192.168.237.127:6601/

  • 启动成功后,windows主机浏览器访问kibana的ip及端口如: http://192.168.237.127:6601

    • 点击ADD data

    • 点击Sample data 依次添加样例数据(样例数据很重要,后续会使用)

    • 进入左上角菜单 Management——>Stack Monitoring

    • 点击 Or, set up with self monitoring 使用内部手机设置

    • 打开 Turn on monitoring

  • 如看到es节点变黄,则在kibana中的Dev tools中执行如下即可

    复制代码
    1
    2
    PUT /_settings { "number_of_replicas": 0 }

    xshell中关闭 ctrl +c

在这里插入图片描述

  • 安装部署成功!

最后

以上就是认真薯片最近收集整理的关于ES集群搭建ES集群搭建的全部内容,更多相关ES集群搭建ES集群搭建内容请搜索靠谱客的其他文章。

本图文内容来源于网友提供,作为学习参考使用,或来自网络收集整理,版权属于原作者所有。
点赞(77)

评论列表共有 0 条评论

立即
投稿
返回
顶部