ES集群搭建
一、安装软件准备
系统:Centos7-2003
软件版本:JDK15.x、ES7.11.X。
官方下载地址:
https://www.elastic.co/cn/downloads/
社区镜像下载
http://dl.elasticsearch.cn/
华为镜像下载
https://mirrors.huaweicloud.com/
Docker容器下载
可选择ES官方镜像或者国内阿里云/网易等镜像
二、操作系统设置
生产环境必须部署在Linux,并需要一些系统配置才可生效。个人环境可以选择Windows,几乎无需配置。无论是单实例还是集群,所有的节点都要以下配置。注意以下配置时,请用root权限账号操作。
2.1 关闭防火墙状态
1
2
3
4#关闭命令 service firewalld stop chkconfig firewalld off
2.2 关闭SELINUX
配置文件:/etc/selinux/config
1
2
3#SELINUX=enforcing SELINUX=disabled
2.3 设置hostname
- 若是虚拟机,建议配置独立的主机名,便于区分
- 配置文件:/etc/sysconfig/network
方式一:直接hostnamectl命令
1
2hostnamectl set-hostname elkserver
方式二:配置文件修改
1
2
3NETWORKING=yes HOSTNAME=elkserver
2.4 关闭swap
- 配置文件 /etc/sysctl.conf
1
2
3
4
5
6[root@elkserver /]# vim /etc/sysctl.conf # 禁用内存与硬盘交换 vm.swappiness=1 # 设置虚拟内存大小 vm.max_map_count=262144
- 执行/sbin/sysctl -p 立即生效
1
2[root@elkserver /]# /sbin/sysctl -p
2.5 文件句柄设置
配置文件: /etc/security/limits.conf
1
2
3
4
5
6
7
8
9
10
11[root@elkserver /]# vim /etc/security/limits.conf # 进程线程数 * soft nproc 131072 * hard nproc 131072 # 文件句柄数 * soft nofile 131072 * hard nofile 131072 # jvm内存锁定交换 * soft memlock unlimited * hard memlock unlimited
2.6 ES专用账号并授权
- 创建ES专用账号
1
2
3
4
5useradd es #授权ES程序目录es账号权限 #假设 ES程序目录、数据目录、日志目录都在/es目录下 chown -R es:es /es/*
2.7 JDK配置
将jdk-15.0.2_linux-x64_bin.tar.gz放到路径/usr/local/src/下并解压
1
2[root@elkserver src]# tar -zxvf jdk-15.0.2_linux-x64_bin.tar.gz
- JDK配置(可选):/etc/profile
ES最新版本自带jdk版本,默认可以不需要配置,建议配置,便于安装其它java程序辅助
本内容中要求安装JDK15.x
1
2
3
4
5
6[root@elkserver src]# vim /etc/profile export JAVA_HOME=/usr/local/src/jdk-15.0.2 export JRE_HOME=$JAVA_HOME/jre export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
- 保存退出后,使配置生效:
1
2[root@elkserver src]# source /etc/profile
1
2
3
4
5[root@elkserver src]# java -version java version "15.0.2" 2021-01-19 Java(TM) SE Runtime Environment (build 15.0.2+7-27) Java HotSpot(TM) 64-Bit Server VM (build 15.0.2+7-27, mixed mode, sharing)
2.8 JVM临时目录
如在es目录下创建/es/jvm-tmpdir
1
2
3[root@elkserver es]# mkdir jvm_tmpdir [root@elkserver elk]# chown -R es:es /es/*
- 方式一:修改环境变量
1
2
3
4[root@elkserver src]# vim /etc/profile export ES_TMPDIR=/es/jvm_tmpdir [root@elkserver src]# source /etc/profile
- 方式二:在jvm.options中设置(原来就有)
1
2-Djava.io.tmpdir=${ES_TMPDIR}
2.9 JNA临时目录
创建ES临时文件目录:jna_tmpdir,仅限制Linux操作系统
如在es目录下创建/es/jna-tmpdir
在启动JVM中增加此参数
在jvm.options中修改
1
2-Djna.tmpdir=/es/jna_tmpdir
1
2
3
4[root@elkserver es]# mkdir jna_tmpdir [root@elkserver es]# ls elasticsearch-7.11.1-linux-x86_64.tar.gz es-9200 jna_tmpdir jvm_tmpdir kibana-7.11.1-linux-x86_64.tar.gz
三、脚本命令
启动ES脚本
Windows系统:elasticsearch.bat
Linux系统:
当前窗口启动: ./elasticsearch
后台进程启动: ./elasticsearch -d
辅助工具
安全、插件、环境、迁移。。。
四、ES配置文件
elasticsearch.yml 核心配置文件、节点实例属性参数
jvm.options 配置堆栈参数等
log4j2.properties 日志常规配置,默认就好
其他
账户安全/动态生成/keystore安全文件
配置elasticsearch.yml
-
集群名称
默认无需设置,建议设置
如:cluster.name: elk01
-
节点名称
单实例不用设置,集群下必须区分设置,建议设置
如:${HOSTNAME}-9200
-
IP地址
必须设置,默认仅限本地访问
如:network.host: 192.168.237.127
-
访问端口
http端口,默认无需设置
tcp端口,默认需求设置
如:http.port: 9200
transport.port: 9300 -
集群发现
discovery.xxx 单节点无需配置
cluster.xx.xx 单节点默认可无需配置
单节点如:discovery.seed_hosts: [“192.168.237.127:9300”]
cluster.initial_master_nodes: [“192.168.237.127:9300”]
多节点集群如:
discovery.seed_hosts: [“192.168.237.127:9301”,“192.168.237.127:9302”,“192.168.237.127:9303”]
cluster.initial_master_nodes: [“192.168.237.127:9301”,“192.168.237.127:9302”,“192.168.237.127:9303”]
-
数据目录与日志目录
path.data: /es/es-9200/elk/data
path.logs: /es/es-9200/elk/logs
-
内存交换锁定,此处需要操作系统设置才生效
bootstrap.memory_lock: true
-
防止批量删除索引
action.destructive_requires_name: true -
设置处理器数量4,默认无需设置,单机器多实例需要设置
node.processors: 4 -
设置跨域操作
http.cors.enabled 是否支持跨域,默认为false http.cors.allow-origin 当设置允许跨域,默认为*,表示支持所有域名,如果我们只是允许某些网站能访问,那么可以使用正则表达式。比如只允许本地地址。
如:http.cors.enabled: true
http.cors.allow-origin: “*”
jvm.options
-
JDK 选择
自主配置或者自带
-
GC选择
7.8.1已经采用G1、原则JDK14以上采用G1,以下采用CMS、7.11.x
-
堆栈大小
默认4GB
不超过1/2系统内存
空余1/2闲置内存
内存上限不超过32G,且不能等于32G
-
GC日志
-
gc.log路径设置
五、Kibana安装配置
Kibana作为ES的开发者工具、管理运维工具(不存储数据)。
-
bin
启动脚本、常用工具脚本
-
config
文件配置
-
其它
默认,无需关系
kibana.yml
-
ES配置指向 必须配置,默认仅本地
如:elasticsearch.hosts: [“http://192.168.237.127:9201”]
-
IP+Port设置 IP地址必须配置,默认仅限本机访问
如:server.port: 6601
server.host: “192.168.237.127”
-
元数据索引名称
默认无需配置,建议配置修改
如:kibana.index: “.kibana-elk03”
-
其它设置
安全、语言、其它
六、单节点实战部署
虚拟机:内存8G及以上,处理器4核,硬盘30G
命令 free -m查看系统资源
之前准备工作创建es文件资源目录 mkdir es,进入目录后pwd命令如下
1
2
3
4
5
6[root@elkserver /]# cd es [root@elkserver es]# pwd /es [root@elkserver es]# hostname elkserver
在es目录下放入es安装包并解压,
1
2[root@elkserver es]# tar -zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz
解压后将解压出的文件夹改名为es-9200
1
2[root@elkserver es]# mv elasticsearch-7.11.1 es-9200
1
2
3
4
5
6[root@elkserver es-9200]# cd elk [root@elkserver elk]# mkdir data [root@elkserver elk]# mkdir logs [root@elkserver elk]# ls data logs
同样解压Kibana,并重命名文件夹
1
2[root@elkserver es]# tar -zxvf kibana-7.11.1-linux-x86_64.tar.gz
1
2[root@elkserver es]# mv kibana-7.11.1-linux-x86_64 kibana-5601
改变目录权限(上面已提到过)
1
2[root@elkserver es]# chown -R es:es /es/*
进入es-9200文件夹,查看es自带jdk版本
1
2
3
4
5
6[root@elkserver es-9200]# cd jdk/bin [root@elkserver bin]# ./java -version openjdk version "15.0.1" 2020-10-20 OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
进入到es-9200/config/文件夹下进行配置文件的配置
1
2[root@elkserver config]# vim jvm.options
增加如下:
-Xms1g
-Xmx1g
其他不建议修改,如修改比如log目录等。保存后退出 :wq
1
2[root@elkserver config]# vim elasticsearch.yml
详细配置见上面 第四章的内容,附录单节点9200的elasticsearch.yml文件内容如下:(注意将临时文件及日志文件路径事先创建好并授权给es用户)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk01 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9200 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9200/elk/data # # Path to log files: # path.logs: /es/es-9200/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9200 transport.port: 9300 # Lock the memory on startup: # #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # # # #防止批量删除索引 action.destructive_requires_name: true # discovery.type : single-node # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9300"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9300"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true http.cors.enabled: true http.cors.allow-origin: "*"
进入kibana配置路径并修改配置
1
2
3
4
5
6
7
8
9[root@elkserver es-9200]# cd .. [root@elkserver es]# ls elasticsearch-7.11.1-linux-x86_64.tar.gz es-9200 es-9201 es-9202 es-9203 jna-tmpdir kibana-5601 kibana-6601 kibana-7.11.1-linux-x86_64.tar.gz [root@elkserver es]# cd kibana-5601 [root@elkserver kibana-5601]# ls bin config data LICENSE.txt node node_modules NOTICE.txt package.json plugins README.txt src x-pack [root@elkserver kibana-5601]# cd config [root@elkserver config]
1
2[root@elkserver config]# vim kibana.yml
详细配置见上面 第四章的内容,附录单节点5601的kibana.yml文件内容如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112# Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "192.168.237.127" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse proxy. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false # Specifies the public URL at which Kibana is available for end users. If # `server.basePath` is configured this URL should end with the same basePath. #server.publicBaseUrl: "" # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["http://192.168.237.127:9200"] # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. kibana.index: ".kibana-elk01" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. #elasticsearch.username: "kibana_system" #elasticsearch.password: "pass" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files are used to verify the identity of Kibana to Elasticsearch and are required when # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's value to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false # Specifies the path where Kibana creates the process ID file. #pid.file: /run/kibana/kibana.pid # Enables you to specify a file where Kibana stores log output. #logging.dest: stdout # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Specifies locale to be used for all localizable strings, dates and number formats. # Supported languages are the following: English - en , by default , Chinese - zh-CN . #i18n.locale: "en"
启动ES
切换到es用户,并回到es-9200目录下进行命令启动
1
2
3[root@elkserver es-9200]# su es [es@elkserver es-9200]# ./bin/elasticsearch
-
启动成功后,windows主机浏览器输入es的ip及端口如:http://192.168.237.127:9200/
显示如下,即ES启动成功
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18{ "name" : "elkserver-9200", "cluster_name" : "elk01", "cluster_uuid" : "_o7bbB61RHSVd9kLbTKCiQ", "version" : { "number" : "7.11.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a", "build_date" : "2021-02-15T13:44:09.394032Z", "build_snapshot" : false, "lucene_version" : "8.7.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }
-
检查节点的健康状态
http://192.168.237.127:9200/_cat/health
复制代码1
21621091301 15:08:21 elk01 green 1 1 6 6 0 0 0 0 - 100.0%
-
查看节点状态
http://192.168.237.127:9200/_cat/nodes
复制代码1
2192.168.237.127 28 88 4 0.04 0.30 0.27 cdhilmrstw * elkserver-9200
-
xshell中关闭 ctrl +c
启动Kibana
进入到es/kibana-5601文件夹下,执行如下命令启动kibana
1
2[es@elkserver kibana-5601]$ ./bin/kibana
-
启动成功后,windows主机浏览器访问kibana的ip及端口如: http://192.168.237.127:5601
-
点击ADD data
-
点击Sample data 依次添加样例数据(样例数据很重要,后续会使用)
-
进入左上角菜单 Management——>Stack Monitoring
-
点击 Or, set up with self monitoring 使用内部手机设置
-
打开 Turn on monitoring
-
-
如看到es节点变黄,则在kibana中的Dev tools中执行如下即可
复制代码1
2PUT /_settings { "number_of_replicas": 0 }
xshell中关闭 ctrl +c
七、集群环境搭建
ES集群配置
Elastic集群模式必须至少2个实例以上,一般建议3个节点以上,可以保障其中一个节点失效,集群仍然可以服务。集群模式与单实例模式大部分配置上是一样的,仅需修改集群通信差异部分。
集群环境说明
在这里在一台机器但起3个节点服务的集群。原则上与三台机器各起一个节点服务是一样的。
服务器IP:192.168.237.127
服务器hostname:elkserver
ES文件夹目录:es/es-9201、es/es-9202、es/es-9203
三个服务的es端口号:9201、9202、9203
Kibana启动一个节点即可:es kibana-6601 端口号:6601
安装说明
注意需重新解压一个es并命名为es-9201
1
2
3[root@elkserver es]# tar -zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz [root@elkserver es]# mv elasticsearch-7.11.1 es-9201
注意需重新解压一个kibana并命名为 kibana-6601
1
2
3[root@elkserver es]# tar -zxvf kibana-7.11.1-linux-x86_64.tar.gz [root@elkserver es]# mv kibana-7.11.1-linux-x86_64 kibana-6601
ES的9201配置文件内容如下:
jvm.options
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96################################################################ ## ## JVM configuration ## ################################################################ ## ## WARNING: DO NOT EDIT THIS FILE. If you want to override the ## JVM options in this file, or set any additional options, you ## should create one or more files in the jvm.options.d ## directory containing your adjustments. ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html ## for more information. ## ################################################################ ################################################################ ## IMPORTANT: JVM heap size ################################################################ ## ## The heap size is automatically configured by Elasticsearch ## based on the available memory in your system and the roles ## each node is configured to fulfill. If specifying heap is ## required, it should be done through a file in jvm.options.d, ## and the min and max should be set to the same value. For ## example, to set the heap to 4 GB, create a new file in the ## jvm.options.d directory containing these lines: ## ## -Xms4g ## -Xmx4g ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html ## for more information ## ################################################################ -Xms1g -Xmx1g ################################################################ ## Expert settings ################################################################ ## ## All settings below here are considered expert settings. Do ## not adjust them unless you understand what you are doing. Do ## not edit them in this file; instead, create a new file in the ## jvm.options.d directory containing your adjustments. ## ################################################################ ## GC configuration 8-13:-XX:+UseConcMarkSweepGC 8-13:-XX:CMSInitiatingOccupancyFraction=75 8-13:-XX:+UseCMSInitiatingOccupancyOnly ## G1GC Configuration # NOTE: G1 GC is only supported on JDK version 10 or later # to use G1GC, uncomment the next two lines and update the version on the # following three lines to your version of the JDK # 10-13:-XX:-UseConcMarkSweepGC # 10-13:-XX:-UseCMSInitiatingOccupancyOnly 14-:-XX:+UseG1GC ## JVM temporary directory -Djava.io.tmpdir=${ES_TMPDIR} ## heap dumps # generate a heap dump when an allocation from the Java heap fails; heap dumps # are created in the working directory of the JVM unless an alternative path is # specified -XX:+HeapDumpOnOutOfMemoryError # specify an alternative path for heap dumps; ensure the directory exists and # has sufficient space -XX:HeapDumpPath=data # specify an alternative path for JVM fatal error logs -XX:ErrorFile=logs/hs_err_pid%p.log ## JDK 8 GC logging 8:-XX:+PrintGCDetails 8:-XX:+PrintGCDateStamps 8:-XX:+PrintTenuringDistribution 8:-XX:+PrintGCApplicationStoppedTime 8:-Xloggc:logs/gc.log 8:-XX:+UseGCLogFileRotation 8:-XX:NumberOfGCLogFiles=32 8:-XX:GCLogFileSize=64m # JDK 9+ GC logging 9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
附es-9201节点的elasticsearch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk03 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9201 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9201/elk/data # # Path to log files: # path.logs: /es/es-9201/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # Lock the memory on startup: #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # #防止批量删除索引 action.destructive_requires_name: true # #设置处理器数量4,默认无需设置,单机器多实例需要设置 node.processors: 4 # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9201 transport.port: 9301 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true http.cors.enabled: true http.cors.allow-origin: "*"
- 创建日志路径及数据路径
1
2
3
4
5
6
7
8
9
10[root@elkserver es-9201]# mkdir elk [root@elkserver es-9201]# ls bin config elk jdk lib LICENSE.txt logs modules NOTICE.txt plugins README.asciidoc [root@elkserver es-9201]# cd elk [root@elkserver elk]# mkdir data [root@elkserver elk]# mkdir logs [root@elkserver elk]# ls data logs [root@elkserver elk]# chown -R es:es /es/*
es-9202、es-9203文件夹的生成方法
- 其余两个节点,可用第一个节点文件夹复制的形式,再修改配置。
1
2
3[root@elkserver es]# cp -r es-9201/ es-9202 [root@elkserver es]# cp -r es-9201/ es-9203
- 只需修改两个节点的elasticsearch.yml 以及临时文件和日志文件的名称即可。
附es-9202节点的elasticsearch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk03 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9202 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9202/elk/data # # Path to log files: # path.logs: /es/es-9202/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # Lock the memory on startup: #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # #防止批量删除索引 action.destructive_requires_name: true # #设置处理器数量4,默认无需设置,单机器多实例需要设置 node.processors: 4 # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9202 transport.port: 9302 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true
附es-9203节点的elasticsearch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk03 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: ${HOSTNAME}-9203 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /es/es-9203/elk/data # # Path to log files: # path.logs: /es/es-9203/elk/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # Lock the memory on startup: #内存交换锁定,此处需要操作系统设置才生效 bootstrap.memory_lock: true # #防止批量删除索引 action.destructive_requires_name: true # #设置处理器数量4,默认无需设置,单机器多实例需要设置 node.processors: 4 # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.237.127 # # Set a custom port for HTTP: # http.port: 9203 transport.port: 9303 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true
- 注意:新生成文件夹后需要进行重新授权。chown -R es:es /es/*
启动ES集群环境
- 切换到es账户下 su - es,分别依次启动三个节点的es服务
1
2
3
4[es@elkserver es-9201]$ ./bin/elasticsearch [es@elkserver es-9202]$ ./bin/elasticsearch [es@elkserver es-9203]$ ./bin/elasticsearch
检查三个节点启动是否成功
- 依次在windows主机的浏览器中输入三个ES节点的地址信息进行查看
http://192.168.237.127:9201/
http://192.168.237.127:9202/
http://192.168.237.127:9203/
检查三个节点的健康状态
http://192.168.237.127:9201/_cat/health
结果:1621074718 10:31:58 elk03 green 3 3 53 25 0 0 0 0 - 100.0%
http://192.168.237.127:9202/_cat/health
结果:1621074689 10:31:29 elk03 green 3 3 53 25 0 0 0 0 - 100.0%
http://192.168.237.127:9203/_cat/health
结果:1621074658 10:30:58 elk03 green 3 3 53 25 0 0 0 0 - 100.0%
检查集群是否成功启动
-在任意节点看即可
http://192.168.237.127:9202/_cat/nodes
结果:
1
2
3
4192.168.237.127 39 97 7 0.28 1.19 0.88 cdhilmrstw * elkserver-9201 192.168.237.127 26 97 7 0.28 1.19 0.88 cdhilmrstw - elkserver-9203 192.168.237.127 43 97 6 0.28 1.19 0.88 cdhilmrstw - elkserver-9202
配置Kibana
附6601节点Kibana的kibana.yml配置
- 注elasticsearch.hosts只需配置集群中一个节点即可
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112# Kibana is served by a back end server. This setting specifies the port to use. server.port: 6601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "192.168.237.127" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse proxy. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false # Specifies the public URL at which Kibana is available for end users. If # `server.basePath` is configured this URL should end with the same basePath. #server.publicBaseUrl: "" # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["http://192.168.237.127:9201"] # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. kibana.index: ".kibana-elk03" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. #elasticsearch.username: "kibana_system" #elasticsearch.password: "pass" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files are used to verify the identity of Kibana to Elasticsearch and are required when # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's value to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false # Specifies the path where Kibana creates the process ID file. #pid.file: /run/kibana/kibana.pid # Enables you to specify a file where Kibana stores log output. #logging.dest: stdout # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Specifies locale to be used for all localizable strings, dates and number formats. # Supported languages are the following: English - en , by default , Chinese - zh-CN . #i18n.locale: "en"
启动Kibana环境
1
2[es@elkserver kibana-6601]$ ./bin/kibana
检查Kibana环境启动是否成功
- windows主机浏览器访问:
http://192.168.237.127:6601/
-
启动成功后,windows主机浏览器访问kibana的ip及端口如: http://192.168.237.127:6601
-
点击ADD data
-
点击Sample data 依次添加样例数据(样例数据很重要,后续会使用)
-
进入左上角菜单 Management——>Stack Monitoring
-
点击 Or, set up with self monitoring 使用内部手机设置
-
打开 Turn on monitoring
-
-
如看到es节点变黄,则在kibana中的Dev tools中执行如下即可
复制代码1
2PUT /_settings { "number_of_replicas": 0 }
xshell中关闭 ctrl +c
- 安装部署成功!
最后
以上就是认真薯片最近收集整理的关于ES集群搭建ES集群搭建的全部内容,更多相关ES集群搭建ES集群搭建内容请搜索靠谱客的其他文章。
发表评论 取消回复