elasticsearch7的二进制部署及索引从6.8迁移至7.17elasticsearch7.17.3的二进制部署将索引从es6迁移至es7

51 阅读 0 评论 34 点赞

我是靠谱客的博主英勇跳跳糖，最近开发中收集的这篇文章主要介绍elasticsearch7的二进制部署及索引从6.8迁移至7.17elasticsearch7.17.3的二进制部署将索引从es6迁移至es7，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

本文介绍es7的二进制部署，及跨es集群的索引迁移。

elasticsearch7.17.3的二进制部署

首先从官网下载es7

[root@VM-20-10-centos ELK]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.3-linux-x86_64.tar.gz
[root@VM-20-10-centos ELK]# ls
elasticsearch-7.17.3-linux-x86_64.tar.gz
[root@VM-20-10-centos ELK]# tar -zxvf elasticsearch-7.17.3-linux-x86_64.tar.gz 
[root@VM-20-10-centos bin]# ./elasticsearch -V
Version: 7.17.3, Build: default/tar/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z, JVM: 18

下载解压以后，修改配置文件，及准备启动脚本

[root@VM-20-10-centos config]# egrep -v "^#|^$" elasticsearch.yml
cluster.name: my-application
node.name: beretxj-node-1
path.data: /data/xj/ELK/es7/data
path.logs: /data/xj/ELK/es7/logs
network.host: 0.0.0.0
http.port: 19200
transport.tcp.port: 19300
discovery.seed_hosts: ["10.0.20.10"]

#启动脚本
[root@VM-20-10-centos system]# cat es7.service 
[Unit]
Description=elasticsearch-7.17.3
After=network.target
[Service]
Type=forking
ExecStart=/data/xj/ELK/elasticsearch-7.17.3/bin/elasticsearch -d
Restart=no
User=elasticsearch
Group=elasticsearch
LimitNOFILE=131070
[Install]
WantedBy=multi-user.target

需要注意的是，elasticsearch不能使用root进行启动，否则会报错，所以可以建elasticsearch或者其他运维账号进行启动，效果都是一样的，同时需要注意，修改一下数据和日志目录的权限。

[root@VM-20-10-centos bin]# grep elasticsearch /etc/passwd
elasticsearch:x:983:977:elasticsearch user:/nonexistent:/sbin/nologin
[root@VM-20-10-centos xj]# chown -R elasticsearch:elasticsearch  ELK
[root@VM-20-10-centos xj]# cd ELK/
[root@VM-20-10-centos ELK]# ls -l
total 304480
drwxr-xr-x 9 elasticsearch elasticsearch      4096 Apr 19  2022 elasticsearch-7.17.3
-rw-r--r-- 1 elasticsearch elasticsearch 311777007 Apr 20  2022 elasticsearch-7.17.3-linux-x86_64.tar.gz
drwxr-xr-x 4 elasticsearch elasticsearch      4096 Jan 10 20:54 es7

做了以上配置以后，启动，会发现程序无法启动，日志没有明显报错，因为elasticsearch默认会使用4g内存，实验环境主机总共才4g内存，明显无法满足，所以需要修改为512m

[root@VM-20-10-centos config]# vim jvm.options
-Xms512m
-Xmx512m

再次启动，可以启动，但访问19200端口，发现集群状态不对

[root@VM-20-10-centos system]# curl localhost:19200
{
  "name" : "beretxj-node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.17.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5ad023604c8d7416c9eb6c0eadb62b14e766caff",
    "build_date" : "2022-04-19T08:11:19.070913226Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

"cluster_uuid" : "_na_",表示集群不可用

同时查看日志发现以下报错org.elasticsearch.discovery.MasterNotDiscoveredException: null

查询资料后发现，是因为es7需要指定主节点才可以正常启动，所以在配置文件增加以下配置

cluster.initial_master_nodes: ["10.0.20.10"]

做以上修改后再启动，es7正常启动

[root@VM-20-10-centos ~]# systemctl start es 7
[root@VM-20-10-centos ~]# systemctl status es7
● es7.service - elasticsearch-7.17.3
   Loaded: loaded (/usr/lib/systemd/system/es7.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-01-10 22:38:15 CST; 1h 19min ago
  Process: 5206 ExecStart=/data/xj/ELK/elasticsearch-7.17.3/bin/elasticsearch -d (code=exited, status=0/SUCCE>
 Main PID: 5398 (java)
    Tasks: 63 (limit: 23722)
   Memory: 867.6M
   CGroup: /system.slice/es7.service
           ├─5398 /data/xj/ELK/elasticsearch-7.17.3/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=6>
           └─5446 /data/xj/ELK/elasticsearch-7.17.3/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jan 10 22:38:09 VM-20-10-centos systemd[1]: Starting elasticsearch-7.17.3...
Jan 10 22:38:15 VM-20-10-centos systemd[1]: Started elasticsearch-7.17.3.
[root@VM-20-10-centos ~]# curl localhost:19200
{
  "name" : "beretxj-node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "JxoEeQSkQKeOaSKT6xj13A",
  "version" : {
    "number" : "7.17.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5ad023604c8d7416c9eb6c0eadb62b14e766caff",
    "build_date" : "2022-04-19T08:11:19.070913226Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@VM-20-10-centos ~]# curl localhost:19200/_cat/nodes
10.0.20.10 60 97 3 0.06 0.02 0.00 cdfhilmrstw * beretxj-node-1

将索引从es6迁移至es7

涉及跨集群迁移时，首先需要在源集群进行配置白名单，然后目标集群才可以从其中读取数据

reindex.remote.whitelist: "*:*"

迁移有2个思路

1，通过POST http://10.0.20.10:19200/_reindex的方式进行迁移，可以使用postman，也可以使用python脚本

2，通过logstash的方式来进行迁移

本文采用第二种方式进行迁移

准备logstash的配置文件

[root@VM-20-10-centos conf.d]# cat logstash_es6_to_es7.conf 
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
    elasticsearch {
        index => "linux-beretxj*"
        hosts => "10.0.20.10:9200"
    }
}

output{
    stdout {}

    elasticsearch {
        hosts => "10.0.20.10:19200"
    }
}

随后启动程序logstash，需要注意的是，如果index使用了通配符，logstash将使用默认索引模板将数据写入es7，且数据都写入一个索引中

 ./logstash -rf ../conf.d/logstash_es6_to_es7.conf  --path.data=/tmp/logstash

使用浏览器插件查看es6的源索引，一共41条记录

再查看es7的索引，可以看到也是41条记录，迁移是成功的

但多个索引数据写入一个索引的效果不是我们希望的，所以我们可以通过指定索引名称的方式来迁移

[root@VM-20-10-centos conf.d]# cat logstash_es6_to_es7.conf 
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
    elasticsearch {
        index => "linux-beretxj-01"
        hosts => "10.0.20.10:9200"
    }
}

output{
    stdout {}

    elasticsearch {
        index => "linux-beretxj-01-new"
        hosts => "10.0.20.10:19200"
    }
}

采用如上方式迁移01和06索引