ElasticSearch - 备份和迁移

方案

离线方案

• Snapshot
• Reindex
• Logstash
• ElasticSearch-dump
• ElasticSearch-Exporter

增量备份方案

• logstash

使用快照进行备份

配置信息

注册前要注意配置文件加上: elasticsearch.yml

path.repo: ["/opt/elasticsearch/backup"]

创建仓库

注册一个仓库，存放快照，记住，这里不是生成快照，只是注册一个仓库

curl -XPUT 'http://10.11.60.5:9200/_snapshot/repo_backup_1' -H 'Content-Type: application/json' -d '{
	"type": "fs",
	"settings": {
		"location": "/opt/elasticsearch/backup",
		"max_snapshot_bytes_per_sec": "20mb",
		"max_restore_bytes_per_sec": "20mb",
		"compress": true
	}
}'

查看仓库信息:

curl -XGET 'http://10.11.60.5:9200/_snapshot/repo_backup_1?pretty'

返回内容

[root@STOR-ES elasticsearch]# curl -XGET 'http://10.11.60.5:9200/_snapshot/repo_backup_1?pretty'
{
  "repo_backup_1" : {
    "type" : "fs",
    "settings" : {
      "location" : "/opt/elasticsearch/backup",
      "max_restore_bytes_per_sec" : "20mb",
      "compress" : "true",
      "max_snapshot_bytes_per_sec" : "20mb"
    }
  }
}

创建快照

curl -XPUT 'http://10.11.60.5:9200/_snapshot/repo_backup_1/snapshot_1?wait_for_completion=true&pretty' -H 'Content-Type: application/json' -d '{
	"indices": "bro-2019-09-14,bro-2019-09-15,wmi-2019-09-14,wmi-2019-09-15,syslog-2019-09-14,sylog-2019-09-15",
	"rename_pattern": "bro_(.+)",
	"rename_replacement": "dev_bro_$1",
	"ignore_unavailable": true,
	"include_global_state": true
}'

执行

{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "version_id" : 2040399,
    "version" : "2.4.3",
    "indices" : [ "bro-2019-09-14", "bro-2019-09-15", "wmi-2019-09-15", "syslog-2019-09-14", "wmi-2019-09-14" ],
    "state" : "SUCCESS",
    "start_time" : "2019-09-18T05:58:08.860Z",
    "start_time_in_millis" : 1568786288860,
    "end_time" : "2019-09-18T06:02:18.037Z",
    "end_time_in_millis" : 1568786538037,
    "duration_in_millis" : 249177,
    "failures" : [ ],
    "shards" : {
      "total" : 25,
      "failed" : 0,
      "successful" : 25
    }
  }
}

恢复数据

方案使用场景

迁移考虑的问题

• 版本问题，从低版本到高版本数据的迁移
• 多租户的适配问题

多个工厂的数据进入不同index, 原有的数据bro-2019-09-15的数据需要进入factorycode-bro-2019-09-15

• 多次或者分批迁移数据
• 数据在迁移时候富化
• FieldMapping 和 数据信息 分离?