使用elasticdump遷移數據到新es集群
阿新 • • 發佈：2019-02-12
stream desc 步驟 3.0 lin rev wbs tar sel 參考文檔
https://blog.csdn.net/u012731379/article/details/79856113
https://blog.csdn.net/u010466329/article/details/79209236
https://blog.csdn.net/laoyang360/article/details/65449407 遷移方法
- 通過logstash的input和output配置遷移（配置靈活適用於長期數據同步等）
- 通過遷移工具如elasticdump等（適用於備份一次性小量數據操作）
- 通過elasticsarch自帶快照功能（適用於一次性遷移大量數據）
操作步驟
1、安裝elasticdump
[root@VM_8_24_centos ~]# yum install nodejs npm [root@VM_8_24_centos ~]# npm install elasticdump /root └─┬ [email protected] ├── [email protected] ├─┬ [email protected] │ ├─┬ [email protected] │ │ ├── [email protected] │ │ └── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├─┬ [email protected] │ │ └── [email protected] │ ├── [email protected] │ └─┬ [email protected] │ └── [email protected] ├── [email protected] ├── [email protected] ├── [email protected] ├── [email protected] ├─┬ [email protected] │ ├── [email protected] │ └── [email protected] ├── [email protected] ├── [email protected] ├─┬ [email protected] │ ├── [email protected] │ └── [email protected] ├─┬ [email protected] │ ├── [email protected] │ ├── [email protected] │ ├─┬ [email protected] │ │ └── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├─┬ [email protected] │ │ └── [email protected] │ ├─┬ [email protected] │ │ ├─┬ [email protected] │ │ │ ├── [email protected] │ │ │ ├── [email protected] │ │ │ ├── [email protected] │ │ │ └─┬ [email protected] │ │ │ └── [email protected] │ │ └── [email protected] │ ├─┬ [email protected] │ │ ├── [email protected] │ │ ├─┬ [email protected] │ │ │ ├── [email protected] │ │ │ ├── [email protected] │ │ │ └── [email protected] │ │ └─┬ [email protected] │ │ ├── [email protected] │ │ ├── [email protected] │ │ ├── [email protected] │ │ ├── [email protected] │ │ ├── [email protected] │ │ ├── [email protected] │ │ ├── [email protected] │ │ └── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├─┬ [email protected] │ │ └── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├── [email protected] │ ├─┬ [email protected] │ │ ├── [email protected] │ │ └── [email protected] │ └── [email protected] ├─┬ [email protected] │ └── [email protected] └─┬ [email protected] ├── [email protected] └─┬ [email protected] ├── [email protected] ├── [email protected] ├── [email protected] ├── [email protected] └── [email protected] npm WARN enoent ENOENT: no such file or directory, open ‘/root/package.json‘ npm WARN root No description npm WARN root No repository field. npm WARN root No README data npm WARN root No license field. [root@VM_8_24_centos ~]# cd node_modules/elasticdump/bin
2、導出mapping
[root@VM_8_24_centos bin]# ./elasticdump --input=http://10.2.3.159:9200/reconciliation --output=http://10.2.100.24:9200/reconciliationonline --type=mapping Mon, 11 Feb 2019 09:28:10 GMT | starting dump Mon, 11 Feb 2019 09:28:10 GMT | got 1 objects from source elasticsearch (offset: 0) Mon, 11 Feb 2019 09:28:16 GMT | sent 1 objects to destination elasticsearch, wrote 1 Mon, 11 Feb 2019 09:28:16 GMT | got 0 objects from source elasticsearch (offset: 1) Mon, 11 Feb 2019 09:28:16 GMT | Total Writes: 1 Mon, 11 Feb 2019 09:28:16 GMT | dump complete [root@VM_8_24_centos bin]# curl 10.2.100.24:9200/_cat/indices|grep reconciliation % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 43 32578 43 14037 0 0 5694 0 0:00:05 0:00:02 0:00:03 5694yellow open reconciliationonline KnZrsU-7TmWWBs-1-dBRrw 5 1 0 0 1.1kb 1.1kb
3、導出data
[root@VM_8_24_centos bin]# ./elasticdump --input=http://10.2.3.159:9200/reconciliation --output=http://10.2.100.24:9200/reconciliationonline --type=data
Mon, 11 Feb 2019 09:28:58 GMT | starting dump
Mon, 11 Feb 2019 09:28:58 GMT | got 100 objects from source elasticsearch (offset: 0)
Mon, 11 Feb 2019 09:29:19 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:29:19 GMT | got 100 objects from source elasticsearch (offset: 100)
Mon, 11 Feb 2019 09:29:51 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:29:51 GMT | got 100 objects from source elasticsearch (offset: 200)
Mon, 11 Feb 2019 09:30:13 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:30:13 GMT | got 100 objects from source elasticsearch (offset: 300)
Mon, 11 Feb 2019 09:30:39 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:30:39 GMT | got 100 objects from source elasticsearch (offset: 400)
Mon, 11 Feb 2019 09:31:53 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:31:53 GMT | got 100 objects from source elasticsearch (offset: 500)
Mon, 11 Feb 2019 09:32:28 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:32:28 GMT | got 100 objects from source elasticsearch (offset: 600)
Mon, 11 Feb 2019 09:33:24 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:33:24 GMT | got 100 objects from source elasticsearch (offset: 700)
Mon, 11 Feb 2019 09:34:19 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:34:19 GMT | got 7 objects from source elasticsearch (offset: 800)
Mon, 11 Feb 2019 09:34:47 GMT | sent 7 objects to destination elasticsearch, wrote 7
Mon, 11 Feb 2019 09:34:47 GMT | got 0 objects from source elasticsearch (offset: 807)
Mon, 11 Feb 2019 09:34:47 GMT | Total Writes: 807
Mon, 11 Feb 2019 09:34:47 GMT | dump complete
[root@VM_8_24_centos bin]# curl 10.2.100.24:9200/_cat/indices|grep reconciliationonline
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0yellow open reconciliationonline KnZrsU-7TmWWBs-1-dBRrw 5 1 807 0 4.3mb 4.3mb
100 32578 100 32578 0 0 2010 0 0:00:16 0:00:16 --:--:-- 7769使用elasticdump遷移數據到新es集群