1. 程式人生 > >elasticsearch 數據遷移

elasticsearch 數據遷移

elasticsearch 數據遷移

現需要將某集群下一個索引下的所有數據遷移到另一個集群上, elasticsearch-dump, Elasticsearch-Exporter 試了一下都不好使,只能老實的寫代碼來實現

import os
import sys
import pyes
import datetime

index_list = [
    ["alias-offer", "offer"]
]

ES_URL = "http://ip1:9200/"
NEW_ES_URL = "http://ip2:9200/"

def main():
    for _index, _type in index_list:
        print "開始索引:"+_index
        conn = pyes.es.ES(ES_URL)
        print "已經連接上原地址"
        search = pyes.query.MatchAllQuery().search(bulk_read=1000)
        hits = conn.search(search, _index, _type, scan=True, scroll="30m", model=lambda _,hit: hit)
        print "原地址返回數據"
        conn2 = pyes.es.ES(NEW_ES_URL)
        print "連接到新地址"
        count = 0
        for hit in hits:
            conn2.index(hit[‘_source‘], _index, _type, hit[‘_id‘], bulk=True)
            count += 1
            if count % 1000 == 0:
                print (datetime.datetime.now().strftime(‘%Y-%m-%d %H:%M:%S‘),count)
                conn2.flush_bulk(forced=True)
        conn2.flush_bulk(forced=True)
        conn2 = None
        conn = None

if __name__ == ‘__main__‘:
    main()
    print "數據導入結束"

雖然慢,但能用,基本滿足需要!

elasticsearch 數據遷移