擴充套件Elasticsearch Azure Plugin支援讀/寫snapshot到多個Azure儲存賬號
忙忙碌碌中2016就要過去，借這篇部落格小小總結一下。像往年一樣每年都是很忙很忙，今年尤其如此，哈哈哈！適逢年末，這一年可以總結的東西有很多，比如：開始讀了這本專業書《Modern Authentication with Azure Active Directory for Web applications》，只可惜還沒有完全讀完，來年仍需要加把勁，多花些時間在讀書，少些時間在手機上！此外，自從搬到28樓後，投身了一項新的群眾體育運動 - 桌球，積極健身好好工作。
預設的Elasticsearch Azure外掛只支援向一個Azure儲存賬號(storage account)寫入/讀出叢集快照(snapshot)資料，索引的快照資料是以 block blob的形式儲存在Azure儲存賬號的blob中的，我在另一篇部落格《
"state": "PARTIAL",
"start_time": "2016-09-21T00:10:09.180Z",
"start_time_in_millis": 1474416609180,
"end_time": "2016-09-21T02:14:36.642Z",
"end_time_in_millis": 1474424076642,
"duration_in_millis": 7467462,"failures": [
{
"node_id": "SVT4jVpiTVmiH8K7ctWrOQ",
"index": "my_index_20160913d",
"reason": "IndexShardSnapshotFailedException[[my_index_20160913d][1] Failed to perform snapshot (index files)]; nested: IOException; nested: StorageException[The server encountered an unknown failure: ]; nested: IOException[Error writing to server]; ",
"shard_id": 1,
"status": "INTERNAL_SERVER_ERROR"
},
...
][2016-07-22 01:27:36,168][WARN ][snapshots ] [ESNode-ElasticSearchData_IN_53] [[myindex.2016_07_10][4]] [snapshot:001008] failed to create snapshot
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [myindex.2016_07_10][4] Failed to perform snapshot (index files)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:509)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:140)
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:85)
at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:871)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:643)
at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:444)
at com.microsoft.azure.storage.blob.BlobOutputStream.access$000(BlobOutputStream.java:53)
at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:388)
at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:385)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)... 3 more
Caused by: com.microsoft.azure.storage.StorageException: The server encountered an unknown failure:
at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:101)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:199)
at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1006)
at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:978)
at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:438)... 9 more
Caused by: java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:666)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:119)
... 12 more
通過擴充套件Elasticsearch Azure Plugin，可以讓它支援讀/寫snapshot資料到多個storage account，從而將Lucene索引資料檔案平均分配到多個storage account中，這樣就避免到了過載一個storage account而導致snapshot失敗。下表總結了將一個包含3777G，185847個Lucene檔案的叢集snapshot寫入到9個Azure儲存賬戶的結果，不難看出，擴充套件的Azure Plugin將快照資料基本上平均（檔案個數和資料大小）寫入到配置的9個賬戶中。基於這個思路，改造了Elasticsearch Azure Plugin支援配置多個storage accounts並讀/寫快照資料。可以在這裡下載到這個擴充套件的Azure Plugin，目前支援的Elasticsearch版本有：
- 1.7.X
- 2.4.1
- 2.4.2
- 2.4.3
- 5.2.1 (包含了Bug#23483修復程式碼)
|
Azure儲存賬號 |
檔案數量 |
檔案大小 |
|
esstorage1 |
20634 |
420 G |
|
esstorage2 |
20597 |
419 G |
|
esstorage3 |
20549 |
418 G |
|
esstorage4 |
20559 |
416 G |
|
esstorage5 |
20637 |
420 G |
|
esstorage6 |
20726 |
420 G |
|
esstorage7 |
20769 |
424 G |
|
esstorage8 |
20697 |
420 G |
|
esstorage9 |
20679 |
417 G |
此外，Elasticsearch支援讀寫snapshot到 型別的storage accounts。通過寫入snapshot到primary location，然後在secondary location恢復資料，可以在不同地域的資料中心之間建立互為備份的Elasticsearch叢集，如下圖所示：
Elasticsearch 1.7.X 版本外掛配置及命令
Elasticsearch.yml
cloud.azure.storage.account: [storageaccount1,storageaccount2,storageaccount3]
cloud.azure.storage.key: [key1, key2, key3]
Commands
#1: define repository
PUT _snapshot/plugintest160921
{
"type": "azure",
"settings": {
"account": "storageaccount1,storageaccount2,storageaccount3",
"container": "plugintest160921"
}
}
#2: take snapshot
PUT _snapshot/plugintest160921/backup0921?wait_for_completion=true
{
}
#3: restore
POST _snapshot/plugintest160921/backup0921/_restore?wait_for_completion=true
{
"ignore_unavailable": "true",
"include_global_state": false
}
#4: define repository for secondary
PUT _snapshot/plugintest160921
{
"type": "azure",
"settings": {
"account": "storageaccount1,storageaccount2,storageaccount3",
"container": "plugintest160921",
"location_mode": "secondary_only"
}
}
Elasticsearch 2.4.X 版本外掛配置及命令
Elasticsearch.yml
cloud.azure.storage.my_account1.account: storageaccount1
cloud.azure.storage.my_account1.key: key1
cloud.azure.storage.my_account1.default: true
cloud.azure.storage.my_account2.account: storageaccount2
cloud.azure.storage.my_account2.key: key2
cloud.azure.storage.my_account2.default: true
cloud.azure.storage.my_account3.account: storageaccount3
cloud.azure.storage.my_account3.key: key3
cloud.azure.storage.my_account3.default: true
Commands
#1: define repository
PUT _snapshot/plugintest160921
{
"type": "azure",
"settings": {
"account": "my_account1,my_account2,my_account3",
"container": "plugintest160921"
}
}
#2: take snapshot
PUT _snapshot/plugintest160921/backup0921?wait_for_completion=true
{
}
#3: restore
POST _snapshot/plugintest160921/backup0921/_restore?wait_for_completion=true
{
"ignore_unavailable": "true",
"include_global_state": false
}
#4: define repository for secondary
PUT _snapshot/plugintest160921
{
"type": "azure",
"settings": {
"account": "my_account1,my_account2,my_account3",
"container": "plugintest160921",
"location_mode": "secondary_only"
}
}
建立了Pull Request#22709給Elasticsearch，希望能夠整合這個功能到Elasticsearch
5.X或者6.X中去。但Elasticsearch團隊對snapshot/restore的未來實現有其他的考慮，所以我的PR未被接受。