MongoDB 分片叢集部署實驗記錄

阿新 • • 發佈：2021-09-28

======================================================== 環境說明 ========================================================

作業系統：Ubuntu 18.04.5 LTS
MongoDB版本：v4.4.4

======================================================== 角色規劃 ========================================================

主機名     IP              角色
-------------------------------------------------------
mongodb-1  192.168.100.11  config-server,shard-1,mongos
mongodb-2  192.168.100.12  config-server,shard-2,mongos
mongodb-3  192.168.100.13  config-server,shard-3,mongos

======================================================== 埠規劃 ========================================================

config-server: 27019
shard1: 27018
shard2: 27020
mongos: 27017

======================================================== systemd服務檔案規劃 ========================================================

注：這裡使用systemd來管理mongodb的所有服務。

/etc/systemd/system/mongo-config-server.service

[Unit]
Description=MongoDB Database Config Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongodb/config-server.conf
PIDFile=/var/run/mongodb/config-server.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

/etc/systemd/system/[email protected]

[Unit]
Description=MongoDB Database Shard Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target
PartOf=mongo-shard.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongodb/shard%i.conf
PIDFile=/var/run/mongodb/shard%i.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=mongo-shard.target

/etc/systemd/system/mongo-shard.target

[Unit]
Description=mongo-shard target allowing to start/stop all [email protected] instances at once

[Install]
WantedBy=multi-user.target

/etc/systemd/system/mongos.service

[Unit]
Description=MongoDB Database Mongos Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongos --config /etc/mongodb/mongos.conf
PIDFile=/var/run/mongodb/mongos.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongos as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

服務管理命令

# systemctl daemon-reload

# systemctl enable mongo-shard@1
# systemctl enable mongo-shard@2
# systemctl enable mongo-shard.target
# systemctl start mongo-shard.target

# systemctl enable mongo-config-server
# systemctl start mongo-config-server

# systemctl enable mongos
# systemctl start mongos

======================================================== 配置檔案規劃 ========================================================

/etc/mongodb/config-server.conf

storage:
  dbPath: /var/lib/mongodb/config-server
  journal:
    enabled: true
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/config-server.log
net:
  port: 27019
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
security:  # 啟用訪問控制
  authorization: enabled
  keyFile: /var/lib/mongodb/ussmongo.key
replication:
  replSetName: configrs
sharding:
  clusterRole: configsvr

/etc/mongodb/shard1.conf

storage:
  dbPath: /var/lib/mongodb/shard1
  journal:
    enabled: true
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/shard1.log
net:
  port: 27018
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
security:  # 啟用訪問控制
  authorization: enabled
  keyFile: /var/lib/mongodb/ussmongo.key
#replication:  # 生產環境建議啟用副本集，這裡為測試環境
#  replSetName: shardrs1
sharding:
  clusterRole: shardsvr

/etc/mongodb/mongos.conf

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongos.log
net:
  port: 27017
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
security:  # 啟用訪問控制
  keyFile: /var/lib/mongodb/ussmongo.key
sharding:
  configDB: configrs/mongodb-1:27019,mongodb-2:27019

======================================================== 資料和日誌目錄規劃 ========================================================

資料目錄：

/var/lib/mongodb/config-server
/var/lib/mongodb/shard

日誌目錄：

/var/log/mongodb/config-server.log
/var/log/mongodb/shard.log
/var/log/mongodb/mongos.log

建立目錄並賦予許可權：

mkdir -p /var/lib/mongodb/{config-server,shard}
chown -R mongodb:mongodb /var/lib/mongodb/

======================================================== 初始化叢集 ========================================================

在其中一個節點初始化config-server副本集

rs.initiate({_id: "configrs",configsvr: true,members: [{ _id : 0, host : "mongodb-1:27019" },{ _id : 1, host : "mongodb-2:27019" },{ _id : 2, host : "mongodb-3:27019" }]})

在所有shard節點檢視分片例項是否執行（這裡沒有使用副本集）

mongo --port 27018

登入其中一個mongos節點新增所有分片

sh.addShard( "mongodb-1:27018")
sh.addShard( "mongodb-2:27018")
sh.addShard( "mongodb-3:27018")

另附副本集分片新增方法：sh.addShard( "/s1-mongo1.example.net:27018,s1-mongo2.example.net:27018,s1-mongo3.example.net:27018")

資料庫和集合啟用分片方法

sh.enableSharding("<database>")
sh.shardCollection("<database>.<collection>", { <shard key field> : "hashed" } )
sh.shardCollection("<database>.<collection>", { <shard key field> : 1, ... } )

叢集狀態

mongos> db.runCommand({ isdbgrid : 1})  # 檢視當前config-server主伺服器
{
        "isdbgrid" : 1,
        "hostname" : "mongodb-1",
        "ok" : 1,
        "operationTime" : Timestamp(1616050575, 2),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1616050575, 2),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

mongos> use admin  # 列出分片
switched to db admin
mongos> db.runCommand({ listshards : 1})
{
        "shards" : [
                {
                        "_id" : "shard0000",
                        "host" : "mongodb-1:27018",
                        "state" : 1
                },
                {
                        "_id" : "shard0001",
                        "host" : "mongodb-2:27018",
                        "state" : 1
                },
                {
                        "_id" : "shard0002",
                        "host" : "mongodb-3:27018",
                        "state" : 1
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1616050609, 3),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1616050609, 3),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

mongos> use config  # 列出啟用了分片的資料庫
switched to db config
mongos> db.databases.find( { "partitioned": true } )
mongos> db.collections.find()
{ "_id" : "config.system.sessions", "lastmodEpoch" : ObjectId("6052f8ba9666f28a1ccae9da"), "lastmod" : ISODate("1970-02-19T17:02:47.296Z"), "dropped" : false, "key" : { "_id" : 1 }, "unique" : false, "uuid" : UUID("7bdf89db-16cf-4edd-a853-e7cf35ba3eb0"), "distributionMode" : "sharded" }

mongos> db.printShardingStatus()  # 檢視分片叢集詳細資訊
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("6052ee3727ddc4b59c6704aa")
  }
  shards:
        {  "_id" : "shard0000",  "host" : "mongodb-1:27018",  "state" : 1 }
        {  "_id" : "shard0001",  "host" : "mongodb-2:27018",  "state" : 1 }
        {  "_id" : "shard0002",  "host" : "mongodb-3:27018",  "state" : 1 }
  active mongoses:
        "4.4.4" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                23 : Failed with error 'aborted', from shard0000 to shard0002
                29 : Failed with error 'aborted', from shard0000 to shard0001
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       1024
                        too many chunks to print, use verbose if you want to force print
mongos> sh.status()  # 檢視分片叢集詳細資訊
...
mongos> sh.getBalancerState()  # 獲取均衡狀態
true
mongos> sh.isBalancerRunning()  # 均衡是否正在執行？
false

mongos> db.runCommand( { removeShard: "shard0002" } )  # 移除分片

======================================================== 重啟分片叢集 ========================================================

1. 禁用均衡

sh.stopBalancer()

2. 停止分片叢集

2.1 在所有路由節點停止mongos服務

use admin
db.shutdownServer()

2.2 停止每一個分片複製集

在每個分片副本整合員上從管理資料庫執行 db.shutdownServer() 以關閉其 mongod 程序。在關閉每個副本集中的主要成員之前關閉所有次要成員。

2.3 停止配置伺服器

從每個配置伺服器上的管理資料庫執行 db.shutdownServer() 以關閉其 mongod 程序。在關閉主要成員之前關閉所有次要成員。

3. 啟動分片叢集

3.1 啟動配置伺服器

mongod --config <path-to-config-file>
mongod --configsvr --replSet <replica set name> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

啟動所有配置伺服器後，連線到主 mongod 並執行 rs.status() 以確認每個 CSRS 成員的健康狀況和可用性。

3.2 啟動每一個分片副本集

mongod --config <path-to-config-file>
mongod --shardsvr --replSet <replSetname> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

啟動每個分片的所有成員後，連線到每個主 mongod 並執行 rs.status() 以確認每個成員的健康狀況和可用性。

3.3 啟動mongos服務

mongos --config <path-to-config>
mongos --configdb <configReplSetName>/cfg1.example.net:27019,cfg2.example.net:27019 --bind_ip localhost,<hostname(s)|ip address(es)>

4. 重新啟用均衡

sh.startBalancer()
sh.getBalancerState() 
sh.isBalancerRunning()

從 MongoDB 4.2 開始， sh.startBalancer() 還為分片叢集啟用自動拆分。

5. 驗證叢集可訪問性

Connect a mongo shell to one of the cluster’s mongos processes. Use sh.status() to check the overall cluster status.
將mongo shell連線到叢集的mongos程序之一。使用sh.status（）檢查整體群集狀態。

To confirm that all shards are accessible and communicating, insert test data into a temporary sharded collection. Confirm that data is being split and migrated between each shard in your cluster. You can connect a mongo shell to each shard primary and use db.collection.find() to validate that the data was sharded as expected.
要確認所有分片均可以訪問和通訊，請將測試資料插入臨時分片集合中。確認正在叢集中的每個分片之間拆分和遷移資料。您可以將mongo shell連線到每個分片主資料庫，並使用db.collection.find（）來驗證是否按預期對資料進行了分片。

Important:
重要的：
To prevent potential data loss or reading stale data, do not start application reads and writes to the cluster until after confirming the cluster is healthy and accessible.
為防止潛在的資料丟失或讀取過時的資料，請先確認叢集執行狀況良好且可訪問，然後再開始對叢集進行應用程式讀寫。

======================================================== 資料庫和集合啟用分片示例 ========================================================

基於範圍分片

use admin
db.runCommand({ enablesharding: "test" })
db.runCommand({ shardcollection: "test.hu1", key: { id: 1 } })

插入資料指令碼：

#!/usr/bin/python3

import pymongo

myclient = pymongo.MongoClient("mongodb://192.168.100.11:27017/?retryWrites=false")
mydb = myclient["test"]
mycol = mydb["hu1"]

for i in range(10, 1000000):
    mydict = {"id": i, "book": "chukiuxiang", "money": 320}
    x = mycol.insert_one(mydict)
    print(x)

myclient.close()

use test
db.hu1.stats()
db.hu1.count()

基於雜湊分片

use admin
db.runCommand({ enablesharding : "hu" })
db.runCommand({ shardcollection: "hu.hutab", key: { id: "hashed" } })

插入資料指令碼：

#!/usr/bin/python3

import pymongo

myclient = pymongo.MongoClient("mongodb://192.168.100.11:27017/?retryWrites=false")
mydb = myclient["hu"]
mycol = mydb["hutab"]

for i in range(10, 100000):
    mydict = {"id": i, "book": "chukiuxiang", "money": 320}
    x = mycol.insert_one(mydict)
    print(x)

myclient.close()

在shard和mongos節點驗證雜湊分片

root@mongodb-1:~# mongo --port 27018
> use hu
switched to db hu
> db.hutab.count()
33753

root@mongodb-2:~# mongo --port 27018
> use hu
switched to db hu
> db.hutab.count()
33139

root@mongodb-3:~# mongo --port 27018
> use hu
switched to db hu
> db.hutab.count()
33098

root@mongodb-1:~# mongo --port 27017
> use hu
switched to db hu
> db.hutab.count()
99990

======================================================== 啟用訪問控制 ========================================================

1. 新增使用者：

連線上mongos新增的使用者會儲存在config副本集中，但是不會儲存到shard副本集，因此新增使用者的操作需要分別在config、shard上執行。

1.1 config副本集

# mongo --port 27019
> use admin
> db.createUser(
...   {
...     user: "admin",
...     pwd: "admin",
...     roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"]
...   }
... )

1.2 shard副本集

# mongo --port 27018
> use admin
> db.createUser(
...   {
...     user: "admin",
...     pwd: "admin",
...     roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"]
...   }
... )

2. 建立祕鑰檔案

啟用訪問控制之後，外部訪問MongoDB服務需要進行身份驗證，而mongos訪問config和shard服務則是通過配置的祕鑰檔案。

# openssl rand -base64 756 >/var/lib/mongodb/ussmongo.key
# chmod 0600 /var/lib/mongodb/ussmongo.key
# chown mongodb:mongodb /var/lib/mongodb/ussmongo.key

將金鑰檔案複製到所有節點上。

3. 新增security配置

3.1 mongos的配置檔案新增如下配置

security:
  keyFile: /var/lib/mongodb/ussmongo.key

3.2 config和shard的配置檔案新增如下配置

security:
  authorization: enabled
  keyFile: /var/lib/mongodb/ussmongo.key

4. 在所有節點上重啟所有MongoDB服務

至此帶訪問控制的MongoDB高可用叢集就部署完成了。

作者：Varden 出處：http://www.cnblogs.com/varden/ 本文內容如有雷同，請聯絡作者！本文版權歸作者和部落格園共有，歡迎轉載，但未經作者同意必須保留此段宣告，且在文章頁面明顯位置給出原文連線，否則保留追究法律責任的權利。