1. 程式人生 > >NoSQL之MongoDB副本集和文件管理

NoSQL之MongoDB副本集和文件管理

MongoDB副本集--MongoDB複製

多個伺服器上儲存資料副本,實現資料同步

提高資料高可用性,安全性,方便資料故障恢復

 

原理

  1. 兩個或兩個節點以上,一個主節點,負責處理客戶端請求,其餘為從節點,負責複製主節點資料
  2. 搭配方式:一主一從,一主多從
  3. 主節點記錄所有操作oplog,從節點定期輪詢主節點獲取這些操作,然後對自己的資料副本執行這些操作,保證從與主節點資料一致

實現方式

master-slave主從複製

啟動一臺加上“-master”引數為主節點,其他加上“-slave”和“-source”為從節點

從節點可以提供資料查詢,降低主節點的訪問,並且可以執行備份,避免鎖定主節點資料

主節點故障時,可以快速切換到從節點,實現高可用

replica sets 副本集--從1.6版本開始支援,優於之前

支援故障自動切換,自動修復成員節點,降低運維成本,此結構類似於高可用叢集

配置replica sets

修改配置檔案,指定主機所在副本集名稱,所有主機副本名稱一致

[[email protected] ~]# vim /usr/local/mongodb/etc/mongodb.conf   //52和53同樣操作
logpath=/usr/local/mongodb/log/mongodb.log
logappend=true
dbpath=/usr/local/mongodb/data/db
fork=true
bind_ip=192.168.4.51
port=27051
replSet=rs1                           //指定副本集名稱

啟動主從節點服務,檢視埠號,可以利用pssh同時遠端操控51,52,53

注:可以傳金鑰給要操作的三臺主機,免密登入,並且配置域名解析對應主機名,主機名寫入檔案!

[[email protected] ~]# tail -3  /etc/hosts    //解析三臺主機
192.168.4.51 host1
192.168.4.52 host2
192.168.4.53 host3
[[email protected] ~]# cat host.txt        //寫入host檔案
[email protected]
host2
host3                
[
[email protected]
~]# ssh-keygen -N '' -f /root/.ssh/id_rsa //非互動生成金鑰檔案 [[email protected] ~]# ssh-copy-id host1 //其餘兩臺同樣傳送金鑰 [[email protected] ~]# pssh -i -h host.txt /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/etc/mongodb.conf [1] 09:26:08 [SUCCESS] host2 about to fork child process, waiting until server is ready for connections. forked process: 1958 child process started successfully, parent exiting [2] 09:26:09 [SUCCESS] host3 ... [3] 09:26:09 [SUCCESS] [email protected] ... [[email protected] ~]# pssh -i -h host.txt netstat -pntul | grep 270 tcp 0 0 tcp192.168.4.51:27051 0.0.0.0:* LISTEN 2006/mongod tcp 0 0 192.168.4.53:27053 0.0.0.0:* LISTEN 1963/mongod tcp 0 0 192.168.4.52:27052 0.0.0.0:* LISTEN 1958/mongod

配置節點資訊,在任意一臺操作即可,連結mongod服務,執行如下命令

[[email protected] ~]# /usr/local/mongodb/bin/mongo --host 192.168.4.51 --port 27051
> config = { _id:"rs1",members:[{_id:0,host:"192.168.4.51:27051"},{_id:1,host:"192.168.4.52:27052"},{_id:2,host:"192.168.4.53:27053"}]}   //顯示如下資訊
{
	"_id" : "rs1",
	"members" : [
		{
			"_id" : 0,
			"host" : "192.168.4.51:27051"
		},
		{
			"_id" : 1,
			"host" : "192.168.4.52:27052"
		},
		{
			"_id" : 2,
			"host" : "192.168.4.53:27053"
		}
	]
}
> rs.help()                                                        //可以檢視幫助資訊

出錯解決辦法

進入配置檔案 註釋掉副本集

關閉服務,重新啟動

進入mongod服務,刪除local庫 

然後把註釋去掉,重新啟動服務,重新配置

初始化副本集 rs.initiate(config)

> rs.initiate(config)
{
	"ok" : 1,                         //顯示為1,成功
	"operationTime" : Timestamp(1545789902, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1545789902, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}
rs1:SECONDARY>                            //回車
rs1:PRIMARY> show dbs                     //當前主機為主庫,主庫可以檢視資料,從庫不可以
admin   0.000GB
config  0.000GB
local   0.000GB
rs1:PRIMARY> use local     
switched to db local
rs1:PRIMARY> show tables            
me                
oplog.rs
replset.election
replset.minvalid
startup_log
system.replset
system.rollback.id

檢視副本集資訊(在51上為例)

rs1:PRIMARY> rs.status()                    //檢視狀態資訊
{
"set" : "rs1",
	"date" : ISODate("2018-12-26T02:29:25.735Z"),
	"myState" : 1,
...
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "192.168.4.51:27051",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
...
},
		{
			"_id" : 1,
			"name" : "192.168.4.52:27052",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
...
	},
		{
			"_id" : 2,
			"name" : "192.168.4.53:27053",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
...
rs1:PRIMARY> rs.isMaster()                   //檢視是否為主庫
{
	"hosts" : [
		"192.168.4.51:27051",
		"192.168.4.52:27052",
		"192.168.4.53:27053"
	],
	"setName" : "rs1",
	"setVersion" : 1,
	"ismaster" : true,
	"secondary" : false,
	"primary" : "192.168.4.51:27051",
	"me" : "192.168.4.51:27051",
...

測試資料同步  客戶端連結主庫,儲存文件,從庫驗證(需開啟從庫同步資料,db.getMongo().setSlaveOk() )

[[email protected] ~]# /usr/local/mongondb/bin/mongo --host 192.168.4.51 --port 27051
rs1:PRIMARY> db.test.save({name:"bob",sex:"m"})          //寫入資料
WriteResult({ "nInserted" : 1 })
rs1:PRIMARY> show tables
test
rs1:PRIMARY> db.test.find()
{ "_id" : ObjectId("5c22f411ba77fd8902a24eb9"), "name" : "bob", "sex" : "m" }
rs1:SECONDARY> db.getMongo().setSlaveOk()       //在從庫上驗證
rs1:SECONDARY> use test
switched to db test
rs1:SECONDARY> show tables
test
rs1:SECONDARY> db.test.find()
{ "_id" : ObjectId("5c22f411ba77fd8902a24eb9"), "name" : "bob", "sex" : "m" }

測試高可用

停止當前的主庫51,會在52和53從庫中選取新的主庫,未被選取為主庫的主機自動做新主庫的從庫,客戶端連結新主庫儲存文件

rs1:PRIMARY> rs.isMaster()              //在53上檢視,發現為主庫
{
	"hosts" : [
		"192.168.4.51:27051",
		"192.168.4.52:27052",
		"192.168.4.53:27053"
	],
	"setName" : "rs1",
	"setVersion" : 1,
	"ismaster" : true,
	"secondary" : false,
	"primary" : "192.168.4.53:27053",
	"me" : "192.168.4.53:27053",
...
rs1:PRIMARY> db.test.save({name:"rose"})         //在53上寫入資料
WriteResult({ "nInserted" : 1 })
rs1:PRIMARY>                         

rs1:SECONDARY> rs.isMaster()               //在52上檢視,發現當前為從庫
{
	"hosts" : [
		"192.168.4.51:27051",
		"192.168.4.52:27052",
		"192.168.4.53:27053"
	],
	"setName" : "rs1",
	"setVersion" : 1,
	"ismaster" : false,
	"secondary" : true,
	"primary" : "192.168.4.53:27053",
	"me" : "192.168.4.52:27052",
rs1:SECONDARY> db.test.find()
{ "_id" : ObjectId("5c22f411ba77fd8902a24eb9"), "name" : "bob", "sex" : "m" }
{ "_id" : ObjectId("5c22fcebec89744f346d253f"), "name" : "rose" }

[[email protected] ~]# /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/etc/mongodb.conf                //重啟51 連結資料庫
[[email protected] ~]# /usr/local/mongodb/bin/mongo --host 192.168.4.51 --port 27051
rs1:SECONDARY> rs.isMaster()
{
	"hosts" : [
		"192.168.4.51:27051",
		"192.168.4.52:27052",
		"192.168.4.53:27053"
	],
	"setName" : "rs1",
	"setVersion" : 1,
	"ismaster" : false,
	"secondary" : true,
	"primary" : "192.168.4.53:27053",
	"me" : "192.168.4.51:27051",
...
rs1:SECONDARY> rs.slaveOk( )
rs1:SECONDARY> db.test.find()                 //發現已經把離線的資料恢復,並自動成為從庫
{ "_id" : ObjectId("5c22f411ba77fd8902a24eb9"), "name" : "bob", "sex" : "m" }
{ "_id" : ObjectId("5c22fcebec89744f346d253f"), "name" : "rose" }

恢復單獨的mongodb服務

停止服務,註釋掉配置檔案中的副本集名稱,重啟服務,刪除local庫,再次重啟!(51上驗證,52/53同操作)

[[email protected] ~]# /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/etc/mongodb.conf   --shutdown
[[email protected] ~]# vim /usr/local/mongodb/etc/mongodb.conf
logpath=/usr/local/mongodb/log/mongodb.log
logappend=true
dbpath=/usr/local/mongodb/data/db
fork=true
bind_ip=192.168.4.51
port=27051
#replSet=rs1                 //註釋掉
[[email protected] ~]#/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/etc/mongodb.conf 
[[email protected] ~]#/usr/local/mongodb/bin/mongo --host 192.168.4.51 --port 27051 
> use local
switched to db local
> db.dropDatabase()
{ "dropped" : "local", "ok" : 1 }
[[email protected] ~]# /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/etc/mongodb.conf   --shutdown
[[email protected] ~]# /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/etc/mongodb.conf
[[email protected]~]# /usr/local/mongodb/bin/mongo --host 192.168.4.51 --port 27051 
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
test    0.000GB

文件管理

  插入

  文件

db.集合名.save({key:"value",key:"value"}) 集合不存在則建立,寫入記錄,_id欄位值存在,修改文件欄位值
db.集合名.insert({key:"value",key:"value"}) 集合不存在建立,寫入,_id欄位值存在,放棄插入,不存在,寫入
db.集合名.insertMany([{key:"value"},{key:"value"}]) 插入多條記錄

  查詢

  文件

db.集合名.find().[count()] 顯示所有行,預設20,輸入it可顯示後面的 count()統計總數
db.集合名.findOne() 顯示第一行

db.集合名.find({條件},{定義顯示的欄位})

db.集合名.find({key:"值",keyname:"值"})

指定查詢條件並指定顯示的欄位,欄位:0 不顯示,欄位:1顯示

不寫條件,則顯示符合定義欄位的所有行

    行數

    顯示

    控制

db.集合名.find().limit(num) 顯示前幾行
db.集合名.find().skip(num) 跳過前幾行
db.集合名.find().sort(欄位名:1/-1) 1升序顯示,-1降序顯示

   範圍

   比較

db.集合名.find({key:{$in:["值1",”值2“,”值3“]}}) 在...裡
db.集合名.find({key:{$nin:["值1",”值2“,”值3“]}}) 不在...裡
db.集合名.find({$or:[{key:"值1"},{keyname:"值2"}]})
正則匹配 db.集合名.find({key:/ 正則規則/}) 正則匹配
數值比較

$lt   $lte   $gt   $gte  $ne

<  <=  >   >=    !=
  db.集合名.find({key:null}) 匹配null,也可以匹配沒有的欄位

 更新 

 文件

db.集合名.update({條件},{修改的欄位}) 預設只修改匹配檔案的第一行,且會刪除本行中的其餘欄位,只留下修改的欄位
db.集合名.update({條件},{$set:{修改的欄位}},false,true) 預設只更新與條件匹配的第一行,加上false,true可以修改全部與條件匹配的行
db.集合名.update({條件},{$unset:{key:values}}) 刪除符合條件的欄位
db.集合名.update({條件},{$inc:{欄位名:+/-num}}) 調價匹配時,欄位值自加或者自減
db.集合名.update({條件},{$push:{陣列名:"值"}}) 向陣列中新增新元素
db.集合名.update({條件},{$addToSet:{陣列:"值"}}) 避免重複新增
db.集合名.update({條件},{$pop:{陣列名:數字}}) 從陣列頭/尾部刪除一個元素,1代表尾部,也就是陣列列最後列一個,-1 代表從頭部!
db.集合名.update({條件},{$pull:{陣列名:"值"}}) 刪除陣列指定元素
刪除文件 db.集合名.drop() 刪除集合的同時刪除索引
db.集合名.remove({})/({條件}) 刪除文件的同時不刪除索引,刪除所有/符合條件的

 插入文件演示

> db.t1.save({_id:1,name:"bob"})
WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : 1 })

> db.t1.insert({_id:2,name:"tom"})
WriteResult({ "nInserted" : 1 })

> db.t1.find()
{ "_id" : 1, "name" : "bob" }
{ "_id" : 2, "name" : "tom" }

> db.t1.save({_id:2,name:"lucy"})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

> db.t1.find()
{ "_id" : 1, "name" : "bob" }
{ "_id" : 2, "name" : "lucy" }

 db.t1.insert({_id:2,name:"jack"})
WriteResult({
	"nInserted" : 0,
	"writeError" : { ....}
> db.t1.find()
{ "_id" : 1, "name" : "bob" }
{ "_id" : 2, "name" : "lucy" }

> db.t1.insertMany([{_id:3,name:"rose"},{_id:2,name:"jack"}])
> db.t1.find()
{ "_id" : 1, "name" : "bob" }
{ "_id" : 2, "name" : "lucy" }
{ "_id" : 3, "name" : "rose" }
> db.t1.insertMany([{_id:4,name:"jack"},{_id:5,name:"kate"}])
{ "acknowledged" : true, "insertedIds" : [ 4, 5 ] }
> db.t1.find()
{ "_id" : 1, "name" : "bob" }
{ "_id" : 2, "name" : "lucy" }
{ "_id" : 3, "name" : "rose" }
{ "_id" : 4, "name" : "jack" }
{ "_id" : 5, "name" : "kate" }

查詢文件演示

> use userdb
switched to db userdb
> show tables
user
> db.user.find()
{ "_id" : ObjectId("5c21f814af1ef5d24b56fcd5"), "name" : "root", "pass" : "x", "uid" : 0, "gid" : 0, "comment" : "root", "hmdir" : "/root", "shell" : "/bin/bash" }
...
Type "it" for more

> db.user.findOne()
{
	"_id" : ObjectId("5c21f814af1ef5d24b56fcd5"),
	"name" : "root",
	"pass" : "x",
	"uid" : 0,
	"gid" : 0,
	"comment" : "root",
	"hmdir" : "/root",
	"shell" : "/bin/bash"
}

> db.user.find().pretty()                             //加上pretty() 可以豎行顯示
{
	"_id" : ObjectId("5c21f814af1ef5d24b56fcd5"),
	"name" : "root",
	"pass" : "x",
	"uid" : 0,
...
	"comment" : "Saslauthd user",
	"hmdir" : "/run/saslauthd",
	"shell" : "/sbin/nologin"
}
Type "it" for more

> db.user.find({},{_id:0,name:1,shell:1})
{ "name" : "root", "shell" : "/bin/bash" }
{ "name" : "bin", "shell" : "/sbin/nologin" }
...
{ "name" : "colord", "shell" : "/sbin/nologin" }
{ "name" : "saslauth", "shell" : "/sbin/nologin" }
Type "it" for more

> db.user.find().count()
41

> db.user.find({shell:"/sbin/nologin"},{_id:0,name:1,shell:1}).limit(3)
{ "name" : "bin", "shell" : "/sbin/nologin" }
{ "name" : "daemon", "shell" : "/sbin/nologin" }
{ "name" : "adm", "shell" : "/sbin/nologin" }
> db.user.find({shell:"/sbin/nologin"},{_id:0,name:1,shell:1}).skip(2)
{ "name" : "adm", "shell" : "/sbin/nologin" }
{ "name" : "lp", "shell" : "/sbin/nologin" }
{ "name" : "mail", "shell" : "/sbin/nologin" }
...
> db.user.find({shell:"/sbin/nologin"},{_id:0,uid:1,shell:1}).sort({uid:1}).limit(10)
{ "uid" : 1, "shell" : "/sbin/nologin" }
{ "uid" : 2, "shell" : "/sbin/nologin" }
{ "uid" : 3, "shell" : "/sbin/nologin" }
{ "uid" : 4, "shell" : "/sbin/nologin" }
{ "uid" : 8, "shell" : "/sbin/nologin" }
{ "uid" : 11, "shell" : "/sbin/nologin" }
{ "uid" : 12, "shell" : "/sbin/nologin" }
{ "uid" : 14, "shell" : "/sbin/nologin" }
{ "uid" : 29, "shell" : "/sbin/nologin" }
{ "uid" : 32, "shell" : "/sbin/nologin" }
> db.user.find({shell:"/sbin/nologin"},{_id:0,uid:1,shell:1}).sort({uid:-1}).limit(10)
{ "uid" : 65534, "shell" : "/sbin/nologin" }
{ "uid" : 999, "shell" : "/sbin/nologin" }
{ "uid" : 998, "shell" : "/sbin/nologin" }
{ "uid" : 997, "shell" : "/sbin/nologin" }
{ "uid" : 996, "shell" : "/sbin/nologin" }
{ "uid" : 995, "shell" : "/sbin/nologin" }
{ "uid" : 994, "shell" : "/sbin/nologin" }
{ "uid" : 993, "shell" : "/sbin/nologin" }
{ "uid" : 992, "shell" : "/sbin/nologin" }
{ "uid" : 192, "shell" : "/sbin/nologin" }

> db.user.find({name:"root",uid:0},{_id:0,name:1,shell:1})
{ "name" : "root", "shell" : "/bin/bash" }
> db.user.find({name:{$in:["root","adm","nobody"]}},{_id:0,name:1,shell:1})
{ "name" : "root", "shell" : "/bin/bash" }
{ "name" : "adm", "shell" : "/sbin/nologin" }
{ "name" : "nobody", "shell" : "/sbin/nologin" }

> db.user.find({shell:{$nin:["/bin/bash","/sbin/nologin"]}},{_id:0,name:1,shell:1})
{ "name" : "sync", "shell" : "/bin/sync" }
{ "name" : "shutdown", "shell" : "/sbin/shutdown" }
{ "name" : "halt", "shell" : "/sbin/halt" }
{ "name" : "mysql", "shell" : "/bin/false" }

> db.user.find({$or:[{name:"root"},{uid:99}]},{_id:0,name:1,uid:1})
{ "name" : "root", "uid" : 0 }
{ "name" : "nobody", "uid" : 99 }


> db.user.find({uid:{$gte:10,$lte:20}},{_id:0,name:1,uid:1})
{ "name" : "operator", "uid" : 11 }
{ "name" : "games", "uid" : 12 }
{ "name" : "ftp", "uid" : 14 }

> db.user.find({name:/^a/},{_id:0,name:1,uid:1})
{ "name" : "adm", "uid" : 3 }
{ "name" : "abrt", "uid" : 173 }
{ "name" : "avahi", "uid" : 70 }

> db.user.find({$or:[{uid:{$lt:10}},{uid:{$gt:70}}]},{_id:0,name:1,uid:1}).sort({uid:1})
{ "name" : "root", "uid" : 0 }
{ "name" : "bin", "uid" : 1 }
{ "name" : "daemon", "uid" : 2 }
{ "name" : "adm", "uid" : 3 }
{ "name" : "lp", "uid" : 4 }
{ "name" : "sync", "uid" : 5 }
{ "name" : "shutdown", "uid" : 6 }
{ "name" : "halt", "uid" : 7 }
{ "name" : "mail", "uid" : 8 }
{ "name" : "tcpdump", "uid" : 72 }
{ "name" : "sshd", "uid" : 74 }
{ "name" : "radvd", "uid" : 75 }
{ "name" : "dbus", "uid" : 81 }
{ "name" : "postfix", "uid" : 89 }
{ "name" : "nobody", "uid" : 99 }
{ "name" : "qemu", "uid" : 107 }
{ "name" : "usbmuxd", "uid" : 113 }
{ "name" : "pulse", "uid" : 171 }
{ "name" : "rtkit", "uid" : 172 }
{ "name" : "abrt", "uid" : 173 }

更新文件

> db.user.find({uid:{$lte:4}},{_id:0})
{ "name" : "root", "pass" : "x", "uid" : 0, "gid" : 0, "comment" : "root", "hmdir" : "/root", "shell" : "/bin/bash" }
...
{ "name" : "lp", "pass" : "x", "uid" : 4, "gid" : 7, "comment" : "lp", "hmdir" : "/var/spool/lpd", "shell" : "/sbin/nologin" }

> db.user.update({uid:{$lte:4}},{pass:666})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.user.find({uid:{$lte:4}},{_id:0})
{ "name" : "bin", "pass" : "x", "uid" : 1, "gid" : 1, "comment" : "bin", "hmdir" : "/bin", "shell" : "/sbin/nologin" }
...
{ "name" : "lp", "pass" : "x", "uid" : 4, "gid" : 7, "comment" : "lp", "hmdir" : "/var/spool/lpd", "shell" : "/sbin/nologin" }

> db.user.find({pass:666})
{ "_id" : ObjectId("5c21f814af1ef5d24b56fcd5"), "pass" : 666 }
> db.user.find({uid:{$lte:4}},{_id:0}).

> db.user.update({name:"bin"},{$set:{pass:"666"}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.user.find({name:"bin"},{_id:0})
{ "name" : "bin", "pass" : "666", "uid" : 1, "gid" : 1, "comment" : "bin", "hmdir" : "/bin", "shell" : "/sbin/nologin" }

> db.user.update({uid:{$lte:4}},{$unset:{pass:"x"}},false,true)
WriteResult({ "nMatched" : 4, "nUpserted" : 0, "nModified" : 3 })
> db.user.find({uid:{$lte:4}},{_id:0})
{ "name" : "bin", "uid" : 1, "gid" : 1, "comment" : "bin", "hmdir" : "/bin", "shell" : "/sbin/nologin" }
{ "name" : "daemon", "uid" : 2, "gid" : 2, "comment" : "daemon", "hmdir" : "/sbin", "shell" : "/sbin/nologin" }
{ "name" : "adm", "uid" : 3, "gid" : 4, "comment" : "adm", "hmdir" : "/var/adm", "shell" : "/sbin/nologin" }
{ "name" : "lp", "uid" : 4, "gid" : 7, "comment" : "lp", "hmdir" : "/var/spool/lpd", "shell" : "/sbin/nologin" }

刪除文件

> db.user.remove({name:"bin"})
WriteResult({ "nRemoved" : 1 })
> db.user.find({uid:{$lt:10}},{_id:0,name:1})       //可以發現已經沒有bin使用者了,bin的uid前面可以到是1
{ "name" : "daemon" }
{ "name" : "adm" }
{ "name" : "lp" }
{ "name" : "sync" }
{ "name" : "shutdown" }
{ "name" : "halt" }
{ "name" : "mail" }