pg_probackup備份與恢復
阿新 • • 發佈：2020-12-26
技術標籤：pg
一、pg_probackup概述
pg_probackup 是一款免費的postgres資料庫叢集備份工具，與其他備份工具相比，它主要有如下一些優勢：
- 提供增量備份，增量備份一定程度上可以節省磁碟空間的使用並且減少備份時間消耗
- 可通過全量備份+增量備份進行增量恢復
- 無需通過實際的資料恢復操作驗證備份檔案是否有效
- Verification: on-demand verification of Postgres Pro instance with the checkdb command.
- 可以通過設定恢復時間以及備份最大檔案數來進行備份檔案以及WAL歸檔日誌的保留策略
- 對於backup、restore、merge、delete、validate、checkdb操作都可開啟併發執行緒執行
- 提供壓縮備份以節省磁碟空間
- 可對遠端例項進行備份恢復
- 可從standby例項進行備份
- 對於PGDATA外的目錄資料（如：指令碼、日誌轉儲、sql dump 檔案等），可使用引數額外指定進行備份
- 可檢視已備份資料備份以及歸檔的列表以及相關詳細資訊
- 支援部分還原（還原部分資料庫）
pg_probackup 下的幾種備份方式：
-
全量備份 ： 全量備份會將資料庫集下所有的資料檔案進行備份
-
增量備份 ： 增量備份僅會備份上一次全量備份之後發生變更的資料，相對於全量備份其空間佔用有了極大的縮減
DELTA模式 ： 在該模式下， pg_probackup 會掃描所有的資料目錄檔案，然後將上一次備份後發生改變的資料頁進行拷貝備份。這種模式下增量備份的IO消耗基本等同於全量備份。
PAGE模式 ： 在該模式下， pg_probackup僅會掃描備份上一次備份結束時刻之後的所有WAL歸檔日誌。這種模式下的增量備份必須保證wal日誌有設定合理的歸檔（wal_level > minimal 、archive_mode = on/always、archive_command 使用 pg_probackup進行archive-push 歸檔）。
PTRACK模式 ： 在該模式下，pg_probackup 會實時跟蹤源備份例項端資料頁的變化，對於距上一次備份後發生更新的資料頁，將其記錄在 bitmap 中，以此來加快增量備份的時間。該模式下不需要關注WAL日誌歸檔的設定，增量備份時間相對於DELTA更快，但是由於需要實時跟蹤發生變化的資料頁，所以對源端資料庫伺服器是有一定的資源消耗的。
pg_probackup 工具的一些侷限性：
- 僅支援Postgres Pro 9.5以上的版本
- Windows系統不支援遠端備份恢復
- 在Unix系統在，如果資料庫版本小於等於 Postgres Pro 10 ，只能通過與OS同賬號的超級使用者postgres進行備份。
- 對於PostgreSQL 9.5版本資料庫，進行備份的資料庫賬號必須具體superuser的角色，否則無法備份 pg_create_restore_point(text) 、 pg_switch_xlog()
- 備份工具與資料庫 block_size 、 wal_block_size 必須一致，否則無法備份（block_size、wal_block_size在安裝包原始碼編譯時設定）
二、安裝部署
2.1 原始碼安裝
1、下載安裝包
# wget -c https://github.com/postgrespro/pg_probackup/archive/2.4.2.tar.gz
# tar xf 2.4.2.tar.gz
# cd pg_probackup-2.4.2/
# ll
總用量 176
drwxrwxr-x 2 root root 4096 7月 1 08:07 doc
-rw-rw-r-- 1 root root 128060 7月 1 08:07 Documentation.md
-rw-rw-r-- 1 root root 4976 7月 1 08:07 gen_probackup_project.pl
-rw-rw-r-- 1 root root 1200 7月 1 08:07 LICENSE
-rw-rw-r-- 1 root root 3962 7月 1 08:07 Makefile
-rw-rw-r-- 1 root root 13345 7月 1 08:07 README.md
drwxrwxr-x 3 root root 4096 7月 1 08:07 src
drwxrwxr-x 4 root root 4096 7月 1 08:07 tests
drwxrwxr-x 2 root root 4096 7月 1 08:07 travis
2、編譯安裝
# PG_CONFIG是我們pg_config程式所在路徑，top_srcdir為postgres原始碼所在路徑
# make USE_PGXS=1 PG_CONFIG=/usr/local/pgsql/bin/pg_config top_srcdir=/usr/local/postgresql-12.2
# ll
總用量 612
drwxrwxr-x 2 root root 4096 7月 1 08:07 doc
-rw-rw-r-- 1 root root 128060 7月 1 08:07 Documentation.md
-rw-rw-r-- 1 root root 4976 7月 1 08:07 gen_probackup_project.pl
-rw-rw-r-- 1 root root 1200 7月 1 08:07 LICENSE
-rw-rw-r-- 1 root root 3962 7月 1 08:07 Makefile
-rwxr-xr-x 1 root root 445832 9月 22 21:55 pg_probackup //編譯後
-rw-rw-r-- 1 root root 13345 7月 1 08:07 README.md
drwxrwxr-x 3 root root 4096 9月 22 21:55 src
drwxrwxr-x 4 root root 4096 7月 1 08:07 tests
drwxrwxr-x 2 root root 4096 7月 1 08:07 travis
3、版本檢查
# 檢視版本
# ./pg_probackup --version
pg_probackup 2.4.2 (PostgreSQL 12.2)
2.2 rpm包安裝部署
三、常用命令
- 初始化備份目錄
$ pg_probackup init -B ${backup_dir}
- 新增例項
## 本地例項
pg_probackup add-instance -B ${backup_dir} -D ${PGDATA} --instance ${instance_name}
## 新增遠端例項
pg_probackup add-instance -B ${backup_dir} -D ${PGDATA} --instance ${instance_name} --remote-prot=ssh --remote-host=${remote_ip} --remote-port=${remote_ssh_port} --remote-user=${remote_ssh_user} --remote-path=${pg_probackup_dir}
- 備份
## 本地例項全量備份
pg_probackup backup -B ${backup_dir} --instance ${instance_name} -b full
## 遠端例項全量備份
pg_probackup backup -B ${backup_dir} --instance ${instance_name} --remote-user=${remote_ssh_user} --remote-host=${remote_ip} --remote-port=${remote_ssh_port} -b full
## 增量備份
pg_probackup backup -B -B ${backup_dir} --instance ${instance_name} -b page|detla|ptrack
- 恢復
## 根據備份集恢復
pg_probackup restore -B ${backup_dir} --instance ${instance_name} -i ${backup_id}
## 不完整恢復，恢復部分database
pg_probackup restore -B ${backup_dir} --instance ${instance_name} --db-include=${database_name1} --db-include=${database_name2}
## 按時間點恢復
pg_probackup restore -B ${backup_dir} --instance ${instance_name} --recovery-target-time='2020-09-22 22:49:34'
pg_probackup restore -B ${backup_dir} --instance ${instance_name} --recovery-target-xid='687'
pg_probackup restore -B ${backup_dir} --instance ${instance_name} --recovery-target-lsn='16/B374D848'
pg_probackup restore -B ${backup_dir} --instance ${instance_name} --recovery-target-name='before_app_upgrade'
pg_probackup restore -B ${backup_dir} --instance ${instance_name} --recovery-target='latest'
pg_probackup restore -B ${backup_dir} --instance ${instance_name} -recovery-target='immediate'
- 檢視備份檔案可用性
pg_probackup show -B ${backup_dir} --instance ${instance_name} -i ${backup_id}
- 檢視備份詳情
pg_probackup show -B ${backup_dir} --instance ${instance_name} -i ${backup_id}
#Configuration
backup-mode = FULL //備份模式
stream = false //是否啟用stream
compress-alg = none
compress-level = 1 //壓縮等級
from-replica = false
#Compatibility
block-size = 8192 //blocksize
xlog-block-size = 8192
checksum-version = 0
program-version = 2.4.2
server-version = 12 //PG Version
#Result backup info
timelineid = 1
start-lsn = 0/A000028
stop-lsn = 0/B0000B8
start-time = '2020-09-22 22:49:33+08'
end-time = '2020-09-22 22:49:39+08'
recovery-xid = 658
recovery-time = '2020-09-22 22:49:34+08'
data-bytes = 41423956
wal-bytes = 16777216
uncompressed-bytes = 41389959
pgdata-bytes = 41389720
status = OK
primary_conninfo = 'user=postgres port=5432 sslmode=disable sslcompression=0 gssencmode=disable krbsrvname=postgres target_session_attrs=any'
content-crc = 1486842437
- 檢視歸檔詳情
pg_probackup show -B ${backup_dir} --instance ${instance_name} --archive
- 配置 Retention Policy
–retention-redundancy=redundancy : 保留備份多少天 FULL
–retention-window=window : 可恢復多少天之前備份
pg_probackup set-config -B ${backup_dir} --instance ${instance_name} --retention-redundancy 7 --retention-window 7
$ /usr/local/pg_probackup-2.4.2/pg_probackup show-config -B /data/pgdata_probackup/ --instance local_5432
# Backup instance information
pgdata = /data/pgsql12/data
system-identifier = 6870373621203487994
xlog-seg-size = 16777216
# Connection parameters
pgdatabase = postgres
# Replica parameters
replica-timeout = 5min
# Archive parameters
archive-timeout = 5min
# Logging parameters
log-level-console = INFO
log-level-file = OFF
log-filename = pg_probackup.log
log-rotation-size = 0TB
log-rotation-age = 0d
# Retention parameters
retention-redundancy = 7
retention-window = 7
wal-depth = 0
# Compression parameters
compress-algorithm = none
compress-level = 1
# Remote access parameters
remote-proto = ssh
- 刪除過期資料
pg_probackup delete -B ${backup_dir} --instance ${instance_name} --delete-expired
--同時刪除過期WAL
pg_probackup delete -B ${backup_dir} --instance ${instance_name} --delete-expired --delete-wal
--使用新策略覆蓋當前策略刪除
pg_probackup delete -B ${backup_dir} --instance ${instance_name} --delete-expired --delete-wal --retention-window=1 --retention-redundancy=1
四、工具使用
4.1 初始化配置
1、初始化備份目錄
## 初始化備份目錄
$ /usr/local/pg_probackup-2.4.2/pg_probackup init -B /data/pgdata_probackup
INFO: Backup catalog '/data/pgdata_probackup' successfully inited
## 初始化其實就是在該目錄下建立一個backups目錄一個wal目錄
$ pwd
/data/pgdata_probackup
$ tree
.
├── backups
└── wal
2 directories, 0 files
2、一些必要的檢查
1）本地備份
資料庫 pg_hba.conf 配置檔案中對 connection 和 replication 的放通
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local replication all trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
2）遠端備份
互信配置
## 在遠端備份例項主機上
# su - postgres
$ ssh-keygen
$ ssh-copy-id [email protected]${備份機_ip}
## 在備份機上
# su - postgres
$ ssh-keygen
$ ssh-copy-id [email protected]${備份例項主機_ip}
## 測試互信
$ ssh [email protected]${對方IP}
資料庫 pg_hba.conf 配置檔案中對 connection 和 replication 的放通
local all all md5
host all all xxx(備份機) trust
3）備份資料庫賬號準備
若使用的資料庫賬號不是postgres超級賬號，如果使用資料庫賬號backup進行備份工作，那麼需要授予以下許可權：
## For PostgreSQL 9.5:
BEGIN;
CREATE ROLE backup WITH LOGIN;
GRANT USAGE ON SCHEMA pg_catalog TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.current_setting(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_is_in_recovery() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_start_backup(text, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_stop_backup() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_create_restore_point(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_switch_xlog() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current_snapshot() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_snapshot_xmax(txid_snapshot) TO backup;
COMMIT;
## For Postgres Pro 9.6:
BEGIN;
CREATE ROLE backup WITH LOGIN;
GRANT USAGE ON SCHEMA pg_catalog TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.current_setting(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_is_in_recovery() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_start_backup(text, boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_stop_backup(boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_create_restore_point(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_switch_xlog() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_last_xlog_replay_location() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current_snapshot() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_snapshot_xmax(txid_snapshot) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_control_checkpoint() TO backup;
COMMIT;
## For Postgres Pro 10 or higher:
BEGIN;
CREATE ROLE backup WITH LOGIN;
GRANT USAGE ON SCHEMA pg_catalog TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.current_setting(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_is_in_recovery() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_start_backup(text, boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_stop_backup(boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_create_restore_point(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_last_wal_replay_lsn() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current_snapshot() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_snapshot_xmax(txid_snapshot) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_control_checkpoint() TO backup;
COMMIT;
3、新增需要備份例項資訊
一個備份目錄下可新增多個例項，可以是本地例項也可以是遠端例項
–instance : 指定instance的名稱
–remote-host : 指定遠端備份例項IP
–remote-port : 指定遠端備份例項SSH埠
–remote-user : 指定遠端SSH使用者
–remote-path : 指定遠端備份例項pg_probackup工具所在路徑
1）新增本地備份例項
## 新增本地例項
$ /usr/local/pg_probackup-2.4.2/pg_probackup add-instance -B /data/pgdata_probackup -D /data/pg5433/ --instance local_5433
INFO: Instance 'local_5433' successfully inited
$ /usr/local/pg_probackup-2.4.2/pg_probackup add-instance -B /data/pgdata_probackup -D /data/pgsql12/data/ --instance local_5432
INFO: Instance 'local_5432' successfully inited
2）新增遠端備份例項
## 新增遠端例項
$ /usr/local/pg_probackup-2.4.2/pg_probackup add-instance -B /data/pgdata_probackup/ -D /data/pgsql/data --instance zijie_5432 --remote-proto=ssh --remote-host=120.27.250.75 --remote-port=22 --remote-user=postgres --remote-path=/usr/local/pg_probackup-2.4.2/
INFO: Instance 'zijie_5432' successfully inited
4、修改需要備份例項的配置檔案
1）本地備份例項
## 本地備份例項
$ vi postgres.conf
max_wal_senders 設定合理值
wal_level = 'replica'
archive_mode = 'on'
archive_command = '/usr/local/pg_probackup-2.4.2/pg_probackup archive-push -B /data/pgdata_probackup --instance local_5432 --wal-file-path=%p --wal-file-name=%f'
2）遠端備份例項
## 遠端備份例項
max_wal_senders 設定合理值
wal_level = 'replica'
archive_mode = 'on'
archive_command = '/usr/local/pg_probackup-2.4.2/pg_probackup archive-push -B /data/pgdata_probackup --instance zijie_5432 --wal-file-path=%p --wal-file-name=%f --remote-proto=ssh --remote-host=124.70.209.222 --remote-port=22 --remote-user=postgres --remote-path=/usr/local/pg_probackup-2.4.2/'
4.2 資料備份(全備+增備)
1）本機例項資料備份
## 記錄t1表中原本的資料量
db1=# select count(*) from t1;
count
-------
8
(1 row)
## 進行第一次全備
$ /usr/local/pg_probackup-2.4.2/pg_probackup backup -B /data/pgdata_probackup/ --instance local_5432 -b full
INFO: Backup start, pg_probackup version: 2.4.2, instance: local_5432, backup ID: QH2EIL, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Wait for WAL segment /data/pgdata_probackup/wal/local_5432/00000001000000000000000A to be archived
INFO: PGDATA size: 39MB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 0
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 4s
INFO: Validating backup QH2EIL
INFO: Backup QH2EIL data files are valid
INFO: Backup QH2EIL resident size: 40MB
INFO: Backup QH2EIL completed
## 檢視備份資訊
$ /usr/local/pg_probackup-2.4.2/pg_probackup show -B /data/pgdata_probackup/
BACKUP INSTANCE 'local_5432'
===================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
===================================================================================================================================
local_5432 12 QH2EIL 2020-09-22 22:49:34+08 FULL ARCHIVE 1/0 6s 40MB 16MB 1.00 0/A000028 0/B0000B8 OK
local_5432 ---- QH2EI6 ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 ---- QH2EHB ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 ---- QH2EGB ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 ---- QH2EFM ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 12 QH2E90 ---- FULL ARCHIVE 1/0 3m:5s 0 0 1.00 0/5000028 0/0 ERROR
BACKUP INSTANCE 'local_5433'
=================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
=================================================================================================================
## 手動插入部分資料，模擬增量資料
db1=# select count(*) from t1;
count
-------
16
(1 row)
## 在全備的基礎上進行第一次增量備份
$ /usr/local/pg_probackup-2.4.2/pg_probackup backup -B /data/pgdata_probackup/ --instance local_5432 -b page
INFO: Backup start, pg_probackup version: 2.4.2, instance: local_5432, backup ID: QH2F68, backup mode: PAGE, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Wait for WAL segment /data/pgdata_probackup/wal/local_5432/00000001000000000000000D to be archived
INFO: Parent backup: QH2EIL
INFO: PGDATA size: 39MB
INFO: Extracting pagemap of changed blocks
INFO: Pagemap successfully extracted, time elapsed: 0 sec
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 0
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup QH2F68
INFO: Backup QH2F68 data files are valid
INFO: Backup QH2F68 resident size: 152kB
INFO: Backup QH2F68 completed
## 檢視備份資訊
$ /usr/local/pg_probackup-2.4.2/pg_probackup show -B /data/pgdata_probackup/
BACKUP INSTANCE 'local_5432'
====================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
====================================================================================================================================
local_5432 12 QH2F68 2020-09-22 23:03:45+08 PAGE ARCHIVE 1/1 2s 152kB 16MB 1.00 0/D000028 0/E0000F0 OK //第一次增備
local_5432 12 QH2EIL 2020-09-22 22:49:34+08 FULL ARCHIVE 1/0 6s 40MB 16MB 1.00 0/A000028 0/B0000B8 OK //全備
local_5432 ---- QH2EI6 ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 ---- QH2EHB ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 ---- QH2EGB ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 ---- QH2EFM ---- FULL ARCHIVE 0/0 0 0 0 1.00 0/0 0/0 ERROR
local_5432 12 QH2E90 ---- FULL ARCHIVE 1/0 3m:5s 0 0 1.00 0/5000028 0/0 ERROR
BACKUP INSTANCE 'local_5433'
=================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
=================================================================================================================
2）遠端例項資料備份
## 遠端例項的全量備份
$ /usr/local/pg_probackup-2.4.2/pg_probackup backup -B /data/pgdata_probackup/ --instance zijie_5432 --remote-user=postgres --remote-host=120.27.250.75 --remote-port=22 -b full
INFO: Backup start, pg_probackup version: 2.4.2, instance: zijie_5432, backup ID: QH3V3Y, backup mode: FULL, wal mode: ARCHIVE, remote: true, compress-algorithm: none, compress-level: 1
Password:
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Wait for WAL segment /data/pgdata_probackup/wal/zijie_5432/0000000100000000000000E7 to be archived
WARNING: By default pg_probackup assume WAL delivery method to be ARCHIVE. If continuous archiving is not set up, use '--stream' option to make autonomous backup. Otherwise check that continuous archiving works correctly.
INFO: PGDATA size: 59MB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 7m:41s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Wait for LSN 0/E80001D8 in archived WAL segment /data/pgdata_probackup/wal/zijie_5432/0000000100000000000000E8
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup QH3V3Y
INFO: Backup QH3V3Y data files are valid
INFO: Backup QH3V3Y resident size: 43MB
INFO: Backup QH3V3Y completed
## 檢視備份資訊
$ /usr/local/pg_probackup-2.4.2/pg_probackup show -B /data/pgdata_probackup/ --instance zijie_5432
=======================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
=======================================================================================================================================
zijie_5432 12 QH3V3Y 2020-09-23 17:57:28+08 FULL ARCHIVE 1/0 13m:59s 43MB 16MB 1.00 0/E7000028 0/E80001D8 OK
zijie_5432 12 QH3U2S ---- FULL ARCHIVE 1/0 17m:12s 54MB 0 1.00 0/E4000028 0/0 ERROR
zijie_5432 ---- QH3U2J ---- FULL ARCHIVE 0/0 4s 0 0 1.00 0/0 0/0 ERROR
zijie_5432 12 QH3U20 ---- FULL ARCHIVE 0/0 5s 0 0 1.00 0/0 0/0 ERROR
3）備份檔案可用性校驗
## 備份檔案校驗
$ /usr/local/pg_probackup-2.4.2/pg_probackup validate -B /data/pgdata_probackup/ --instance local_5432
INFO: Validate backups of the instance 'local_5432'
INFO: Validating backup QH2F68
INFO: Backup QH2F68 data files are valid
INFO: Backup QH2F68 WAL segments are valid
INFO: Validating backup QH2EIL
INFO: Backup QH2EIL data files are valid
INFO: Backup QH2EIL WAL segments are valid
ERROR: Backup QH2EI6 has server version , but current pg_probackup binary compiled with server version 12
$ /usr/local/pg_probackup-2.4.2/pg_probackup validate -B /data/pgdata_probackup/ --instance zijie_5432
INFO: Validate backups of the instance 'zijie_5432'
INFO: Validating backup QH3V3Y
INFO: Backup QH3V3Y data files are valid
INFO: Backup QH3V3Y WAL segments are valid
WARNING: Backup QH3U2S has status ERROR. Skip validation.
ERROR: Backup QH3U2J has server version , but current pg_probackup binary compiled with server version 12
4.3 資料恢復
1、全量備份恢復
# 先關閉源資料庫並備份原先的資料目錄
$ pg_ctl -D /data/pgsql12/data/ stop
waiting for server to shut down.... done
server stopped
$ mv /data/pgsql12/data/ /data/pgsql12/data_bak/
## 通過全備恢復
$ /usr/local/pg_probackup-2.4.2/pg_probackup restore -B /data/pgdata_probackup/ --instance local_5432 --recovery-target-time='2020-09-22 22:49:34'
INFO: Validating backup QH2EIL
INFO: Backup QH2EIL data files are valid
INFO: Backup validation completed successfully on time 2020-09-22 22:49:34+08, xid 658 and LSN 0/B0000B8
INFO: Backup QH2EIL is valid.
INFO: Restoring the database from backup at 2020-09-22 22:49:33+08
INFO: Start restoring backup files. PGDATA size: 39MB
INFO: Backup files are restored. Transfered bytes: 39MB, time elapsed: 0
INFO: Restore incremental ratio (less is better): 100% (39MB/39MB)
INFO: Syncing restored files to disk
INFO: Restored backup files are synced, time elapsed: 4s
INFO: Restore of backup QH2EIL completed.
## 啟動資料庫並檢查t1表資料恢復情況
$ pg_ctl -D /data/pgsql12/data/ start
waiting for server to start....2020-09-22 23:21:23.529 CST [32368] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit
2020-09-22 23:21:23.530 CST [32368] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-09-22 23:21:23.530 CST [32368] LOG: listening on IPv6 address "::", port 5432
2020-09-22 23:21:23.537 CST [32368] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-09-22 23:21:23.552 CST [32368] LOG: redirecting log output to logging collector process
2020-09-22 23:21:23.552 CST [32368] HINT: Future log output will appear in directory "/data/pgsql12/logs".
done
server started
db1=# select count(*) from t1;
count
-------
8 //可以看到資料恢復成功
(1 row)
2、增量資料恢復
## 再次關閉資料庫並刪除已有的資料目錄
$ pg_ctl -D /data/pgsql12/data/ stop
waiting for server to shut down.... done
server stopped
$ rm -rf /data/pgsql12/data
## 將資料恢復到第一次增備時間點，並檢查t1表的資料恢復情況
$ /usr/local/pg_probackup-2.4.2/pg_probackup restore -B /data/pgdata_probackup/ --instance local_5432 --recovery-target-time='2020-09-22 23:03:45'
INFO: Validating parents for backup QH2F68
INFO: Validating backup QH2EIL
INFO: Backup QH2EIL data files are valid
INFO: Validating backup QH2F68
INFO: Backup QH2F68 data files are valid
INFO: Backup validation completed successfully on time 2020-09-22 23:03:45+08, xid 659 and LSN 0/E0000F0
INFO: Backup QH2F68 is valid.
INFO: Restoring the database from backup at 2020-09-22 23:03:44+08
INFO: Start restoring backup files. PGDATA size: 39MB
INFO: Backup files are restored. Transfered bytes: 39MB, time elapsed: 0
INFO: Restore incremental ratio (less is better): 100% (39MB/39MB)
INFO: Syncing restored files to disk
INFO: Restored backup files are synced, time elapsed: 4s
INFO: Restore of backup QH2F68 completed.
[[email protected]_test pgsql12]$ pg_ctl -D /data/pgsql12/data/ start
waiting for server to start....2020-09-22 23:22:42.746 CST [32391] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit
2020-09-22 23:22:42.746 CST [32391] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-09-22 23:22:42.746 CST [32391] LOG: listening on IPv6 address "::", port 5432
2020-09-22 23:22:42.750 CST [32391] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-09-22 23:22:42.761 CST [32391] LOG: redirecting log output to logging collector process
2020-09-22 23:22:42.761 CST [32391] HINT: Future log output will appear in directory "/data/pgsql12/logs".
done
server started
db1=# select count(*) from t1;
count
-------
16 //可以看到資料恢復成功
(1 row)
```
五、模擬原機誤刪除表
當前備份情況
# su - postgres
$ pg_probackup validate -B /var/lib/pgsql/pg_probackup_dir
$ pg_probackup show -B /var/lib/pgsql/pg_probackup_dir
BACKUP INSTANCE 'pg11'
====================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
====================================================================================================================================
pg11 11 Q6O08Q 2020-03-04 18:38:51+08 PAGE ARCHIVE 1/1 2s 124kB 16MB 1.00 0/1E000028 0/1F0000B8 OK
pg11 11 Q6NYFO 2020-03-04 17:59:52+08 DELTA ARCHIVE 1/1 5s 68MB 16MB 1.00 0/1C000028 0/1C000160 OK
pg11 11 Q6NYEY 2020-03-04 17:59:26+08 PAGE ARCHIVE 1/1 5s 68MB 16MB 1.00 0/15000028 0/160000B8 OK
pg11 11 Q6NYDK 2020-03-04 17:58:38+08 PAGE ARCHIVE 1/1 7s 67MB 16MB 1.00 0/E000028 0/F0000F0 OK
pg11 11 Q6NYCA 2020-03-04 17:57:48+08 PAGE ARCHIVE 1/1 3s 212kB 16MB 1.00 0/6000028 0/70000B8 OK
pg11 11 Q6NXZ6 2020-03-04 17:50:00+08 FULL ARCHIVE 1/0 7s 39MB 16MB 1.00 0/4000028 0/4000160 OK
模擬誤刪除表
$ psql
psql (11.5)
Type "help" for help.
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+--------------+----------+----------
public | tmp_t0 | table | postgres
(1 rows)
postgres=# \! date
Wed Mar 4 21:17:56 CST 2020
postgres=# drop table tmp_t0;
DROP TABLE
postgres=# \! date
Wed Mar 4 21:18:10 CST 2020
現在需要恢復到表刪除之前的這個時間點 Wed Mar 4 21:17:56 CST 2020
$ pg_probackup validate -B /var/lib/pgsql/pg_probackup_dir --instance pg11 --recovery-target-time='2020-03-04 21:17:56+08'
INFO: Validating parents for backup Q6O08Q
INFO: Validating backup Q6NXZ6
INFO: Backup Q6NXZ6 data files are valid
INFO: Validating backup Q6NYCA
INFO: Backup Q6NYCA data files are valid
INFO: Validating backup Q6NYDK
INFO: Backup Q6NYDK data files are valid
INFO: Validating backup Q6NYEY
INFO: Backup Q6NYEY data files are valid
INFO: Validating backup Q6NYFO
INFO: Backup Q6NYFO data files are valid
INFO: Validating backup Q6O08Q
INFO: Backup Q6O08Q data files are valid
WARNING: Thread [1]: Could not read WAL record at 0/20000000
ERROR: Thread [1]: WAL segment "/var/lib/pgsql/pg_probackup_dir/wal/pg11/000000010000000000000020" is absent
WARNING: Recovery can be done up to time 2020-03-04 18:38:51+08, xid 716 and LSN 0/1F0000B8
ERROR: Not enough WAL records to time 2020-03-04 21:17:56+08
提示 ERROR ，因為備份集的 WAL 只能恢復到 2020-03-04 21:17:56+08，需要使用到當前的 WAL，要做個 switch wal
$ psql -c "select pg_switch_wal();"
$ pg_probackup validate -B /var/lib/pgsql/pg_probackup_dir --instance pg11 --recovery-target-time='2020-03-04 21:17:56+08'
INFO: Validating parents for backup Q6O08Q
INFO: Validating backup Q6NXZ6
INFO: Backup Q6NXZ6 data files are valid
INFO: Validating backup Q6NYCA
INFO: Backup Q6NYCA data files are valid
INFO: Validating backup Q6NYDK
INFO: Backup Q6NYDK data files are valid
INFO: Validating backup Q6NYEY
INFO: Backup Q6NYEY data files are valid
INFO: Validating backup Q6NYFO
INFO: Backup Q6NYFO data files are valid
INFO: Validating backup Q6O08Q
INFO: Backup Q6O08Q data files are valid
INFO: Backup validation completed successfully on time 2020-03-04 21:18:03+08, xid 716 and LSN 0/200097D0
INFO: Backup Q6O08Q is valid.
INFO: Validate of backup Q6O08Q completed.
沒有問題，可以利用這些備份集進行異機恢復了。
六、異機恢復
另外一臺機器需要安裝 postgresql 11 軟體，及 pg_probackup 2.2.7
# su - postgres
$
$ which pg_ctl
/usr/pgsql-11/bin/pg_ctl
$ pg_ctl --version
pg_ctl (PostgreSQL) 11.5
$
$ which pg_probackup
/bin/pg_probackup
$ pg_probackup --version
pg_probackup 2.2.7 (PostgreSQL 11.5)
所有的操作都異機在 /temp 下完成
異機建立 PGDATA 目錄
$ cd /tmp
$ mkdir pgsql11data;
chmod 700 /tmp/pgsql11data ;
建立 pg_probackup 初始化目錄
$ cd /tmp
$ mkdir pgprobackupdata
$ pg_probackup init -D /tmp/pgsql11data -B /tmp/pgprobackupdata
$ mkdir -p ./pgprobackupdata/backups/pg11;
mkdir -p ./pgprobackupdata/wal/pg11;
backups 及 wal 目錄下的那個 pg11 檔案代表的是 pg_probackup 時指定的 instance_name，需要和原庫保持一致。
原機上的備份及WAL
需要將原機上的備份及WAL拷貝到異機的指定目錄。
以下操作在原機上操作
# su - postgres
$ pwd
/var/lib/pgsql
$ cd /var/lib/pgsql/pg_probackup_dir/backups/pg11
$ tar -zcvf ./backup.tar.gz ./*
$ scp ./backup.tar.gz [email protected]:/tmp/pgprobackupdata/backups/pg11
$ cd /var/lib/pgsql/pg_probackup_dir/wal/pg11
$ tar -zcvf ./wal.tar.gz ./*
$ scp ./wal.tar.gz [email protected]:/tmp/pgprobackupdata/wal/pg11
異機解壓相關的檔案
# su - postgres
$ cd /tmp
$ cd /tmp/pgprobackupdata/backups/pg11
$ tar -zxvf ./backup.tar.gz
$ cd /tmp/pgprobackupdata/wal/pg11
$ tar -zxvf ./wal.tar.gz
異機驗證恢復
$ pg_probackup show -B /tmp/pgprobackupdata
BACKUP INSTANCE 'pg11'
====================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
====================================================================================================================================
pg11 11 Q6O08Q 2020-03-04 18:38:51+08 PAGE ARCHIVE 1/1 2s 124kB 16MB 1.00 0/1E000028 0/1F0000B8 OK
pg11 11 Q6NYFO 2020-03-04 17:59:52+08 DELTA ARCHIVE 1/1 5s 68MB 16MB 1.00 0/1C000028 0/1C000160 OK
pg11 11 Q6NYEY 2020-03-04 17:59:26+08 PAGE ARCHIVE 1/1 5s 68MB 16MB 1.00 0/15000028 0/160000B8 OK
pg11 11 Q6NYDK 2020-03-04 17:58:38+08 PAGE ARCHIVE 1/1 7s 67MB 16MB 1.00 0/E000028 0/F0000F0 OK
pg11 11 Q6NYCA 2020-03-04 17:57:48+08 PAGE ARCHIVE 1/1 3s 212kB 16MB 1.00 0/6000028 0/70000B8 OK
pg11 11 Q6NXZ6 2020-03-04 17:50:00+08 FULL ARCHIVE 1/0 7s 39MB 16MB 1.00 0/4000028 0/4000160 OK
$ pg_probackup validate -D /tmp/pgsql11data/ -B /tmp/pgprobackupdata/ --instance pg11 --recovery-target-time='2020-03-04 21:17:56+08'
INFO: Validating parents for backup Q6O08Q
INFO: Validating backup Q6NXZ6
INFO: Backup Q6NXZ6 data files are valid
INFO: Validating backup Q6NYCA
INFO: Backup Q6NYCA data files are valid
INFO: Validating backup Q6NYDK
INFO: Backup Q6NYDK data files are valid
INFO: Validating backup Q6NYEY
INFO: Backup Q6NYEY data files are valid
INFO: Validating backup Q6NYFO
INFO: Backup Q6NYFO data files are valid
INFO: Validating backup Q6O08Q
INFO: Backup Q6O08Q data files are valid
INFO: Backup validation completed successfully on time 2020-03-04 21:18:03+08, xid 716 and LSN 0/200097D0
INFO: Backup Q6O08Q is valid.
INFO: Validate of backup Q6O08Q completed.
$ pg_probackup restore -D /tmp/pgsql11data/ -B /tmp/pgprobackupdata/ --instance pg11 --recovery-target-time='2020-03-04 21:17:56+08'
INFO: Validating parents for backup Q6O08Q
INFO: Validating backup Q6NXZ6
INFO: Backup Q6NXZ6 data files are valid
INFO: Validating backup Q6NYCA
INFO: Backup Q6NYCA data files are valid
INFO: Validating backup Q6NYDK
INFO: Backup Q6NYDK data files are valid
INFO: Validating backup Q6NYEY
INFO: Backup Q6NYEY data files are valid
INFO: Validating backup Q6NYFO
INFO: Backup Q6NYFO data files are valid
INFO: Validating backup Q6O08Q
INFO: Backup Q6O08Q data files are valid
INFO: Backup validation completed successfully on time 2020-03-04 21:18:03+08, xid 716 and LSN 0/200097D0
INFO: Backup Q6O08Q is valid.
INFO: Restore of backup Q6O08Q completed.
異機恢復資料庫
自動生成了 recovery.conf 檔案
$ cd /tmp/pgprobackupdata
$ cat recovery.conf
# recovery.conf generated by pg_probackup 2.2.7
recovery_target_time = '2020-03-04 21:17:56+08'
recovery_target_action = 'pause'
restore_command = '/usr/bin/pg_probackup-11 archive-get -B /tmp/pgprobackupdata --instance pg11 --wal-file-path=%p --wal-file-name=%f'
修改引數
$ vi postgresql.conf
data_directory = '/tmp/pgsql11data'
hba_file = '/tmp/pgsql11data/pg_hba.conf'
ident_file = '/tmp/pgsql11data/pg_ident.conf'
port=5433
archive_mode = off
archive_command = ''
archive_timeout = 0
啟動pgsql
$ /usr/pgsql-11/bin/pg_ctl start -D /tmp/pgsql11data/
waiting for server to start....2020-03-04 22:34:59.914 CST [2961] LOG: listening on IPv4 address "0.0.0.0", port 5433
2020-03-04 22:34:59.914 CST [2961] LOG: listening on IPv6 address "::", port 5433
2020-03-04 22:34:59.916 CST [2961] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5433"
2020-03-04 22:34:59.921 CST [2961] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2020-03-04 22:34:59.961 CST [2961] LOG: redirecting log output to logging collector process
2020-03-04 22:34:59.961 CST [2961] HINT: Future log output will appear in directory "log".
. done
server started
檢視日誌，已經恢復到指定時間，並處於 pause 狀態
$ tail -n 50 -f /tmp/pgsql11data/log/postgresql-Wed.log
2020-03-04 22:34:59.965 CST [2963] LOG: database system was interrupted; last known up at 2020-03-04 18:38:50 CST
2020-03-04 22:34:59.965 CST [2963] LOG: creating missing WAL directory "pg_wal/archive_status"
2020-03-04 22:35:00.625 CST [2963] LOG: starting point-in-time recovery to 2020-03-04 21:17:56+08
INFO: pg_probackup archive-get from /tmp/pgprobackupdata/wal/pg11/00000001000000000000001E to /tmp/pgsql11data/pg_wal/RECOVERYXLOG
INFO: pg_probackup archive-get completed successfully
2020-03-04 22:35:00.721 CST [2963] LOG: restored log file "00000001000000000000001E" from archive
2020-03-04 22:35:00.729 CST [2963] LOG: redo starts at 0/1E000028
INFO: pg_probackup archive-get from /tmp/pgprobackupdata/wal/pg11/00000001000000000000001F to /tmp/pgsql11data/pg_wal/RECOVERYXLOG
INFO: pg_probackup archive-get completed successfully
2020-03-04 22:35:00.850 CST [2963] LOG: restored log file "00000001000000000000001F" from archive
2020-03-04 22:35:00.855 CST [2963] LOG: consistent recovery state reached at 0/1F0000B8
2020-03-04 22:35:00.856 CST [2961] LOG: database system is ready to accept read only connections
INFO: pg_probackup archive-get from /tmp/pgprobackupdata/wal/pg11/000000010000000000000020 to /tmp/pgsql11data/pg_wal/RECOVERYXLOG
INFO: pg_probackup archive-get completed successfully
2020-03-04 22:35:00.958 CST [2963] LOG: restored log file "000000010000000000000020" from archive
2020-03-04 22:35:00.962 CST [2963] LOG: recovery stopping before commit of transaction 716, time 2020-03-04 21:18:03.525552+08
2020-03-04 22:35:00.962 CST [2963] LOG: recovery has paused
2020-03-04 22:35:00.962 CST [2963] HINT: Execute pg_wal_replay_resume() to continue.
異機登入資料庫
$ psql -p 5433
psql (11.5)
Type "help" for help.
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+--------------+----------+----------
public | tmp_t0 | table | postgres
(1 rows)