Barman

Barman (Backup and Recovery Manager) 是一個用Python語言實現的PostgreSQL災難恢復管理工具，它由第二象限公司（2ndQuadrant）開源並維護。

http://docs.pgbarman.org/release/2.13/

功能

備份元資料、增量備份、保留策略、遠端回覆、WAL檔案歸檔壓縮和備份

分類

• 僅流複製
• 流複製+ssh/rsync

結構圖

安裝配置

依賴

• Python >= 3.4
• Python modules:
  • argcomplete
  • argh >= 0.21.2
  • psycopg2 >= 2.4.2
  • python-dateutil
  • setuptools
• PostgreSQL >= 8.3
• rsync >= 3.0.4 (optional for PostgreSQL >= 9.2)

建立使用者(barman)

useradd -m -d /ups/app/postgresql/barman barman

這裡

建立目錄

mkdir -p /ups/app/postgresql/barman/{python,log,bkdir}
chown -R barman.barman /ups/app/postgresql/barman

安裝依賴軟體(barman使用者執行)

python

su - barman

tar -xf Python-3.8.2.tgz
cd Python-3.8.2
./configure --prefix=/ups/app/postgresql/barman/python --enable-optimizations
make -j2 && make -j2 install

/ups/app/postgresql/barman/python/bin/pip3 install --upgrade pip
/ups/app/postgresql/barman/python/bin/python3 -V
 

安裝argcomplete

/ups/app/postgresql/barman/python/bin/pip3 install argcomplete-1.12.3-py2.py3-none-any.whl

驗證

/ups/app/postgresql/barman/python/bin/pip3 list

安裝argh

/ups/app/postgresql/barman/python/bin/pip3 install argh-0.26.2-py2.py3-none-any.whl

安裝python_dateutil

/ups/app/postgresql/barman/python/bin/pip3 install python_dateutil-2.8.2-py2.py3-none-any.whl
 

Processing ./python_dateutil-2.8.2-py2.py3-none-any.whl
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil
Successfully installed python-dateutil-2.8.2 six-1.16.0

安裝six

/ups/app/postgresql/barman/python/bin/pip3 install six-1.16.0-py2.py3-none-any.whl

安裝psycopg2

tar -xf psycopg2-2.9.1.tar.gz
cd psycopg2-2.9.1
/ups/app/postgresql/barman/python/bin/python3 setup.py build_ext --pg-config /ups/app/postgresql/pgsql-12/bin/pg_config build
/ups/app/postgresql/barman/python/bin/python3 setup.py build_ext --pg-config /ups/app/postgresql/pgsql-12/bin/pg_config install

檢查已安裝python 包

ls -lrt /ups/app/postgresql/barman/python/lib/python3.8/site-packages

/ups/app/postgresql/barman/python/bin/pip3 list

安裝barman

tar -xf barman-release-2.13.tar.gz
cd barman-release-2.13
/ups/app/postgresql/barman/python/bin/python3 setup.py build
/ups/app/postgresql/barman/python/bin/python3 setup.py install --user

驗證環境

export PGHOME=/ups/app/postgresql/pgsql-12
export LD_LIBRARY_PATH=$PGHOME/lib

/ups/app/postgresql/barman/python/bin/barman -h

配置barman配置檔案

cp -r barman-release-2.13/doc/barman.conf /ups/app/postgresql/barman/barman.conf
cp -r barman-release-2.13/doc/barman.d /ups/app/postgresql/barman/barman.d

配置PG Server端與barman（備份端）SSH互信

當PG Server伺服器與備份端(barman)不在同一裝置上，需要備份遠端的PG Server資料的環境，需要配置SSH

建立ssh-key

export SSH='ssh -o ConnectTimeout=3 -o ConnectionAttempts=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no'

ssh-keygen -b 4096 -t rsa -P '' -f ~/.ssh/id_rsa
ssh-keygen -b 4096 -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/*.pub >> ~/.ssh/authorized_keys

${SSH} barman cat  ~/.ssh/*.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys barman:~/.ssh/authorized_keys

chmod 600 ${HOME}/.ssh/authorized_keys

使用

用法

# /ups/app/postgresql/barman/python/bin/barman -h
usage: barman [-h] [-v] [-c CONFIG] [--color {never,always,auto}] [--log-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-q] [-d] [-f {console,json}]
              {archive-wal,backup,check,check-backup,cron,delete,diagnose,get-wal,list-backup,list-files,list-server,put-wal,rebuild-xlogdb,receive-wal,recover,show-backup,show-server,replication-status,status,switch-wal,switch-xlog,sync-info,sync-backup,sync-wals}
              ...

positional arguments:
  {archive-wal,backup,check,check-backup,cron,delete,diagnose,get-wal,list-backup,list-files,list-server,put-wal,rebuild-xlogdb,receive-wal,recover,show-backup,show-server,replication-status,status,switch-wal,switch-xlog,sync-info,sync-backup,sync-wals}
    archive-wal         Execute maintenance operations on WAL files for a given server. This command processes any incoming WAL files for the server and archives them along the catalogue.
    backup              Perform a full backup for the given server (supports 'all')
    check               Check if the server configuration is working. This command returns success if every checks pass, or failure if any of these fails
    check-backup        Make sure that all the required WAL files to check the consistency of a physical backup (that is, from the beginning to the end of the full backup) are correctly
                        archived. This command is automatically invoked by the cron command and at the end of every backup operation.
    cron                Run maintenance tasks (global command)
    delete              Delete a backup
    diagnose            Diagnostic command (for support and problems detection purpose)
    get-wal             Retrieve WAL_NAME file from SERVER_NAME archive. The content will be streamed on standard output unless the --output-directory option is specified.
    list-backup         List available backups for the given server (supports 'all')
    list-files          List all the files for a single backup
    list-server         List available servers, with useful information
    put-wal             Receive a WAL file from SERVER_NAME and securely store it in the incoming directory. The file will be read from standard input in tar format.
    rebuild-xlogdb      Rebuild the WAL file database guessing it from the disk content.
    receive-wal         Start a receive-wal process. The process uses the streaming protocol to receive WAL files from the PostgreSQL server.
    recover             Recover a server at a given time, name, LSN or xid
    show-backup         This method shows a single backup information
    show-server         Show all configuration parameters for the specified servers
    replication-status  Shows live information and status of any streaming client
    status              Shows live information and status of the PostgreSQL server
    switch-wal          Execute the switch-wal command on the target server
    switch-xlog         Execute the switch-wal command on the target server
    sync-info           Output the internal synchronisation status. Used to sync_backup with a passive node
    sync-backup         Command that synchronises a backup from a master to a passive node
    sync-wals           Command that synchronises WAL files from a master to a passive node

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -c CONFIG, --config CONFIG
                        uses a configuration file (defaults: ~/.barman.conf, /etc/barman.conf, /etc/barman/barman.conf)
  --color {never,always,auto}, --colour {never,always,auto}
                        Whether to use colors in the output (default: 'auto')
  --log-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Override the default log level
  -q, --quiet           be quiet (default: False)
  -d, --debug           debug output (default: False)
  -f {console,json}, --format {console,json}
                        output format (default: 'console')

Barman by EnterpriseDB (www.enterprisedb.com)
[root@progs bin]# 

PostgreSQL服務端配置

配置pg_hba.conf檔案

host    all             barman          0.0.0.0/0               trust
host    replication     barmstr         0.0.0.0/0               trust

postgresql.conf 檔案配置

# WAL級別設定至少保證archive（v9.5為replica）以上的
wal_level = replica
# 開啟歸檔模式
archive_mode = on
# 如果使用流複製的方式傳輸WAL到barman，archive_command則不是必須項
archive_command = ' '
# 最大WAL sender程序的數量，總數 > wal archive + streaming/logical replication
max_wal_senders = 5
# 最少保留WAL檔案的數量，防止WAL被服務節點過早移除
wal_keep_segments = 64

建立資料庫使用者

建立連線PG的使用者

• 可以建立superuser 許可權使用者(barman)

  createuser -s -P barman
  # psql -c "create user barman with LOGIN superuser password 'barman123';"
  

• 也可以建立普通使用者並授權必要的許可權

  createuser --replication -P barman
  psql <<EOF
  GRANT EXECUTE ON FUNCTION pg_start_backup(text, boolean, boolean) to barman;
  GRANT EXECUTE ON FUNCTION pg_stop_backup() to barman;
  GRANT EXECUTE ON FUNCTION pg_stop_backup(boolean, boolean) to barman;
  GRANT EXECUTE ON FUNCTION pg_switch_wal() to barman;
  GRANT EXECUTE ON FUNCTION pg_create_restore_point(text) to barman;
  
  GRANT pg_read_all_settings TO barman;
  GRANT pg_read_all_stats TO barman;
  EOF
  

  • 注意：barman switch-wal命令使用--force選項時，不生效

• 測試登入

  psql -c "SELECT version()" -U barman postgres
  

建立streaming replication連線賬號

當計劃使用 WAL streaming or streaming backup功能時

• 建立賬號

  createuser -P --replication streaming_barman
  # psql -c "create user streaming_barman with LOGIN replication password 'barman123'"
  

• 檢查確認

  psql -U streaming_barman \
    -c "IDENTIFY_SYSTEM" \
    replication=1
    
  psql "dbname=postgres port=5432 replication=database user=streaming_barman password=barman123" -c "IDENTIFY_SYSTEM;"
  

barman配置

配置檔案內容

• global/general 配置
  • 配置檔案預設路徑：/etc/barman.conf
  • 配置引數項：包含主目錄，系統使用者，日誌檔案等
• server 配置
  • 檔案目錄：/etc/barman.d下以.conf字尾名的檔案

由於歷史原因，可以使用一個檔案包含全域性和Server選項的配置檔案。但不推薦

配置項型別

• string
• enum
• integer
• boolean, on/true/1 are accepted as well are off/false/0.

選項範圍

• 全域性選項由 [barman] 標籤標識

  [barman]
  ; ... global and global/server options go here
  

• Server選項：在barman中標識PostgreSQL ID

  [pg]
  ; Configuration options for the
  ; server named 'pg' go here
  

建立軟連線

sudo ln -s /ups/app/postgresql/barman/barman.conf /etc/barman.conf
sudo ln -s /ups/app/postgresql/barman/barman.d /etc/barman.d

barman.conf 配置

cat > /ups/app/postgresql/barman/barman.conf <<EOF
[barman]
barman_user = barman
configuration_files_directory = /ups/app/postgresql/barman/barman.d
;指定barman_home，即備份檔案的存放路徑
barman_home = /ups/app/postgresql/barman/bkdir
log_file = /ups/app/postgresql/barman/log/barman.log
log_level = INFO
compression = gzip
backup_options = concurrent_backup
minimum_redundancy = 0
; 備份集，備份集中資料，WAL檔案的保留策略
retention_policy = RECOVERY WINDOW OF 1 WEEK
retention_policy_mode = auto
wal_retention_policy = main
; 指定目標資料庫備份配置的所在目錄
configuration_files_directory = /ups/app/postgresql/barman/barman.d
check_timeout = 5
EOF

streaming-server.conf 配置

cat > /ups/app/postgresql/barman/barman.d/pg12.conf <<EOF
[pg12]
description =  "pg12-5432 master database(Streaming Only)"
; Barman通過conninfo連線字元
conninfo = host=192.168.10.181 port=5432 user=barman dbname=postgres application_name=pg12
; 建立流複製的連線字串和WAL流複製歸檔方式
streaming_conninfo = host=192.168.10.181 port=5432 user=streaming_barman dbname=postgres password=barman123 application_name=pg12
; 開啟流複製方式的歸檔
streaming_archiver = on
; 通過插槽的方式管控WAL的流傳輸
slot_name = streaming_barman
create_slot = auto
; 建議針對資料庫的備份方式
backup_method = postgres
backup_options = concurrent_backup
; 通過流複製傳輸的WAL存放的位置
streaming_wals_directory=/ups/app/postgresql/barman/streaming
minimum_redundancy = 2
retention_policy = RECOVERY WINDOW OF 1 DAYS
; 指定Barman伺服器客戶端psql的安裝位置
path_prefix = "/ups/app/postgresql/pgsql-12/bin"
EOF

建立複製槽slot

barman --config /ups/app/postgresql/barman/barman.conf receive-wal --create-slot pg12

在後臺持續接收PostgreSQL服務端的WAL日誌

barman receive-wal pg12 &

備份前檢查驗證

驗證備份服務配置的狀態

barman --config /ups/app/postgresql/barman/barman.conf check pg12

確保所有狀態OK後，才可以開啟備份

備份操作

barman backup pg12

恢復操作

恢復目標庫

全量恢復

barman recover <target_database> <backup_id> <destination_directory> 
barman recover pg12 20210916T202157  ${PGDATA}

基於時間點(PITR)恢復

barman recover --target-time "2021-09-16 20:15:00" pg12 20210916T202157 ${PGDATA}

常用維護命令

#列出所有的備份服務資訊(--minimal 引數僅顯示服務名稱，不包含描述資訊)
$ barman list-server
#列出單個備份服務的所有配置資訊
$ barman show-server <target_database>
#列出單個備份服務的備份集資訊
$ barman list-backup <target_database>
 
# 檢視單個備份集的資訊
$ barman show-backup <target_database> <backup_id>
 
#檢查備份服務配置狀態
$ barman check <target_database>

# 檢查備份服務的實時資訊
$ barman status <target_database>
 
#以Json格式列出所有備份服務及備份集的詳細資訊
$ barman diagnose

#更多資訊可以參考
barman --help

故障排錯

receive-wal running: FAILED

現象

處理

# 可以嘗試通過重建replication slot重新整理WAL的接收狀態
barman receive-wal --stop <target_database>
barman receive-wal --reset <target_database>
barman receive-wal --create-slot <target_database>
barman switch-wal <target_database>

WAL archive: FAILED

處理

barman cron

## 切換日誌
barman switch-wal pg12  # pg 版本 >10
barman switch-xlog pg12 # pg 版本 <10

檢查結果