昨天公司的測試伺服器由於別的空間滿了導致mysql不能訪問，後來釋放了部分空間後，雖然主節點可以正常重啟。但是從節點就起不來了。從節點日誌中報如下錯誤：
/var/lib/mysql/mysql-error.log:

xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 13 - Permission denied)
xbstream: failed to create file.
2015/06/02 15:00:01 socat[17491] E write(1, 0x161d400, 1512): Broken pipe
WSREP_SST: [ERROR] Error while getting data from
 
 donor node:  exit codes: 1 1 (20150602 15:00:01.125)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150602 15:00:01.128)
2015-06-02 15:00:01 17227 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.135' --auth 'root:root123' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'
 
 --parent '17227'  '' : 32 (Broken pipe)
2015-06-02 15:00:01 17227 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-06-02 15:00:01 17227 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-06-02 15:00:01 17227 [ERROR] Aborting

其中可以看到Can’t create/write to file ‘././backup-my.cnf’，這時可以同時檢視主節點日誌：
innobackup.backup.log

xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[02] Compressing and streaming ./mysql/slave_worker_info.ibd
[02]        ...done

類似一堆這種錯誤，猜測是xtrabackup備份時有檔案不能寫，故打算重新做次備份。故首先刪除datadir(/var/lib/mysql)並重建一個（注意將其許可權改為mysql:mysql），然後為確保沒有mysql程序以及能重新其起來，一是檢視ps -ef|grep mysql，二是刪除/var/lock/subsys/mysql空檔案，這時再啟動，可以發現主節點不報錯了，但是從節點依然報錯：

WSREP_SST: [ERROR] Cleanup after exit with status:1 (20150603 16:13:46.587)
2015-06-03 16:13:46 23751 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.135' --auth 'root:root123' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '23751'  '' : 1 (Operation not permitted)
2015-06-03 16:13:46 23751 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-06-03 16:13:46 23751 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2015-06-03 16:13:46 23751 [ERROR] Aborting

但是可以發現和剛才報錯不一樣了，之後查了很多資料，雖然有很多類似的現象，但是仍有細微之處不太一樣，最後谷歌中有個哥們，他把my.cnf中的wsrep_sst_method值從xtrabackup-v2改成了rsync，然後再新同步，竟然能成功，然後又改回xtrabackup-v2再重啟，竟然也能成功，這個還是不能理解，但是確實解決了這個問題。先mark一下。