背景

因為 hadoop 環境需要 master 能免密碼 ssh localhost，所以我們需要建立與本機 localhost 的互信，方法很簡單：

1. ssh-keygen -t rsa
   #Press enter for each line 
2. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3. chmod og-wx ~/.ssh/authorized_keys 

這三步執行下來就能順利 ssh localhost 免密碼登入了，但是昨天剛建好的互信，今天下午突然不能用了，ssh localhost 需要密碼，第一反應是可能哪裡設定和配置被改動了，看了下檔案版本、配置修改時間都無變化，然而登入時的提示資訊又過於簡單，這個時候排查陷入僵局了。

work@test_zz_Master 192.168.187.213 18:45:18 ~ >
ssh localhost            
work@localhost's password: 

work@test_zz_Master 192.168.187.213 18:45:24 ~ >

1、怎麼排查？

1.1 debug 日誌

首先還是要拿到明細 debug 日誌，看看卡在哪裡了。linux 下的不少命令都自帶除錯功能，比如 ssh 就自帶 debug 功能：

ssh -vvv localhost
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/work/.ssh/identity type -1
debug1: identity file /home/work/.ssh/identity-cert type -1
...
debug3: remaining preferred: keyboard-interactive,password
// 啟用公鑰登入
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/work/.ssh/identity
debug3: no such identity: /home/work/.ssh/identity
debug1: Offering public key: /home/work/.ssh/id_rsa
debug3: send_pubkey_test
// 傳送公鑰包，等待伺服器認證響應
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1741
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/work/.ssh/id_dsa
debug3: no such identity: /home/work/.ssh/id_dsa
debug1: Trying private key: /home/work/.ssh/id_ecdsa
debug3: no such identity: /home/work/.ssh/id_ecdsa
// 沒通過認證，禁用該認證方法
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
// 下一個認證方法：啟用密碼登入
debug1: Next authentication method: password
work@localhost's password: 
 

可以看到，確實是認證失敗了，但是僅憑一句 we did not send a packet, disable method，咱們還是無法看到失敗的深層次原因，那咱們再對比下正常的認證流程應該是怎樣的：

可以看到右邊正常的會接受公鑰，左邊的則沒有得到響應，繼續走別的認證方式。

1.2 檢查配置

開啟伺服器的 /etc/ssh/sshd_config

確認下面幾行是這樣的:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys

#GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
 

配置沒問題，此路還是不通。

1.3 Debugging SSH public key 

在B機器上，we sent a public key packet, wait for reply 之後則是緊跟著"debug1: Server accepts key: pkalg ssh-rsa blen 277"。由此可以看出，是A機器的sshd不認可publickey。

至於為什麼不認可，在google上查了許多，毫無頭緒，直到使用類似“ssh publickey ignore debug diagnose”這樣的關鍵詞，發現這個頁面，其中的第二條和第六條給出瞭解答：

2. Debugging on the remote host by running sshd in debug mode: Run ‘/usr/sbin/sshd -d -p 2222′ on the remote host and connect to it. ’2222′ here is the port number of the sshd process you started on the remote host. 6. Check the permissions on your home directory, .ssh directory, and the authorized_keys file: If your ssh server is running with ‘StrictModes on’, it will refuse to use your public keys in the ~/.ssh/authorized_keys file. Your home directory should be writable only by you, ~/.ssh should be 700, and authorized_keys should be 600.

通過執行 /usr/sbin/sshd -d -p 2222 （在2222埠啟動一個帶debug輸出的sshd） ，

然後 ssh -vv localhost -p 2222 ，可以看到 sshd 的輸出：

[root(hostname)@bjdhj-187-213 ~]# /usr/sbin/sshd -d -p 2222
debug1: sshd version OpenSSH_5.3p1
debug1: read PEM private key done: type RSA
...
debug1: trying public key file /home/work/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
Authentication refused: bad ownership or modes for directory /home/work
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 500/500 (e=0/0)
debug1: trying public key file /home/work/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
Authentication refused: bad ownership or modes for directory /home/work
debug1: restore_uid: 0/0
Failed publickey for work from 127.0.0.1 port 45548 ssh2

可以看到倒數第三行：Authentication refused: bad ownership or modes for directory /home/work，

正好與那第六條相對應，再檢查一下 /home/work ，其許可權是否是其他組可讀寫。

同時，咱們也能從 /var/log/secure 看到明細的 debug 日誌：

[root(hostname)@bjdhj-187-213 ~]# tail -f /var/log/secure
Sep  1 18:52:20 bjdhj-187-213 sshd[30936]: Server listening on 0.0.0.0 port 22.
Sep  1 18:52:23 bjdhj-187-213 sshd[30944]: Authentication refused: bad ownership or modes for directory /home/work
Sep  1 18:52:23 bjdhj-187-213 sshd[30944]: Authentication refused: bad ownership or modes for directory /home/work
Sep  1 18:52:25 bjdhj-187-213 sshd[30948]: Connection closed by 127.0.0.1

2、最終解決方案

ssh 為了保證通訊安全，防止 key 被篡改或竊取，對目錄和檔案的許可權要求相當嚴格，

咱們最終需要確保相關目錄許可權與下述一致：

chmod 0755 ~               # 或 chmod g-w ~   
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

sudo service sshd restart

後記：

當然了，這篇文章所反映的問題雖然很小，最後的答案也很簡單，但是其展現的排查思路和方法卻很獨特，值得借鑑，畢竟很多時候咱們不能像平時一樣，直接 debug 原始碼。

Refer：

[1] 記一次sshd異常：無法通過建立信任關係登入

https://www.felix021.com/blog/read.php?2085

[2] ssh公鑰登入無效

https://my.oschina.net/sukai/blog/686981

[3] Debugging SSH public key authentication problems

https://blog.codefront.net/2007/02/28/debugging-ssh-public-key-authentication-problems/