dubbo提供者停止服務後zookeeper註冊中心仍然存在
阿新 • • 發佈：2020-10-22
dubbo服務停掉了，可是zk上面還有該節點，這樣的話，客戶端在消費的時候就會出現呼叫失敗的情況。造成該問題的原因有很多，開篇先說我的解決方案，是將dubbo版本從2.7.1升級到2.7.3。
ok，說完解決方案，接下來簡單描述一下這個問題的解決過程。
我剛進入YF專案組。今天接到反映，說測試環境在呼叫下發介面的時候報“內部服務異常”。
檢查log，發現dubbo服務調用出現異常：
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method merIsExistProject in the service com.auth.merchant.api.service.MerProjectService. Tried 1 times of the providers [127.0.0.1:20886] (1/8) from the registry 192.168.40.49:2181 on the consumer 192.168.40.51 using the dubbo version 2.5.3. Last error is: Failed to invoke remote method: merIsExistProject, provider: dubbo://127.0.0.1:20886/com.auth.merchant.api.service.MerProjectService?anyhost=true&application=risk-service-trans-business&bean.name=providers:dubbo:com.auth.merchant.api.service.MerProjectService:1.0.0&check=false&default.deprecated=false&default.dynamic=false&default.register=true&deprecated=false&dubbo=2.0.2&dynamic=false&generic=false&interface=com.auth.merchant.api.service.MerProjectService&methods=updateMerJhLevyProject,queryHall,query,insertMerJhLevy,update,insert,list,delete,listByParam,updateHall,merIsExistProject,listHall,insertBatch,getMerJhLevyProjectList&pid=16218®ister=true&release=2.7.1&retries=0&revision=0.0.1-SNAPSHOT&side=consumer&timeout=50000×tamp=1603070513879&version=1.0.0, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.auth.merchant.api.service.MerProjectService.null:1.0.0:20886 in [], may be version or group mismatch , channel: consumer: /192.168.40.51:20886 --> provider: /192.168.40.51:20886, message:RpcInvocation [methodName=merIsExistProject, parameterTypes=[class com.auth.merchant.api.vo.MerJhLevyProjectVo], arguments=[MerJhLevyProject{id=null, jhMerNo=89900000080215728529, levyMerNo=null, levyId=null, projectId=5113}], attachments={path=com.auth.merchant.api.service.MerProjectService, input=398, _isCallBackServiceInvoke=true, dubbo=2.5.3, interface=com.auth.merchant.api.service.MerProjectService, version=1.0.0, timeout=50000}]com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.auth.merchant.api.service.MerProjectService.null:1.0.0:20886 in [], may be version or group mismatch , channel: consumer: /192.168.40.51:20886 --> provider: /192.168.40.51:20886, message:RpcInvocation [methodName=merIsExistProject, parameterTypes=[classcom.auth.merchant.api.vo.MerJhLevyProjectVo], arguments=[MerJhLevyProject{id=null, jhMerNo=89900000080215728529, levyMerNo=null, levyId=null, projectId=5113}], attachments={path=com.auth.merchant.api.service.MerProjectService, input=398, _isCallBackServiceInvoke=true, dubbo=2.5.3, interface=com.auth.merchant.api.service.MerProjectService, version=1.0.0, timeout=50000}]
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method merIsExistProject in the service com.auth.merchant.api.service.MerProjectService. Tried 1 times of the providers [127.0.0.1:20886] (1/8) from the registry 192.168.40.49:2181 on the consumer 192.168.40.51 using the dubbo version 2.5.3. Last error is: Failed to invoke remote method: merIsExistProject, provider: dubbo://127.0.0.1:20886/com.auth.merchant.api.service.MerProjectService?anyhost=true&application=risk-service-trans-business&bean.name=providers:dubbo:com.auth.merchant.api.service.MerProjectService:1.0.0&check=false&default.deprecated=false&default.dynamic=false&default.register=true&deprecated=false&dubbo=2.0.2&dynamic=false&generic=false&interface=com.auth.merchant.api.service.MerProjectService&methods=updateMerJhLevyProject,queryHall,query,insertMerJhLevy,update,insert,list,delete,listByParam,updateHall,merIsExistProject,listHall,insertBatch,getMerJhLevyProjectList&pid=16218®ister=true&release=2.7.1&retries=0&revision=0.0.1-SNAPSHOT&side=consumer&timeout=50000×tamp=1603070513879&version=1.0.0, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.auth.merchant.api.service.MerProjectService.null:1.0.0:20886 in [], may be version or group mismatch , channel: consumer: /192.168.40.51:20886 --> provider: /192.168.40.51:20886, message:RpcInvocation [methodName=merIsExistProject, parameterTypes=[class com.auth.merchant.api.vo.MerJhLevyProjectVo], arguments=[MerJhLevyProject{id=null, jhMerNo=89900000080215728529, levyMerNo=null, levyId=null, projectId=5113}], attachments={path=com.auth.merchant.api.service.MerProjectService, input=398, _isCallBackServiceInvoke=true, dubbo=2.5.3, interface=com.auth.merchant.api.service.MerProjectService, version=1.0.0, timeout=50000}]
com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.auth.merchant.api.service.MerProjectService.null:1.0.0:20886 in [], may be version or group mismatch , channel: consumer: /192.168.40.51:20886 --> provider: /192.168.40.51:20886, message:RpcInvocation [methodName=merIsExistProject, parameterTypes=[class com.auth.merchant.api.vo.MerJhLevyProjectVo], arguments=[MerJhLevyProject{id=null, jhMerNo=89900000080215728529, levyMerNo=null, levyId=null, projectId=5113}], attachments={path=com.auth.merchant.api.service.MerProjectService, input=398, _isCallBackServiceInvoke=true, dubbo=2.5.3, interface=com.auth.merchant.api.service.MerProjectService, version=1.0.0, timeout=50000}]
一個夥伴找運維重啟了zookeeper和dubbo服務應用，問題依然存在。
我檢查了一下dubbo介面定義，呼叫方和服務方的契約是一致的。
異常是dubbo.remoting.RemotingException: Not found exported service，網上搜了一下，未果。
後來，從zkui上看服務的providers，發現除了伺服器的ip以外，還有好多ip地址。用ZooInspector同樣證實。看來小組裡有開發人員的本地服務註冊上去了。可是現在他本地並沒有起服務。
接下來，從zkui上把這些ip的提供者摘掉，重新呼叫發現ok了。
為什麼dubbo服務都停了，zk上卻還存在呢？從小組的夥伴瞭解到，這個問題出現已久。
而我此前用dubbo作為rpc框架的專案，並沒有出現這個問題。就是說，服務停掉後，zk上面就不再有這個節點了。
於是，我決定嘗試fix掉這個問題。
經過對dubbo配置和jar包的不斷調整和測試，最後，發現是dubbo版本在作祟。
我們知道，dubbo註冊到zk的節點是臨時節點。即，服務註冊後，zk會增加服務節點；當服務停用後，zk客戶端斷開，zk服務端會自動刪除這個服務節點。
而版本2.7.1呢，zk會把dubbo節點註冊為持久節點。
後來，同事發現一篇部落格，對此解釋得很透徹：https://www.cnblogs.com/goodAndyxublog/p/10878186.html
我們看dubbo-2.7.1與dubbo-2.7.3對AbstractServiceConfig.dynamic的註釋：
//public abstract class AbstractServiceConfig -- dubbo-2.7.3 Whether to register as a dynamic service or not on register center, the value is true, the status will be enabled after the service registered,and it needs to be disabled manually; if you want to disable the service, you also need manual processing. (baidu翻譯：是否在註冊中心註冊為動態服務，值為true，服務註冊後狀態為啟用，需要手動禁用；如果要禁用該服務，還需要手動處理。） protected Boolean dynamic = true; -- dubbo-2.7.1 Whether to register as a dynamic service or not on register center, it the value is false, the status will be disabled after the service registered,and it needs to be enabled manually; if you want to disable the service, you also need manual processing. （baidu翻譯：是否在註冊中心註冊為動態服務，如果值為false，則服務註冊後狀態為禁用，需要手動啟用；如果要禁用該服務，還需要手動處理。） protected Boolean dynamic = false;
其他解決方案：baidu：dubbo提供者停止服務後zookeeper臨時節點不能自動摘除
ZKUI截圖：