微服務中處理雪崩的方案

阿新 • • 發佈：2020-12-24

一、什麼是雪崩效應

二、雪落而不雪崩的解決方案

隔離

熔斷

降級

限流

超時

三、容錯

容錯的三個核心思想

保證自己不被上游服務壓垮
保證自己不被下游服務拖垮
保證自己不受外界環境影響

四、解決方案（Sentinel）

Sentinel規則

1. 流控規則

流量控制，其原理是監控流量的QPS（每秒查詢率）或者併發執行緒數等指標
1.1流控模式

直接（預設）
關聯（當關聯的資源達到限流條件時，開啟限流（適用於資源讓步））
鏈路（當從某個介面的過來的資源達到限流時，開啟限流，有點類似於針對來源配置項，區別在針對來源配置項針對上級微服務，鏈路流控針對於上級介面，也就是粒度最細）
如果要使鏈路規則生效，則版本需要2.1.1之後，並且增加配置
```
spring.cloud.sentinel.web-context-unify=false
```

1.2 流控效果

快速失敗
warm up（預熱）
排隊等待

2. 降級規則

降級規則指的就是在滿足什麼條件的時候對服務進行降級

慢呼叫比例
RT：平均響應時間
時間視窗：降級持續時間
當平均響應時間大於 輸入的值的時候，進入預降級狀態，如果1秒內持續進入的請求響應時間依然大於輸入的值，則進入降級狀態，降級時間為時間視窗輸入的值
異常比例
當資源的每秒異常總數佔總量的比例超過閾值後，就會進入降級狀態
異常數
當1分鐘之內的異常數大於異常數，則進入降級狀態，注意時間視窗需要大於60秒

3. 熱點規則

熱點引數流控規則是一種更細粒度的規則，它允許將規則具體細化到具體的引數上

4. 授權規則

自定義授權規則,繼承 RequestOriginParse,自定義授權。

import com.alibaba.csp.sentinel.adapter.spring.webmvc.callback.RequestOriginParser;
import org.apache.commons.lang3.StringUtils;
import org.springframework.context.annotation.Configuration;

import javax.servlet.http.HttpServletRequest;

/**
 * 自定義授權規則
 *
 * @author mrhy
 * @date 2020/10/3 11:25 下午
 * Copyright (C), 2018-2020
 */
@Configuration
public class MyRequestOriginParser implements RequestOriginParser {
    @Override
    public String parseOrigin(HttpServletRequest request) {
//        判斷來源或者指定帶有的規則
        String origin = request.getHeader("origin");
        if (StringUtils.isEmpty(origin)){
            throw new RuntimeException("origin is not empty");
        }
        return origin;
    }
}

5. 系統規則

（不推薦）

自定義異常介面

繼承UrlBlockHandler（Spring cloud alibaba 2.2.0 realease 之前）

繼承 BlockExceptionHandler（Spring cloud alibaba 2.2.0 realease 之後）

import com.alibaba.csp.sentinel.adapter.spring.webmvc.callback.BlockExceptionHandler;
import com.alibaba.csp.sentinel.slots.block.BlockException;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityException;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeException;
import com.alibaba.csp.sentinel.slots.block.flow.FlowException;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowException;
import com.alibaba.csp.sentinel.slots.system.SystemBlockException;
import com.alibaba.fastjson.JSON;
import com.mrhy.wisdomcommon.common.ObjectResponse;
import com.mrhy.wisdomcommon.common.OperationFlag;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.MediaType;

import javax.print.attribute.standard.Media;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * 自定義sentinel介面
 *
 * @author mrhy
 * @date 2020/10/3 11:34 下午
 * Copyright (C), 2018-2020
 */
@Configuration
public class ExceptionHandlerPage implements BlockExceptionHandler {
    @Override
    public void handle(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, BlockException e) throws Exception {
        ObjectResponse response = null;
        if (e instanceof FlowException) {
            response = new ObjectResponse(OperationFlag.INTERFACE_LIMITING.getReturnCode(), OperationFlag.INTERFACE_LIMITING.getDescription());
        } else if (e instanceof DegradeException) {
            response = new ObjectResponse(OperationFlag.SERVICE_DEGRADATION.getReturnCode(), OperationFlag.SERVICE_DEGRADATION.getDescription());
        } else if (e instanceof ParamFlowException) {
            response = new ObjectResponse(OperationFlag.HOT_PARAMETER_LIMITING.getReturnCode(), OperationFlag.HOT_PARAMETER_LIMITING.getDescription());
        } else if (e instanceof SystemBlockException) {
            response = new ObjectResponse(OperationFlag.TRIGGER_SYSTEM_PROTECT_ROLE.getReturnCode(), OperationFlag.TRIGGER_SYSTEM_PROTECT_ROLE.getDescription());
        } else if (e instanceof AuthorityException) {
            response = new ObjectResponse(OperationFlag.AUTHORIZATION_RULES.getReturnCode(), OperationFlag.AUTHORIZATION_RULES.getDescription());
        }
        httpServletResponse.setCharacterEncoding("utf-8");
        httpServletResponse.setContentType(MediaType.APPLICATION_JSON_VALUE);
        httpServletResponse.getWriter().write(JSON.toJSONString(response));
    }
}

狀態碼

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * 操作符
 *
 * @author mrhy
 */
@AllArgsConstructor
@Getter
public enum OperationFlag {
    /**
     * 成功
     */
    SUCCESS(0, "操作成功"),
    /**
     * 失敗
     */
    FAIL(-1, "操作失敗"),
    /**
     * 引數違法
     */
    ILLEGAL_ARGUMENT(-2, "引數違法"),
    /**
     * 未登入
     */
    NOT_LOGIN(401, "未登入"),
    /**
     * 服務降級
     */
    SERVICE_DEGRADATION(100, "服務降級"),
    /**
     * 介面限流
     */
    INTERFACE_LIMITING(101, "介面限流"),
    /**
     * 熱電引數限流
     */
    HOT_PARAMETER_LIMITING(102, "熱點引數限流"),
    /**
     * 觸發系統保護規則
     */
    TRIGGER_SYSTEM_PROTECT_ROLE(103, "觸發系統保護規則"),
    /**
     * 授權規則不通過
     */
    AUTHORIZATION_RULES(104, "授權規則不通過"),
    ;

    private final Integer returnCode;

    private final String description;
}

/**
 * 響應實體類
 * @author mrhy
 */
public class ObjectResponse {
    private Integer returnCode;
    private String description;
    private Object result;

    public ObjectResponse() {
        this.returnCode = OperationFlag.SUCCESS.getReturnCode();
        this.description = OperationFlag.SUCCESS.getDescription();
    }

    public ObjectResponse(Integer returnCode) {
        this(returnCode, (String) null);
    }

    public ObjectResponse(Object result) {
        this.returnCode = OperationFlag.SUCCESS.getReturnCode();
        this.description = OperationFlag.SUCCESS.getDescription();
        this.result = result;
    }

    public ObjectResponse(Integer returnCode, String description) {
        this(returnCode, description, (Object) null);
    }

    public ObjectResponse(Integer returnCode, String description, Object result) {
        this.returnCode = returnCode;
        this.description = description;
        this.result = result;
    }

    public Integer getReturnCode() {
        return returnCode;
    }

    public void setReturnCode(Integer returnCode) {
        this.returnCode = returnCode;
    }

    public String getDescription() {
        return description;
    }

    public void setDescription(String description) {
        this.description = description;
    }

    public Object getResult() {
        return result;
    }

    public void setResult(Object result) {
        this.result = result;
    }
}

將配置同步到nacos

引入依賴

     <dependency>
            <groupId>com.alibaba.csp</groupId>
            <artifactId>sentinel-datasource-nacos</artifactId>
        </dependency>

配置application

## ds指的資料來源，可以隨便寫
## server-addr 指的是nacos地址
spring.cloud.sentinel.datasource.ds.nacos.server-addr=localhost:8848 
## 對應配置中的data-id
spring.cloud.sentinel.datasource.ds.nacos.data-id=sentinel-wisdom-web
## 對應配置中的group-id
spring.cloud.sentinel.datasource.ds.nacos.group-id=DEFAULT_GROUP
## 對應檔案型別
spring.cloud.sentinel.datasource.ds.nacos.data-type=json
## 降級規則
spring.cloud.sentinel.datasource.ds.nacos.rule-type=flow

配置nacos（配置內容）

[
    {
        "resource": "test", // 資源名稱 
        "limitApp": "default", //來源應用
        "grade": 1,//閾值型別 0 執行緒數 1QPS
        "count": 5,// 單機閾值
        "strategy": 0,// 流控模式 0 直接 1 關聯 2 鏈路
        "controlBehavior": 0,// 流控效果 0 快速失敗 1 warmup 2 排隊等待
        "clusterMode": false // 是否是叢集模式
    }
]

參考網站sentinel github 網站

將配置持久化到本地

編寫配置類

import com.alibaba.csp.sentinel.command.handler.ModifyParamFlowRulesCommandHandler;
import com.alibaba.csp.sentinel.datasource.*;
import com.alibaba.csp.sentinel.init.InitFunc;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRule;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRuleManager;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRuleManager;
import com.alibaba.csp.sentinel.slots.system.SystemRule;
import com.alibaba.csp.sentinel.slots.system.SystemRuleManager;
import com.alibaba.csp.sentinel.transport.util.WritableDataSourceRegistry;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.TypeReference;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;

import java.io.File;
import java.io.IOException;
import java.util.List;

/**
 * sentinel 配置持久化到本地
 *
 * @author mrhy
 * @date 2020/10/4 10:17 上午
 * Copyright (C), 2018-2020
 */
@Configuration
public class FilePersistence implements InitFunc {
    private final String APPLICATION_NAME = "wisdom-web";

    @Override
    public void init() throws Exception {
        String ruleDir = System.getProperty("user.home") + "/sentinel-rules/" + APPLICATION_NAME;
        String flowRulePath = ruleDir + "/flow-rule.json";
        String degradeRulePath = ruleDir + "/degrade-rule.json";
        String systemRulePath = ruleDir + "/system-rule.json";
        String authorityRulePath = ruleDir + "/authority-rule.json";
        String paramFlowRulePath = ruleDir + "/param-flow-rule.json";

        this.mkdirIfNotExits(ruleDir);
        this.createFileIfNotExits(flowRulePath);
        this.createFileIfNotExits(degradeRulePath);
        this.createFileIfNotExits(systemRulePath);
        this.createFileIfNotExits(authorityRulePath);
        this.createFileIfNotExits(paramFlowRulePath);

        // 註冊一個可讀資料來源，用來定時讀取本地的json檔案，更新到規則快取中
        // 流控規則
        ReadableDataSource<String, List<FlowRule>> flowRuleRDS =
                new FileRefreshableDataSource<>(flowRulePath, flowRuleListParser);
        // 將可讀資料來源註冊至FlowRuleManager
        // 這樣當規則檔案發生變化時，就會更新規則到記憶體
        FlowRuleManager.register2Property(flowRuleRDS.getProperty());
        WritableDataSource<List<FlowRule>> flowRuleWDS = new FileWritableDataSource<>(
                flowRulePath,
                this::encodeJson
        );
        // 將可寫資料來源註冊至transport模組的WritableDataSourceRegistry中
        // 這樣收到控制檯推送的規則時，Sentinel會先更新到記憶體，然後將規則寫入到檔案中
        WritableDataSourceRegistry.registerFlowDataSource(flowRuleWDS);

        // 降級規則
        ReadableDataSource<String, List<DegradeRule>> degradeRuleRDS = new FileRefreshableDataSource<>(
                degradeRulePath,
                degradeRuleListParser
        );
        DegradeRuleManager.register2Property(degradeRuleRDS.getProperty());
        WritableDataSource<List<DegradeRule>> degradeRuleWDS = new FileWritableDataSource<>(
                degradeRulePath,
                this::encodeJson
        );
        WritableDataSourceRegistry.registerDegradeDataSource(degradeRuleWDS);

        // 系統規則
        ReadableDataSource<String, List<SystemRule>> systemRuleRDS = new FileRefreshableDataSource<>(
                systemRulePath,
                systemRuleListParser
        );
        SystemRuleManager.register2Property(systemRuleRDS.getProperty());
        WritableDataSource<List<SystemRule>> systemRuleWDS = new FileWritableDataSource<>(
                systemRulePath,
                this::encodeJson
        );
        WritableDataSourceRegistry.registerSystemDataSource(systemRuleWDS);

        // 授權規則
        ReadableDataSource<String, List<AuthorityRule>> authorityRuleRDS = new FileRefreshableDataSource<>(
                authorityRulePath,
                authorityRuleListParser
        );
        AuthorityRuleManager.register2Property(authorityRuleRDS.getProperty());
        WritableDataSource<List<AuthorityRule>> authorityRuleWDS = new FileWritableDataSource<>(
                authorityRulePath,
                this::encodeJson
        );
        WritableDataSourceRegistry.registerAuthorityDataSource(authorityRuleWDS);

        // 熱點引數規則
        ReadableDataSource<String, List<ParamFlowRule>> paramFlowRuleRDS = new FileRefreshableDataSource<>(
                paramFlowRulePath,
                paramFlowRuleListParser
        );
        ParamFlowRuleManager.register2Property(paramFlowRuleRDS.getProperty());
        WritableDataSource<List<ParamFlowRule>> paramFlowRuleWDS = new FileWritableDataSource<>(
                paramFlowRulePath,
                this::encodeJson
        );
        ModifyParamFlowRulesCommandHandler.setWritableDataSource(paramFlowRuleWDS);
    }

    private Converter<String, List<FlowRule>> flowRuleListParser = source -> JSON.parseObject(
            source,
            new TypeReference<List<FlowRule>>() {
            }
    );
    private Converter<String, List<DegradeRule>> degradeRuleListParser = source -> JSON.parseObject(
            source,
            new TypeReference<List<DegradeRule>>() {
            }
    );
    private Converter<String, List<SystemRule>> systemRuleListParser = source -> JSON.parseObject(
            source,
            new TypeReference<List<SystemRule>>() {
            }
    );

    private Converter<String, List<AuthorityRule>> authorityRuleListParser = source -> JSON.parseObject(
            source,
            new TypeReference<List<AuthorityRule>>() {
            }
    );

    private Converter<String, List<ParamFlowRule>> paramFlowRuleListParser = source -> JSON.parseObject(
            source,
            new TypeReference<List<ParamFlowRule>>() {
            }
    );

    private void mkdirIfNotExits(String filePath) throws IOException {
        File file = new File(filePath);
        if (!file.exists()) {
            file.mkdirs();
        }
    }

    private void createFileIfNotExits(String filePath) throws IOException {
        File file = new File(filePath);
        if (!file.exists()) {
            file.createNewFile();
        }
    }

    private <T> String encodeJson(T t) {
        return JSON.toJSONString(t);
    }
}

在resource 下新建這個目錄

META-INF.services

在這個目錄下新建檔案

com.alibaba.csp.sentinel.init.InitFunc

新增內容

上面寫的java檔案的路徑 copy reference