問題描述

在使用Azure Function App的SendGrid Binging功能，呼叫SendGrid伺服器傳送郵件功能時，遇到見間歇性，偶發性的異常。在重新執行SendGrid的Function，卻又能恢復執行。

所以本文基於Azure Function使用SendGrid的異常錯誤日誌，一步一步，分析原始碼中的方法內容。 然後調查為什麼 Azure Function 沒有自動重試(Retry)？

(如需要參考如何使用Azure Function SendGrid，參考：Azure Functions SendGrid 繫結)

錯誤訊息：

System.Threading.Tasks.TaskCanceledException : The operation was canceled.
 

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.ConnectHelper.ConnectAsync(String host,Int32 port,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at 
 
async System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request,Boolean allowHttp2,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`
 
1.get_Result()
   at async System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request,Boolean doRequestAuth,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,HttpRequestMessage request,CancellationTokenSource cts,Boolean disposeCts)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.SendGridClient.RequestAsync(Method method,String requestBody,String queryParams,String urlPath,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async SendGrid.SendGridClient.SendEmailAsync(SendGridMessage msg,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Client.SendGridClient.SendMessageAsync(SendGridMessage msg,CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Client\SendGridClient.cs : 23
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Extensions.Bindings.SendGridMessageAsyncCollector.FlushAsync(CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Bindings\SendGridMessageAsyncCollector.cs : 56
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Bindings.AsyncCollectorValueProvider`2.SetValueAsync[TUser,TMessage](Object value,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\AsyncCollector\AsyncCollectorValueProvider.cs : 49
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Binder.Complete(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\Binder.cs : 150
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Bindings.Runtime.RuntimeValueProvider.SetValueAsync(Object value,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\RuntimeValueProvider.cs : 34
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ParameterHelper.ProcessOutputParameters(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 925
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 518
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 279
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 326
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 94

問題分析

檢視異常日誌：

1）從最後一行看， 根據方法Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync 可以得出，程式碼已經進入Function平臺級別。可以初步排除是自己寫的程式碼錯誤。

2）在逐行上看，發現C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.SendGrid\Client\SendGridClient.cs : 23 中，呼叫了Client.SendGridClient.SendMessageAsync(SendGridMessage msg,CancellationToken cancellationToken) 並丟擲異常。

因為Azure Funciton的Source Code已經開源，所以可以在Github中檢視到WebJobs.Extensions.SendGrid\Client\SendGridClient.cs 的原始碼。

注意： 程式碼中直接呼叫SendGird SDK中的Client物件，傳送郵件 SendEmailAsync。

3) 繼續向上檢視，發現進入 SendGrid 的SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request,CancellationToken cancellationToken) 中。同第二步一樣，繼續在Github中搜索SendGrid的原始碼，檢視MakeRequest方法中進行了那些操作。

注意：方法MakeRequest中也沒有多餘配置，只是更深一步傳遞 Request請求。然後對獲取的結果進行異常處理或成功還回，外加設定響應編碼為UTF8

4)繼續向上檢視，進入SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) 中。在此，發現可以設定以下四種條件的Retry次數。

       private static readonly List<HttpStatusCode> RetriableServerErrorStatusCodes =
            new List<HttpStatusCode>()
            {
                HttpStatusCode.InternalServerError,
                HttpStatusCode.BadGateway,
                HttpStatusCode.ServiceUnavailable,
                HttpStatusCode.GatewayTimeout,
            };

RetryDeletegatingHandler.SendAsync 程式碼：

注意：如果設定了MaximumNumberOfRetries值，則當傳送請求到SendGrid不成功時，會自動重試所設定的次數。重試次數必須在0 ~ 5 次之間。

            if (maximumNumberOfRetries < 0)
            {
                throw new ArgumentOutOfRangeException(nameof(maximumNumberOfRetries), "maximumNumberOfRetries must be greater than 0");
            }

            if (maximumNumberOfRetries > 5)
            {
                throw new ArgumentOutOfRangeException(nameof(maximumNumberOfRetries), "The maximum number of retries allowed is 5");
            }

5)繼續向上檢視，發現後期的方法全是System.Net.Http 類中對HTTP請求的處理。至此，程式碼檢視到達源端。

6）回看異常訊息System.Threading.Tasks.TaskCanceledException : The operation was canceled. 這是因為引數 cancellationToken 的原因。因為請求為非同步 Task操作。當在請求長時間沒有處理完成並且cancellationToken中設定的取消時間已到，當前任務會被取消並丟擲以上訊息：The operation was canceled.

為什麼需要取消令牌(CancellationToken)？

因為Task沒有方法支援在外部取消Task，只能通過一個公共變數存放執行緒的取消狀態，線上程內部通過變數判斷執行緒是否被取消，當CancellationToken是取消狀態，Task內部未啟動的任務不會啟動新執行緒。

(Source: https://www.cnblogs.com/fanfan-90/p/12660996.html)

通過檢視原始碼，我們找到了為什麼出現TaskCanceledException的異常，但是為什麼 Function App 沒有設定SendGrid的重試引數呢？ 繼續檢視原始碼，發現，Function App在初始化SendGrid Client物件時，並沒有做任何的配置修改。所以Retry預設保持為0.

/// <summary>
/// Gets the maximum number of retries to execute against when sending an HTTP Request before throwing an exception. 
/// Defaults to 0 (no retries, you must explicitly enable).
/// </summary>
public int MaximumNumberOfRetries { get; }
        

Azure Function 的 SendGridOptions.cs 原始檔中，並沒有任何MaximumNumberOfRetries設定。

// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.

using SendGrid.Helpers.Mail;

namespace Microsoft.Azure.WebJobs.Extensions.SendGrid
{
    /// <summary>
    /// Defines the configuration options for the SendGrid binding.
    /// </summary>
    public class SendGridOptions
    {
        /// <summary>
        /// Gets or sets the SendGrid ApiKey. If not explicitly set, the value will be defaulted
        /// to the value specified via the 'AzureWebJobsSendGridApiKey' app setting or the
        /// 'AzureWebJobsSendGridApiKey' environment variable.
        /// </summary>
        public string ApiKey { get; set; }

        /// <summary>
        /// Gets or sets the default "to" address that will be used for messages.
        /// This value can be overridden by job functions.
        /// </summary>
        /// <remarks>
        /// An example of when it would be useful to provide a default value for 'to' 
        /// would be for emailing your own admin account to notify you when particular
        /// jobs are executed. In this case, job functions can specify minimal info in
        /// their bindings, for example just a Subject and Text body.
        /// </remarks>
        public EmailAddress ToAddress { get; set; }

        /// <summary>
        /// Gets or sets the default "from" address that will be used for messages.
        /// This value can be overridden by job functions.
        /// </summary>
        public EmailAddress FromAddress { get; set; }
    }
}

參考資料

Azure Functions SendGrid 繫結:https://docs.azure.cn/zh-cn/azure-functions/functions-bindings-sendgrid?tabs=csharp

WebJobs.Extensions.SendGrid:https://github.com/Azure/azure-webjobs-sdk-extensions/tree/dev/src/WebJobs.Extensions.SendGrid

SendGrid:https://github.com/sendgrid/sendgrid-csharp/tree/main/src/SendGrid

多執行緒筆記-CancellationToken（取消令牌）：https://www.cnblogs.com/fanfan-90/p/12660996.html

當在複雜的環境中面臨問題，格物之道需：濁而靜之徐清，安以動之徐生。 雲中，恰是如此!