最近一用户在使用BeetleX.HttpClient组件并发访问延时比较高的https服务时引起了卡死现像。由于组件更多使用场景是内部服务和非https,一直没有这情况出现;但用户提供测试场景下这情况必现,所以翻查了一些相关代码。
protected virtual void OnSslAuthenticate(SslStream sslStream)
{
Task task;
if (SslProtocols == null)
SslProtocols = System.Security.Authentication.SslProtocols.Tls | System.Security.Authentication.SslProtocols.Tls11 |
System.Security.Authentication.SslProtocols.Tls12;
task = sslStream.AuthenticateAsClientAsync(SslServiceName, CertificateCollection.Count > 0 ? CertificateCollection : null, SslProtocols.Value, false);
task.Wait();
}
BeetleX的tcpclient中使了无限超时来等待ssl验证环节,正常局域网环境这个等待是不会卡死的,毕竟网内环境良好要么成功要因异常触发取消等。但在网络差的环境下就存在问题了,socket的异步receive无法触发异常引起这个Wait把线程永久挂起了,当一段时间太多这种情况出现那非常的事情就是大量线程被这个Wait抽干导致服务不能正常工作。
为了解决这些比较特殊的情况加上超时时间就好了
protected virtual void OnSslAuthenticate(SslStream sslStream)
{
Task task;
if (SslProtocols == null)
SslProtocols = System.Security.Authentication.SslProtocols.Tls | System.Security.Authentication.SslProtocols.Tls11 |
System.Security.Authentication.SslProtocols.Tls12;
task = sslStream.AuthenticateAsClientAsync(SslServiceName, CertificateCollection.Count > 0 ? CertificateCollection : null, SslProtocols.Value, false);
if (!task.Wait(5000))
{
throw new BeetleX.BXException($"connect {mIPAddress}:{mPort} SSL Authenticate timeout!");
}
}
把超时设置成5秒,在创建连接5秒后ssl还没有握手成功就直接超时关闭连接重新创建。
基础的问题解决了,但httpclientpool层面还有一个问题需要处理。当连接池一开始处于满负载请求,同时创建大量的Client会引起大量线程等待导致服务不太稳定(毕竟在不很多稳定的网络环境下ssl握手时间有些长)。为了解决问题使用一个指定的线程数的队列来创建ssl握手处理机制,这样就可以避免同时创建大量连接线程引起短暂卡壳现象。
public Task<HttpClientHandler> Pop()
{
HttpClientHandler result;
TaskCompletionSource<HttpClientHandler> completionSource;
lock (this)
{
if (mPools.Count > 0)
{
result = mPools.Pop();
result.Using = true;
result.TimeOut = BeetleX.TimeWatch.GetElapsedMilliseconds() + TimeOut;
return Task.FromResult(result);
}
if (Clients.Count > MaxConnections)
{
if (mWaitQueue.Count < MaxWaitLength)
{
completionSource = new TaskCompletionSource<HttpClientHandler>();
mWaitQueue.Enqueue(completionSource);
return completionSource.Task;
}
else
{
throw new HttpClientException($"Request {Host} connections limit");
}
}
}
completionSource = new TaskCompletionSource<HttpClientHandler>();
mCreateDispatchCenter.Next().Enqueue(
new CreateClientTask { ClientHandlerPool = this, CompletionSource = completionSource });
return completionSource.Task;
}
struct CreateClientTask
{
public TaskCompletionSource<HttpClientHandler> CompletionSource;
public HttpClientHandlerPool ClientHandlerPool;
}
private static BeetleX.Dispatchs.DispatchCenter<CreateClientTask> mCreateDispatchCenter
= new Dispatchs.DispatchCenter<CreateClientTask>(OnProcessCreateClient, 20);
private static void OnProcessCreateClient(CreateClientTask e)
{
try
{
var result = e.ClientHandlerPool.Create();
result.Using = true;
result.TimeOut = BeetleX.TimeWatch.GetElapsedMilliseconds() + e.ClientHandlerPool.TimeOut;
e.CompletionSource.TrySetResult(result);
}
catch (Exception e_)
{
e.CompletionSource.TrySetException(e_);
}
}
简单修改一下连接池Pop方法的代码就可以了。
BeetleX
开源跨平台通讯框架(支持TLS)
提供高性能服务和大数据处理解决方案
https://beetlex.io