.NetCore Enyim.Caching Memcached源码分析

由于asp.net core中的session用着感觉并不顺手，所以从最开始就重新写了套缓存框架，底层可以用本地缓存、redis、memcached，当然作为缓存，我还是优先选择了memcached。本身用.net core的人就很少，坑的地方实在太多，像这种比较出名的框架都可能有bug，还得把源码弄下来看，文献就更少了。我用的是EnyimMemcachedCore，这是从.net上移植过来的，感觉也还不错，就在我发布到Windows服务器上跑了1天不到，就一直出错（linux内网内目前来说好像没问题）

Global:Failed to write to the socket '*.*.*.*:11211'. Error: ConnectionReset 2017/10/13 9:22:31
System.Exception: Failed to write to the socket '*.*.*.*:11211'. Error: ConnectionReset
   at Enyim.Caching.Memcached.ThrowHelper.ThrowSocketWriteError(EndPoint endpoint, SocketError error)
   at Enyim.Caching.Memcached.PooledSocket.Write(IList`1 buffers)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
   at Enyim.Caching.MemcachedClient.PerformStore(StoreMode mode, String key, Object value, UInt32 expires, UInt64& cas, Int32& statusCode)
   at Enyim.Caching.MemcachedClient.Store(StoreMode mode, String key, Object value, DateTime expiresAt)
   at Common.MemcachedHelper.Add(String key, Object o, Int32 time) in D:\Documents\workspaces\VS2017\Search\Common\MemcachedHelper.cs:line 19
   at BLL.Cache.CreateSession() in D:\Documents\workspaces\VS2017\Search\BLL\Cache.cs:line 71
   at Search.Controllers.BaseController.OnActionExecuting(ActionExecutingContext context) in D:\Documents\workspaces\VS2017\Search\Search\Controllers\BaseController.cs:line 27
   at Microsoft.AspNetCore.Mvc.Controller.<OnActionExecutionAsync>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionFilter.<OnActionExecutionAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Common.LogHelper.ErrorHandler.<Invoke>d__2.MoveNext() in D:\Documents\workspaces\VS2017\Search\Common\LogHelper.cs:line 116
------------------------------------------------------------------------------------------------

真的让我感到非常奇怪，在这种框架中居然会有这种错误，照理来说数据库连接框架都有连接池自行维护的，并且是非常稳定的。就这样，看了下这个框架源码，我这写几个比较主要的地方

首先说下这个框架的机制：

services.AddEnyimMemcached(options => _configuration.GetSection("enyimMemcached").Bind(options));

app.UseEnyimMemcached();

通过在startup.cs中添加这2句便可完成依赖注入的配置，因为本身.net core就有一套依赖注入的组件，移植过来的这个人选择通过依赖注入来使用MemcachedClient，所以每次使用的时候需要通过HttpContext.RequestServices.GetService<IMemcachedClient>()来创建MemcachedClient这个类，外层可以创建个对象池来管理创建的MemcachedClient。而且看一下MemcachedClient的接口，可以看出连接的控制都被封装了，那么就是说无需关心连接是正常还是断开的，只需要做增删改查的操作即可。那么现在可以确定，内部一定有连接池，我每次操作都应该是自动分配到一个连接，操作完成后又把这个连接放回池内，并且这些连接在底层一定是长连接

介绍从外到内几个比较关键的类：MemcachedClient->MemcachedNode->PooledSocket

MemcachedClient：这个类包含private IServerPool pool; pool中又有private IMemcachedNode[] allNodes; 这层关系很清晰，就是在client中通过hash算法来确定这个key应该分配到哪个MemcachedNode上执行，并且要维护有效的MemcachedNode。但很奇怪的是只有1个可用节点，为什么会分配到已经dead的node上，并且没有重连机制？？

MemcachedNode：InternalPoolImpl：client是如何判断node有效的呢，就是通过这个alive属性，而alive又是被socket决定的。在最开始的时候node创建一个池，用于维护自身的socket集合，每次执行的时候分配一个socket，用完后又将socket放回，如果执行中socket状态不正常，那么就直接pooledsocket就会抛异常，在node这层捕获，并且socket中的alive会被设置为false，node的alive是否为false是有一定的策略的，捕获到异常node会执行markasdead()，  var shouldFail = ownerNode.FailurePolicy.ShouldFail(); 会判断这个node是否还有效，从源码中可以看出是通过时间和失败次数判断的

 private void MarkAsDead()
            {
                if (_isDebugEnabled) _logger.LogDebug("Mark as dead was requested for {0}", this.endPoint);

                var shouldFail = ownerNode.FailurePolicy.ShouldFail();

                if (_isDebugEnabled) _logger.LogDebug("FailurePolicy.ShouldFail(): " + shouldFail);

                if (shouldFail)
                {
                    if (_logger.IsEnabled(LogLevel.Warning)) _logger.LogWarning("Marking node {0} as dead", this.endPoint);

                    this.isAlive = false;
                    this.markedAsDeadUtc = DateTime.UtcNow;

                    var f = this.ownerNode.Failed;

                    if (f != null)
                        f(this.ownerNode);
                }
            }

	bool INodeFailurePolicy.ShouldFail()
		{
			var now = DateTime.UtcNow;

			if (lastFailed == DateTime.MinValue)
			{
				if (LogIsDebugEnabled) log.Debug("Setting fail counter to 1.");

				failCounter = 1;
			}
			else
			{
				var diff = (int)(now - lastFailed).TotalMilliseconds;
				if (LogIsDebugEnabled) log.DebugFormat("Last fail was {0} msec ago with counter {1}.", diff, this.failCounter);

				if (diff <= this.resetAfter)
					this.failCounter++;
				else
				{
					this.failCounter = 1;
				}
			}

			lastFailed = now;

			if (this.failCounter == this.failureThreshold)
			{
				if (LogIsDebugEnabled) log.DebugFormat("Threshold reached, node will fail.");

				this.lastFailed = DateTime.MinValue;
				this.failCounter = 0;

				return true;
			}

			if (LogIsDebugEnabled) log.DebugFormat("Current counter is {0}, threshold not reached.", this.failCounter);

			return false;
		}
	}

PooledSocket：

  public PooledSocket(EndPoint endpoint, TimeSpan connectionTimeout, TimeSpan receiveTimeout, ILogger logger)
        {
            _logger = logger;

            this.isAlive = true;

            var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            // TODO test if we're better off using nagle
            //PHP: OPT_TCP_NODELAY
            //socket.NoDelay = true;

            var timeout = connectionTimeout == TimeSpan.MaxValue
                            ? Timeout.Infinite
                            : (int)connectionTimeout.TotalMilliseconds;

            var rcv = receiveTimeout == TimeSpan.MaxValue
                ? Timeout.Infinite
                : (int)receiveTimeout.TotalMilliseconds;

            socket.ReceiveTimeout = rcv;
            socket.SendTimeout = rcv;
            socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
            ConnectWithTimeout(socket, endpoint, timeout);

            this.socket = socket;
            this.endpoint = endpoint;

            this.inputStream = new BasicNetworkStream(socket);            
        }

看到这很明显，在这地方仅仅有一个keepalive来保持长连接，而没有任何的中途断开的处理机制，这也就导致服务器和memcached都在外网时，可能因为很多原因，导致连接意外断开，然而并没有1-3次左右的重连，直接被标记为dead

附上我修改后的pooledsocket.cs

//#define DEBUG_IO
using System;
using System.Linq;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Dawn.Net.Sockets;
using System.Runtime.InteropServices;
using System.Reflection;

namespace Enyim.Caching.Memcached
{
    [DebuggerDisplay("[ Address: {endpoint}, IsAlive = {IsAlive} ]")]
    public partial class PooledSocket : IDisposable
    {
        private readonly ILogger _logger;

        private bool isAlive;
        private Socket socket;
        private EndPoint endpoint;

        private Stream inputStream;
        private AsyncSocketHelper helper;
        private TimeSpan connectionTimeout;
        private TimeSpan receiveTimeout;

        public void CreateSocket()
        {
            var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            // TODO test if we're better off using nagle
            //PHP: OPT_TCP_NODELAY
            //socket.NoDelay = true;

            var timeout = connectionTimeout == TimeSpan.MaxValue
                ? Timeout.Infinite
                : (int)connectionTimeout.TotalMilliseconds;

            var rcv = receiveTimeout == TimeSpan.MaxValue
                ? Timeout.Infinite
                : (int)receiveTimeout.TotalMilliseconds;

            socket.ReceiveTimeout = rcv;
            socket.SendTimeout = rcv;
            socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
            ConnectWithTimeout(socket, endpoint, timeout);
            this.socket = socket;
            this.inputStream = new BasicNetworkStream(socket);
        }

        public void ReCreateSocket()
        {
            this.inputStream.Dispose();
            this.socket.Dispose();
            CreateSocket();
        }


        public PooledSocket(EndPoint endpoint, TimeSpan connectionTimeout, TimeSpan receiveTimeout, ILogger logger)
        {
            _logger = logger;

            this.isAlive = true;

            this.connectionTimeout = connectionTimeout;
            this.receiveTimeout = receiveTimeout;
            this.endpoint = endpoint;

            CreateSocket();

           
        }

        private void ConnectWithTimeout(Socket socket, EndPoint endpoint, int timeout)
        {
            //var task = socket.ConnectAsync(endpoint);
            //if(!task.Wait(timeout))
            //{
            //    using (socket)
            //    {
            //        throw new TimeoutException("Could not connect to " + endpoint);
            //    }
            //}  

            if (endpoint is DnsEndPoint && !RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
            {
                var dnsEndPoint = ((DnsEndPoint)endpoint);
                var host = dnsEndPoint.Host;
                var addresses = Dns.GetHostAddresses(dnsEndPoint.Host);
                var address = addresses.FirstOrDefault(ip => ip.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork);
                if (address == null)
                {
                    throw new ArgumentException(String.Format("Could not resolve host '{0}'.", host));
                }
                _logger.LogDebug($"Resolved '{host}' to '{address}'");
                endpoint = new IPEndPoint(address, dnsEndPoint.Port);
            }

            var completed = new AutoResetEvent(false);
            var args = new SocketAsyncEventArgs();
            args.RemoteEndPoint = endpoint;
            args.Completed += OnConnectCompleted;
            args.UserToken = completed;
            socket.ConnectAsync(args);
            if (!completed.WaitOne(timeout) || !socket.Connected)
            {
                using (socket)
                {
                    throw new TimeoutException("Could not connect to " + endpoint);
                }
            }

            /*
            var mre = new ManualResetEvent(false);
            socket.Connect(endpoint, iar =>
            {
                try { using (iar.AsyncWaitHandle) socket.EndConnect(iar); }
                catch { }

                mre.Set();
            }, null);

            if (!mre.WaitOne(timeout) || !socket.Connected)
                using (socket)
                    throw new TimeoutException("Could not connect to " + endpoint);
           */
        }

        private void OnConnectCompleted(object sender, SocketAsyncEventArgs args)
        {
            EventWaitHandle handle = (EventWaitHandle)args.UserToken;
            handle.Set();
        }

        public Action<PooledSocket> CleanupCallback { get; set; }

        public int Available
        {
            get { return this.socket.Available; }
        }

        public void Reset()
        {
            // discard any buffered data
            this.inputStream.Flush();

            if (this.helper != null) this.helper.DiscardBuffer();

            int available = this.socket.Available;

            if (available > 0)
            {
                if (_logger.IsEnabled(LogLevel.Warning))
                    _logger.LogWarning("Socket bound to {0} has {1} unread data! This is probably a bug in the code. InstanceID was {2}.", this.socket.RemoteEndPoint, available, this.InstanceId);

                byte[] data = new byte[available];

                this.Read(data, 0, available);

                if (_logger.IsEnabled(LogLevel.Warning))
                    _logger.LogWarning(Encoding.ASCII.GetString(data));
            }

            if (_logger.IsEnabled(LogLevel.Debug))
                _logger.LogDebug("Socket {0} was reset", this.InstanceId);
        }

        /// <summary>
        /// The ID of this instance. Used by the <see cref="T:MemcachedServer"/> to identify the instance in its inner lists.
        /// </summary>
        public readonly Guid InstanceId = Guid.NewGuid();

        public bool IsAlive
        {
            get { return this.isAlive; }
        }

        /// <summary>
        /// Releases all resources used by this instance and shuts down the inner <see cref="T:Socket"/>. This instance will not be usable anymore.
        /// </summary>
        /// <remarks>Use the IDisposable.Dispose method if you want to release this instance back into the pool.</remarks>
        public void Destroy()
        {
            this.Dispose(true);
        }

        ~PooledSocket()
        {
            try { this.Dispose(true); }
            catch { }
        }

        protected void Dispose(bool disposing)
        {
            if (disposing)
            {
                GC.SuppressFinalize(this);

                try
                {
                    if (socket != null)
                        try { this.socket.Dispose(); }
                        catch { }

                    if (this.inputStream != null)
                        this.inputStream.Dispose();

                    this.inputStream = null;
                    this.socket = null;
                    this.CleanupCallback = null;
                }
                catch (Exception e)
                {
                    _logger.LogError(nameof(PooledSocket), e);
                }
            }
            else
            {
                Action<PooledSocket> cc = this.CleanupCallback;

                if (cc != null)
                    cc(this);
            }
        }

        void IDisposable.Dispose()
        {
            this.Dispose(false);
        }

        private void CheckDisposed()
        {
            if (this.socket == null)
                throw new ObjectDisposedException("PooledSocket");
        }

        /// <summary>
        /// Reads the next byte from the server's response.
        /// </summary>
        /// <remarks>This method blocks and will not return until the value is read.</remarks>
        public int ReadByte()
        {
            this.CheckDisposed();

            try
            {
                return this.inputStream.ReadByte();
            }
            catch (IOException)
            {
                this.isAlive = false;

                throw;
            }
        }

        public async Task<byte[]> ReadBytesAsync(int count)
        {
            using (var awaitable = new SocketAwaitable())
            {
                awaitable.Buffer = new ArraySegment<byte>(new byte[count], 0, count);
                await this.socket.ReceiveAsync(awaitable);
                return awaitable.Transferred.Array;
            }
        }

        /// <summary>
        /// Reads data from the server into the specified buffer.
        /// </summary>
        /// <param name="buffer">An array of <see cref="T:System.Byte"/> that is the storage location for the received data.</param>
        /// <param name="offset">The location in buffer to store the received data.</param>
        /// <param name="count">The number of bytes to read.</param>
        /// <remarks>This method blocks and will not return until the specified amount of bytes are read.</remarks>
        public void Read(byte[] buffer, int offset, int count)
        {
            this.CheckDisposed();

            int read = 0;
            int shouldRead = count;

            while (read < count)
            {
                try
                {
                    int currentRead = this.inputStream.Read(buffer, offset, shouldRead);
                    if (currentRead < 1)
                        continue;

                    read += currentRead;
                    offset += currentRead;
                    shouldRead -= currentRead;
                }
                catch (IOException)
                {
                    this.isAlive = false;
                    throw;
                }
            }
        }

        public void Write(byte[] data, int offset, int length)
        {
            this.CheckDisposed();

            SocketError status;

            this.socket.Send(data, offset, length, SocketFlags.None, out status);

            if (status != SocketError.Success)
            {
                this.isAlive = false;

                ThrowHelper.ThrowSocketWriteError(this.endpoint, status);
            }
        }

        public void Write(IList<ArraySegment<byte>> buffers)
        {
            this.CheckDisposed();

            SocketError status;

#if DEBUG
            int total = 0;
            for (int i = 0, C = buffers.Count; i < C; i++)
                total += buffers[i].Count;

            if (this.socket.Send(buffers, SocketFlags.None, out status) != total)
                System.Diagnostics.Debugger.Break();
#else
            this.socket.Send(buffers, SocketFlags.None, out status);
#endif

            if (status != SocketError.Success)
            {
                try
                {
                    this.ReCreateSocket();
                    Write(buffers);
                }
                catch (Exception e)
                {
                    this.isAlive = false;
                    ThrowHelper.ThrowSocketWriteError(this.endpoint, status);
                }
            }
        }

        public async Task WriteSync(IList<ArraySegment<byte>> buffers)
        {
            using (var awaitable = new SocketAwaitable())
            {
                awaitable.Arguments.BufferList = buffers;
                try
                {
                    await this.socket.SendAsync(awaitable);
                }
                catch
                {
                    try
                    {
                        this.ReCreateSocket();
                        WriteSync(buffers);
                    }
                    catch (Exception e)
                    {
                        this.isAlive = false;
                        ThrowHelper.ThrowSocketWriteError(this.endpoint, awaitable.Arguments.SocketError);
                    }
                }

                if (awaitable.Arguments.SocketError != SocketError.Success)
                {
                    this.isAlive = false;
                    ThrowHelper.ThrowSocketWriteError(this.endpoint, awaitable.Arguments.SocketError);
                }
            }
        }

        /// <summary>
        /// Receives data asynchronously. Returns true if the IO is pending. Returns false if the socket already failed or the data was available in the buffer.
        /// p.Next will only be called if the call completes asynchronously.
        /// </summary>
        public bool ReceiveAsync(AsyncIOArgs p)
        {
            this.CheckDisposed();

            if (!this.IsAlive)
            {
                p.Fail = true;
                p.Result = null;

                return false;
            }

            if (this.helper == null)
                this.helper = new AsyncSocketHelper(this);

            return this.helper.Read(p);
        }
    }
}

#region [ License information          ]
/* ************************************************************
 * 
 *    Copyright (c) 2010 Attila Kisk? enyim.com
 *    
 *    Licensed under the Apache License, Version 2.0 (the "License");
 *    you may not use this file except in compliance with the License.
 *    You may obtain a copy of the License at
 *    
 *        http://www.apache.org/licenses/LICENSE-2.0
 *    
 *    Unless required by applicable law or agreed to in writing, software
 *    distributed under the License is distributed on an "AS IS" BASIS,
 *    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *    See the License for the specific language governing permissions and
 *    limitations under the License.
 *    
 * ************************************************************/
#endregion

经过我的测试，发现还是挺有趣的，就是enyim本身是有node不可用重新建立的一套机制，但这套机制可能是收到对方断开连接后才会开启一个维护线程去不断重连，如果是因为长时间未使用导致socket 不可用，node被标记为as dead后并没有触发这套机制？？