A transport-level error has occurred when sending the request to the server

http://www.diranieh.com/DataAccessPatterns/ResouceRetryer.htm

Purpose

Automatically retries operations whose failure is expected under certain defined conditions. This pattern enables fault-tolerance for data access operations.

When database operations fail, they fail because of conditions outside the application's control. These errors should not cause the application to terminate if it can reasonably recover from then. Client code should analyze the severity of the errors, recover from them as much as possible, and potentially retry operations that failed as a result.

Error recovery, analysis and retry operations is not complex, but it does benefit from a structured approach. Otherwise, this logic may clutter the application code making it more difficult to modify. The Retryer pattern describes a generic structure and logic for automatically retrying data access operations expected to fail under certain conditions, This pattern defines a clear separation between retry logic and retryable database operations.

Structure & UML

The following figure illustrates the static structure for the Retryer:

IRetryable collects data access operations that are potentially retryable. ConcreteRetryable implements this interface to define logic for each retryable operation. IRetryable.Attempt contains logic to perform the data access operation and returns true if this operation was successful, or false if an expected error occurs, or throws an exception if an unexpected error occurs. IRetryable.Attempt contains recovery logic and is called following a failed attempt. Client code does not deal with IRetryable directly, in other words, client code does not call Attempt and Recover . Instead, a Retryer class is used to encapsulate all generic retry logic.

The following figure illustrates the sequence diagram for the Retryer:

The client does not call Attempt and Recover methods directly, but rather through a Retryer which wraps the details of these two calls. The Retryer invokes ConcreteRetryable.Attempt operations and checks the return value. If the return value is true, the operation is successful and it returns control back to the client. If the return value is false, the Retryer calls ConcreteRetryable.Recover to perform any logic required to recover from the failed attempt. The Retryer may then wait for a configurable amount of time to allow the failed system to recover. After waiting, the Retryer invokes ConcreteRetryable.Attempt again. This cycle is then repeated until either an Attempt call is successful or it reaches the permitted maximum number of attempts. Usually, a small limit, say 5 retries, is sufficient.

Example

The following example implements Retryer for database query operations:

public interface IRetryable
{
bool Attempt();
void Recover();
}

public class RetryableQuery : IRetryable
{
    // Data members
    private string m_strConnectionString = "";
    private string m_strSQL = "";
    private DataSet m_ds = new DataSet();

    // Constructor
    public RetryableQuery( string strConn, string strSQL )
    {
        m_strConnectionString = strConn;
        m_strSQL = strSQL;
    }

    // IRetryable operation
    public bool Attempt()
    {
        try
        {
            SqlConnection conn = new SqlConnection( m_strConnectionString );
            SqlCommand cmd = new SqlCommand();
            cmd.CommandText = m_strSQL;
            cmd.CommandType = CommandType.Text;

            SqlDataAdapter da = new SqlDataAdapter();
            da.SelectCommand = cmd;
            da.Fill( m_ds );
            return true;
        }
        catch ( SqlException ex )
        {
            // Return false for some pre-determined error numbers
            switch (ex.Number)
            {
                case 1234:
                case 5678:
                case 1122:
                    return false;
                default:
                    throw new InvalidOperationException( "Operation has failed" );
            }
        }
    }

    public void Recover()
    {
        // Empty implementation. Nothing to do in this example
    }

    // Public interface
    public DataSet GetResult()
    {
        return m_ds;
    }
}

public class Retryer
{
    private IRetryable m_obRetryable;
    private int          m_nMaxRetryies = 5;    // Typically obtained from config files or database

    public Retryer(IRetryable retryable)
    {
        m_obRetryable = retryable;
    }

    // Retry the operation until is succeeds or retries reach the maximum allowed retry-limit
    public bool Invoke()
    {
        for ( int i =0; i < m_nMaxRetryies; i ++ )
        {
            if (m_obRetryable.Attempt() )
                return true;
            else
                m_obRetryable.Recover() ;
        }
        return false;
    }
}

// Client code
void InvokeQuery()
{
    // Create retryable objects
    RetryableQuery ob = new RetryableQuery ( "SomeConnection", "select * from T1" );
    Retryer obRetryer = new Retryer ( ob );

    // Invoke a retryable method
    bool bRet = obRetryer.Invoke();

    // Get data if operation succeeded
    DataSet ds = null;
    if (bRet)
        ds = ob.GetResult();
}

Applicability

Use this pattern when:

You want your applications to react to certain, defined error conditions without terminating.
Your application cannot detect certain recoverable error conditions ahead of time, and you want the ability to react to them after they occur.
You want to decouple retry logic from data access code.

Strategies / Variants

Consider these strategies when designing a retryer.

Command semantics
Retryer is a specialization of the Command pattern . As a result, you must define data access operations using command semantics. Rather than creating data access or resource objects and then invoking their methods, you need to group a sequence of related operations into a single Retryable implementation.

The Retryer class is responsible for invoking Attempt and Recover as needed, and client code is responsible for interacting with the Retryer 's input and output parameters directly. This is because Attempt and Recover do not and should not take any parameters that relate to data access logic, instead, input parameters are passed through constructors or set operations. The following template illustrates:

public class RetryableOpen : IRetryable
{
    private string m_strConnection;

    RetryableOpen( string str )
    {
        m_strConnection = str
    }

    // IRetryable
    public bool Attempt()
    {
        try
        {

        }
        catch( SomeExpectedException )
        {
             return false;
        }
    }

    public void Recover()
    {
        ...
    }

    // Properties
    public string ConnectionString
    {
        set { m_strConnection = value; }
        get { return m_strConnection; }
    }
}

Success and Failure Definition
A retryable implementation class strictly defines its own notion of success and failure. There may be occasions where operations fail due to some unexpected reason but they do not necessarily need to be retryed. In most cases, retrying an operation when an unexpected exception occurs is futile. Retryable implementations indicate this fact by throwing an exception back to the client.
Configuration
It is recommended that configuration mechanisms be used to control settings such as maximum number of retries, waiting period between retries and so on.
Error Analysis
Error analysis is fundamental with a concrete retryable implementation since it needs to distinguish anticipated errors from serious failures. Resource Descriptor can be used to isolate specific error codes for each database platform you support.

Benefits

Encapsulation
The Retriever pattern encapsulates all retry logic keeping it separated from the application and data access logic. If you redefine or re-implement retry logic, then you only need to change the code in the retry module.
Clearly defined operations
Retryable implementations define a data access operation characteristics clearly. This includes logic, expected error conditions, and corresponding recover logic.