Download SocketAsyncServerAndClient.zip - 200.2 KB
Introduction
Writing fast, scalable socket server code for TCP/IP is not easy. In order to help you write scalable, high performance socket server code, Microsoft created the SocketAsyncEventArgs class. A proven way to write scalable, high performance socket code for TCP/IP in Windows can be seen in this article on I/O completion ports. Also, here's a link to a Microsoft page on I/O Completion Ports. SocketAsyncEventArgs uses I/O Completion ports via the asynchronous methods in the .NET Socket class. SocketAsyncEventArgs object allows us to access the accepting socket with all the functionality of the SocketAsyncEventArgs class, like working asynchronously, raising the Completed event, setting buffer space, etc.
But there are other issues that need to be addressed in order to create dependable socket server code. One issue is buffers. Buffers in TCP are unmanaged, that is, not controlled by the .NET Framework, but by Windows system. So the buffer gets "pinned" to one place in memory, thereby causing memory fragmentation, since the .NET GarbageCollector will not be able to collect that space. This situation is improved by putting all the buffers together in one block of memory, and just reusing that same space over and over. The example in the BufferManager code on Microsoft's page for the SetBuffer method shows how to build it.
Background
You may have started your research into this topic at Microsoft's main page for the SocketAsyncEventArgs class. The example code on that page got me started. But it was also confusing. It seems that they have removed some example code about the UserToken property. Also, some of the method names were a bit confusing, as were some of the variable names. Their reason for using a Semaphore was not explained really. While their code for building the BufferManager was good, the way that they dealt with the SetBuffer method in their ProcessReceive method in example code for theSocketAsyncEventArgs class will pretty much work in only the narrowest of examples. If you send a 10 byte string, and then a 20 byte string, it won't work. Their code sets the buffer to be whatever size you send on the first message from the client. So after the first message it would just send back the first 10 bytes. So, we need a better example of how to get the data and use it. Plus, in the explanation of the example code they said "For example, if a server application needs to have 15 socket accept operations outstanding at all times to support incoming client connection rates, it can allocate 15 reusable SocketAsyncEventArgs objects for that purpose." But then their example only included reusable SocketAsyncEventArgs objects for receive/send, not for accept. The SocketAsyncEventArgs object for the accept operation would wait until the receive/send finished to do another accept op. Instead we can use a pool, as they mentioned in their explanation, and post accept operations faster.
So, this article was done in an attempt to make it clearer how to use the SocketAsyncEventArgs class. The code in this article was developed on Visual Studio 2008 using .NET 3.5. Pay special attention to the code related to buffers, as buffer-related stuff seems to be an area where people have more difficulty. This article assumes some knowledge of delegates and event handling in Windows.
Regarding the SocketAsyncEventArgs class, Microsoft's website says it requires "Platforms: Windows 7, Windows Vista, Windows XP SP3, Windows Server 2008, Windows Server 2003. (The) .NET Framework Supported in: 4, 3.5 SP1, 3.0 SP1, 2.0 SP1. (The) .NET Framework Client Profile Supported in: 4, 3.5 SP1."
TCP Protocol and Socket Basics
If you have experience with socket server code, you can skip this section. For those new to socket programming, there are four main steps in using a socket server with TCP. (It's often described as six parts, but I like to put the first three together into one.)
1) Listen for connection requests
In order to listen you need to
(a) create a socket.
(b) bind that socket to a port,
(c) listen with that socket.
A client (not a server) can initiate a connection request, by sending a SYN packet. The client does not listen for incoming connections. It always initiates connections by using the Connect or ConnectAsync method. When a client initiates a connection, then the Windows TCP/IP subsystem of the server will respond with SYN, ACK. After the client machine's Windows TCP/IP subsystem responds back with an ACK packet, the connection is established. Windows will handle this TCP/IP stuff for you. In other words, SYN, ACK, PSH, packets, and similar parts of TCP/IP do not have to be coded by you. Very nice. The server's listening socket can maintain a queue of connection requests waiting to be accepted. This queue is called the "backlog". The listening socket passes the connection info to another socket via an "accept" operation, and then gets the next incoming connection in the backlog queue, or if there is none, waits till there is a new connection from a client.
2) Accept connection requests
In order to have multiple connections on the same port, the listening socket must pass off the connection info to another socket, which accepts it. The accepting socket is not bound to the port. You post an accept operation to pass the connection from the listening socket to the accepting socket. The accept operation can be posted before the incoming connection is established, so that the listening socket immediately passes off the new connection info to the accepting socket. The client does not need to perform an accept operation.
3) Receive/Send via the connection
After the accept operation has completed you can now receive or send data with that connection. (The same SocketAsyncEventArgs object that did the accept operation could also do the receiving or sending, if we post a receive or send on it and have buffer space for it.) In the design of the code below, the SocketAsyncEventArgs which did the accept operation passes the connection info over to another SocketAsyncEventArgs object to do receiving/sending. (We could also split the receiving and sending into 2 separate SocketAsyncEventArgs objects, if we wish. One reason to do that might be a need for different buffer sizes for send vs. receive. We would just need to have the connection info in both the SocketAsyncEventArgsobject that sends and the SocketAsyncEventArgs that receives. And they might need a reference to each other also.)
4) Close the connection
Either client or server can initiate an operation to close the connection. Usually the client would initiate that. Again, the lower level TCP/IP of the disconnect is handled by Windows operating system. The connection can be closed using the Close method, which destroys the Socket and cleans up its managed and unmanaged resources. You can also close by using the Disconnect or DisconnectAsync method, which allows reuse of the Socket.
With TCP there is no guarantee that one send operation on the client will be equal to one receive operation on the server. One send operation on the client might be equal to one, two or more receive operations on the server. And the same is true going back to client from server. So you must have some way of determining where a TCP message begins and/or ends. Three possible ways to handle TCP messages are:
1) Prefix every message with an integer that tells the length of the message.
2) All messages be fixed length. And both client and server must know the length before the message is sent.
3) Append every message with a delimiter to show where it ends. And both client and server must know what the delimiter is before the message is sent.
Also, your communications protocol must include whether there will be a response (send operation) from the server back to the client after each received message or not. Will that response be after one complete received TCP message, or can it be after more than one message? If it is after one message, the code is simpler probably.
Okay, so let's think about the possible situations that might occur with the data that the server receives in one receive operation with our communication protocol.
1) On the first receive op, receive less bytes than the length of the prefix.
2) After having received part of the prefix on a previous receive op or ops, then receive another part of the prefix, but not all of it.
3) After having received part of the prefix on a previous receive op or ops, then receive the rest of the prefix, but nothing more.
4) After having received part of the prefix on a previous receive op or ops, we then receive the rest of it, plus part of the message.
5) After having received part of the prefix on a previous receive op or ops, we then receive the rest of it, plus all of the message.
6) Receive exactly the number of bytes that are in the prefix, but nothing more.
7) After having received exactly the number of bytes that are in the prefix on a previous receive op or ops, we then receive part of the message.
8) After having received exactly the number of bytes that are in the prefix on a previous receive op or ops, we then receive all of the message.
9) Receive the number of bytes for the prefix plus part of the message, but NOT all of the message.
10) After having received the prefix and part of the message on a previous receive op or ops, we then receive another part of the message, but not all of it.
11) After having received the prefix and part of the message on a previous receive op or ops, we then receive all the rest of the message.
12) Receive the number of bytes for the prefix plus all of the message on the first receive op. This is actually the most common thing that will happen. But all of the above things can happen and do happen. If both client and server have buffer sizes larger than the messages, then the situations above may not happen when running both client and server on the same machine, or even on a LAN. But TCP is unpredictable over the Internet. So your code needs to allow for all of those possibilities.
Let's look at some code
Accept operations. In this app the socket which does the accept can be accessed thru a
SocketAsyncEventArgs
object, in its
AcceptSocketproperty. On Microsoft's
AcceptSocket
page it says, "If not supplied (set to null) before calling the
Socket.AcceptAsync method, a new socket will be created automatically." That's what is done in the code below. We just allow a new Socket object to be created for every new connection. (There is the option of reusing sockets in .NET and having a pool of Socket objects, but so far I have not found it to yield a significant advantage.) We can have a pool of these
SocketAsyncEventArgs
objects, each one containing a separate Socket object, to deal with accept operations. In this pool you do not need one object for each connection the server is maintaining, because after the accept operation completes, a reference to the socket is handed off to another
SocketAsyncEventArgs
object pretty fast. It does
not
seem to help to put a lot of
SocketAsyncEventArgs
objects in the pool. Again, repeating for clarity, the socket which does the accept operation is in
SocketAsyncEventArgs.
AcceptSocket
property of the
SocketAsyncEventArgs
objects that comes out of the pool of
SocketAsyncEventArgs
objects that we create for accept operations.
Receive/Send operations. In this app the receive and send operations are handled via SocketAsyncEventArgs objects that come out of a pool ofSocketAsyncEventArgs objects that we create for receive/send operations. This is NOT the same pool as we just examined regarding accept operations. To improve performance we have a pool of these objects which do receive and send operations. The number of SocketAsyncEventArgsobjects in the pool for receive/send operations should probably be equal to the maximum number of concurrent connections allowed.
What is our communication protocol in this code?
1) One message from client will correspond with one message from the server.
2) After a connection is made the client will send a message first, and then post a receive op to wait for the response from the server. And for each message that the client sends the server will respond with a message to the client. Then it will post another receive op and wait for the next message from the client. In our code the server will make a few changes to the data before responding to the client, so that we do more than just echo data sent by the client, which is what the Microsoft example does. That approach should help you see what happens with the buffers better than just echoing would.
3) We will prefix every message with an integer that tells the length of the message.
Okay, let's look at the server code. Sometimes in code comments I abbreviate "SocketAsyncEventArgs" as "SAEA".
Collapse
class Program
{
public static readonly Int32 testBufferSize = 25;
public static readonly Int32 maxSimultaneousAcceptOps = 4;
public static readonly Int32 backlog = 100;
public static readonly Int32 opsToPreAlloc = 2;
public static readonly Int32 maxNumberOfConnections = 100;
public static bool respondToClient = true;
public static readonly Int32 port = 4444;
public static readonly Int32 receivePrefixLength = 4;
public static readonly Int32 sendPrefixLength = 4;
public static Int32 mainTransMissionId = 10001;
public static object lockerForTid = new object();
public static List<DataHolder> listOfDataHolders;
public static object lockerForList = new object();
public static Int32 mainSessionId = 1000000001;
public static object lockerForSesId = new object();
public static readonly Int32 msDelayAfterGettingMessage = -1;
static void Main(String[] args)
{
try
{
IPAddress[] addressList =
Dns.GetHostEntry(Environment.MachineName).AddressList;
IPEndPoint localEndPoint =
new IPEndPoint(addressList[addressList.Length - 1], port);
SocketListenerSettings theSocketListenerSettings =
new SocketListenerSettings(maxNumberOfConnections, backlog,
maxSimultaneousAcceptOps, receivePrefixLength,
testBufferSize, sendPrefixLength, opsToPreAlloc, localEndPoint);
SocketListener socketListener =
new SocketListener(theSocketListenerSettings);
string closeString = "z";
string closeCheck = "";
while (closeCheck != closeString)
{
Console.WriteLine("type '" + closeString
+ "' and press Enter to close the program.");
closeCheck = Console.ReadLine();
}
}
catch (IndexOutOfRangeException)
{
Console.WriteLine("port invalid");
}
catch (FormatException)
{
Console.WriteLine("port invalid");
}
catch (Exception ex)
{
Console.WriteLine("Error: " + ex.Message);
}
}
}
The primary class is SocketListener.
Collapse
class SocketListener
{
BufferManager theBufferManager;
Socket listenSocket;
Semaphore theMaxConnectionsEnforcer;
SocketListenerSettings socketListenerSettings;
SocketAsyncEventArgsPool poolOfAcceptEventArgs;
SocketAsyncEventArgsPool poolOfRecSendEventArgs;
public SocketListener(SocketListenerSettings theSocketListenerSettings)
{
this.socketListenerSettings = theSocketListenerSettings;
this.theBufferManager =
new BufferManager(this.socketListenerSettings.BufferSize *
this.socketListenerSettings.MaxConnections *
this.socketListenerSettings.OpsToPreAllocate,
this.socketListenerSettings.BufferSize *
this.socketListenerSettings.OpsToPreAllocate);
this.poolOfRecSendEventArgs =
new SocketAsyncEventArgsPool(this.socketListenerSettings.MaxConnections);
this.poolOfAcceptEventArgs =
new SocketAsyncEventArgsPool(this.socketListenerSettings.MaxAcceptOps);
this.theMaxConnectionsEnforcer =
new Semaphore(this.socketListenerSettings.MaxConnections + 1,
this.socketListenerSettings.MaxConnections + 1);
Init();
StartListen();
}
internal void Init()
{
this.theBufferManager.InitBuffer();
for (Int32 i = 0; i < this.socketListenerSettings.MaxAcceptOps; i++)
{
this.poolOfAcceptEventArgs.Push(CreateNewSaeaForAccept(poolOfAcceptEventArgs));
}
SocketAsyncEventArgs eventArgObjectForPool;
for (Int32 i = 0; i < this.socketListenerSettings.MaxConnections; i++)
{
eventArgObjectForPool = new SocketAsyncEventArgs();
this.theBufferManager.SetBuffer(eventArgObjectForPool);
eventArgObjectForPool.Completed +=
new EventHandler<SocketAsyncEventArgs>(IO_Completed);
DataHoldingUserToken theTempReceiveSendUserToken =
new DataHoldingUserToken(eventArgObjectForPool,
eventArgObjectForPool.Offset, eventArgObjectForPool.Offset +
this.socketListenerSettings.BufferSize,
this.socketListenerSettings.ReceivePrefixLength,
this.socketListenerSettings.SendPrefixLength, i + 1);
theTempReceiveSendUserToken.CreateNewDataHolder();
eventArgObjectForPool.UserToken = theTempReceiveSendUserToken;
this.poolOfRecSendEventArgs.Push(eventArgObjectForPool);
}
}
internal SocketAsyncEventArgs CreateNewSaeaForAccept(SocketAsyncEventArgsPool pool)
{
SocketAsyncEventArgs acceptEventArg = new SocketAsyncEventArgs();
acceptEventArg.Completed +=
new EventHandler<SocketAsyncEventArgs>(AcceptEventArg_Completed);
AcceptOpUserToken theAcceptOpToken = new AcceptOpUserToken(pool.GetTokenId());
acceptEventArg.UserToken = theAcceptOpToken;
return acceptEventArg;
}
internal void StartListen()
{
listenSocket = new
Socket(this.socketListenerSettings.LocalEndPoint.AddressFamily,
SocketType.Stream, ProtocolType.Tcp);
listenSocket.Bind(this.socketListenerSettings.LocalEndPoint);
listenSocket.Listen(this.socketListenerSettings.Backlog);
StartAccept();
}
internal void StartAccept()
{
SocketAsyncEventArgs acceptEventArg;
if (this.poolOfAcceptEventArgs.Count > 1)
{
try
{
acceptEventArg = this.poolOfAcceptEventArgs.Pop();
}
catch
{
acceptEventArg = CreateNewSaeaForAccept(poolOfAcceptEventArgs);
}
}
else
{
acceptEventArg = CreateNewSaeaForAccept(poolOfAcceptEventArgs);
}
this.theMaxConnectionsEnforcer.WaitOne();
bool willRaiseEvent = listenSocket.AcceptAsync(acceptEventArg);
if (!willRaiseEvent)
{
ProcessAccept(acceptEventArg);
}
}
private void AcceptEventArg_Completed(object sender, SocketAsyncEventArgs e)
{
ProcessAccept(e);
}
private void ProcessAccept(SocketAsyncEventArgs acceptEventArgs)
{
LoopToStartAccept();
SocketAsyncEventArgs receiveSendEventArgs = this.poolOfRecSendEventArgs.Pop();
((DataHoldingUserToken)receiveSendEventArgs.UserToken).CreateSessionId();
receiveSendEventArgs.AcceptSocket = acceptEventArgs.AcceptSocket;
acceptEventArgs.AcceptSocket = null;
this.poolOfAcceptEventArgs.Push(acceptEventArgs);
StartReceive(receiveSendEventArgs);
}
private void LoopToStartAccept()
{
StartAccept();
}
private void StartReceive(SocketAsyncEventArgs receiveSendEventArgs)
{
bool willRaiseEvent =
receiveSendEventArgs.AcceptSocket.ReceiveAsync(receiveSendEventArgs);
new EventHandler<SocketAsyncEventArgs>(IO_Completed);
if (!willRaiseEvent)
{
ProcessReceive(receiveSendEventArgs);
}
}
void IO_Completed(object sender, SocketAsyncEventArgs e)
{
DataHoldingUserToken receiveSendToken = (DataHoldingUserToken)e.UserToken;
switch (e.LastOperation)
{
case SocketAsyncOperation.Receive:
ProcessReceive(e);
break;
case SocketAsyncOperation.Send:
ProcessSend(e);
break;
default:
throw new ArgumentException("The last operation completed on
the socket was not a receive or send");
}
}
private void ProcessReceive(SocketAsyncEventArgs e)
{
DataHoldingUserToken receiveSendToken = (DataHoldingUserToken)e.UserToken;
bool incomingTcpMessageIsReady = false;
if (e.BytesTransferred > 0 && e.SocketError == SocketError.Success)
{
Int32 recPrefixBytesDoneThisOp = 0;
Int32 remainingBytesToProcess = e.BytesTransferred;
if (receiveSendToken.receivedPrefixBytesDoneCount <
this.socketListenerSettings.ReceivePrefixLength)
{
if (receiveSendToken.receivedPrefixBytesDoneCount == 0)
{
receiveSendToken.byteArrayForPrefix =
new Byte[this.socketListenerSettings.ReceivePrefixLength];
}
if (remainingBytesToProcess >=
this.socketListenerSettings.ReceivePrefixLength -
receiveSendToken.receivedPrefixBytesDoneCount)
{
Buffer.BlockCopy(e.Buffer, receiveSendToken.receiveMessageOffset
- this.socketListenerSettings.ReceivePrefixLength
+ receiveSendToken.receivedPrefixBytesDoneCount,
receiveSendToken.byteArrayForPrefix,
receiveSendToken.receivedPrefixBytesDoneCount,
this.socketListenerSettings.ReceivePrefixLength
- receiveSendToken.receivedPrefixBytesDoneCount);
remainingBytesToProcess = remainingBytesToProcess
- this.socketListenerSettings.ReceivePrefixLength
+ receiveSendToken.receivedPrefixBytesDoneCount;
recPrefixBytesDoneThisOp =
this.socketListenerSettings.ReceivePrefixLength
- receiveSendToken.receivedPrefixBytesDoneCount;
receiveSendToken.receivedPrefixBytesDoneCount =
this.socketListenerSettings.ReceivePrefixLength;
receiveSendToken.lengthOfCurrentIncomingMessage
= BitConverter.ToInt32(receiveSendToken.byteArrayForPrefix, 0);
}
else
{
Buffer.BlockCopy(e.Buffer, receiveSendToken.receiveMessageOffset
- socketListenerSettings.ReceivePrefixLength
+ receiveSendToken.receivedPrefixBytesDoneCount,
receiveSendToken.byteArrayForPrefix,
receiveSendToken.receivedPrefixBytesDoneCount,
remainingBytesToProcess);
recPrefixBytesDoneThisOp = remainingBytesToProcess
- receiveSendToken.receivedPrefixBytesDoneCount;
receiveSendToken.receivedPrefixBytesDoneCount +=
remainingBytesToProcess;
remainingBytesToProcess = 0;
receiveSendToken.receiveMessageOffset =
receiveSendToken.receiveMessageOffset
- receiveSendToken.receivedPrefixBytesDoneCount;
StartReceive(e);
}
}
if ((receiveSendToken.receivedPrefixBytesDoneCount ==
this.socketListenerSettings.ReceivePrefixLength)
& (remainingBytesToProcess > 0))
{
if (receiveSendToken.theDataHolder.dataMessageReceived == null)
{
receiveSendToken.theDataHolder.dataMessageReceived =
new Byte[receiveSendToken.lengthOfCurrentIncomingMessage];
}
if (remainingBytesToProcess +
receiveSendToken.receivedMessageBytesDoneCount ==
receiveSendToken.lengthOfCurrentIncomingMessage)
{
Buffer.BlockCopy(e.Buffer, receiveSendToken.receiveMessageOffset,
receiveSendToken.theDataHolder.dataMessageReceived,
receiveSendToken.receivedMessageBytesDoneCount,
remainingBytesToProcess);
incomingTcpMessageIsReady = true;
receiveSendToken.receivedPrefixBytesDoneCount = 0;
receiveSendToken.receivedMessageBytesDoneCount = 0;
remainingBytesToProcess = 0;
receiveSendToken.receiveMessageOffset =
receiveSendToken.bufferOffsetReceive
+ this.socketListenerSettings.ReceivePrefixLength;
}
else if (remainingBytesToProcess
+ receiveSendToken.receivedMessageBytesDoneCount
< receiveSendToken.lengthOfCurrentIncomingMessage)
{
Buffer.BlockCopy(e.Buffer, receiveSendToken.receiveMessageOffset,
receiveSendToken.theDataHolder.dataMessageReceived,
receiveSendToken.receivedMessageBytesDoneCount,
remainingBytesToProcess);
receiveSendToken.receiveMessageOffset =
receiveSendToken.receiveMessageOffset
- recPrefixBytesDoneThisOp;
receiveSendToken.receivedMessageBytesDoneCount +=
remainingBytesToProcess;
remainingBytesToProcess = 0;
StartReceive(e);
}
}
if (incomingTcpMessageIsReady == true)
{
receiveSendToken.theMediator
.HandleData(receiveSendToken.theDataHolder);
receiveSendToken.CreateNewDataHolder();
incomingTcpMessageIsReady = false;
receiveSendToken.theMediator.PrepareOutgoingData();
StartSend(receiveSendToken.theMediator.GiveBack());
}
}
else
{
CloseClientSocket(e);
}
}
private void StartSend(SocketAsyncEventArgs receiveSendEventArgs)
{
DataHoldingUserToken receiveSendToken =
(DataHoldingUserToken)receiveSendEventArgs.UserToken;
if (receiveSendToken.sendBytesRemainingCount
<= this.socketListenerSettings.BufferSize)
{
receiveSendEventArgs.SetBuffer(receiveSendToken.bufferOffsetSend,
receiveSendToken.sendBytesRemainingCount);
Buffer.BlockCopy(receiveSendToken.dataToSend,
receiveSendToken.bytesSentAlreadyCount,
receiveSendEventArgs.Buffer, receiveSendToken.bufferOffsetSend,
receiveSendToken.sendBytesRemainingCount);
}
else
{
receiveSendEventArgs.SetBuffer(receiveSendToken.bufferOffsetSend,
this.socketListenerSettings.BufferSize);
Buffer.BlockCopy(receiveSendToken.dataToSend,
receiveSendToken.bytesSentAlreadyCount,
receiveSendEventArgs.Buffer, receiveSendToken.bufferOffsetSend,
this.socketListenerSettings.BufferSize);
}
bool willRaiseEvent =
receiveSendEventArgs.AcceptSocket.SendAsync(receiveSendEventArgs);
if (!willRaiseEvent)
{
ProcessSend(receiveSendEventArgs);
}
}
private void ProcessSend(SocketAsyncEventArgs receiveSendEventArgs)
{
DataHoldingUserToken receiveSendToken =
(DataHoldingUserToken)receiveSendEventArgs.UserToken;
receiveSendToken.sendBytesRemainingCount =
receiveSendToken.sendBytesRemainingCount
- receiveSendEventArgs.BytesTransferred;
receiveSendToken.bytesSentAlreadyCount +=
receiveSendEventArgs.BytesTransferred;
if (receiveSendEventArgs.SocketError == SocketError.Success)
{
if (receiveSendToken.sendBytesRemainingCount == 0)
{
receiveSendEventArgs.SetBuffer(receiveSendToken.bufferOffsetReceive,
this.socketListenerSettings.BufferSize);
StartReceive(receiveSendEventArgs);
}
else
{
StartSend(receiveSendEventArgs);
}
}
else
{
CloseClientSocket(receiveSendEventArgs);
}
}
private void CloseClientSocket(SocketAsyncEventArgs e)
{
DataHoldingUserToken receiveSendToken = (DataHoldingUserToken)e.UserToken;
try
{
e.AcceptSocket.Shutdown(SocketShutdown.Send);
}
catch
{
}
e.AcceptSocket.Close();
if (receiveSendToken.theDataHolder.dataMessageReceived != null)
{
receiveSendToken.CreateNewDataHolder();
}
this.poolOfRecSendEventArgs.Push(e);
this.theMaxConnectionsEnforcer.Release();
}
Collapse
class BufferManager
{
Int32 totalBytesInBufferBlock;
byte[] bufferBlock;
Stack<int> freeIndexPool;
Int32 currentIndex;
Int32 bufferSize;
public BufferManager(Int32 totalBytes, Int32 bufferSize)
{
totalBytesInBufferBlock = totalBytes;
this.currentIndex = 0;
this.bufferSize = bufferSize;
this.freeIndexPool = new Stack<int>();
}
internal void InitBuffer()
{
this.bufferBlock = new byte[totalBytesInBufferBlock];
}
internal bool SetBuffer(SocketAsyncEventArgs args)
{
if (this.freeIndexPool.Count > 0)
{
args.SetBuffer(this.bufferBlock, this.freeIndexPool.Pop(),
this.bufferSize);
}
else
{
if ((totalBytesInBufferBlock - this.bufferSize) < this.currentIndex)
{
return false;
}
args.SetBuffer(this.bufferBlock, this.currentIndex, this.bufferSize);
this.currentIndex += this.bufferSize;
}
return true;
}
internal void FreeBuffer(SocketAsyncEventArgs args)
{
this.freeIndexPool.Push(args.Offset);
args.SetBuffer(null, 0, 0);
}
}
Collapse
class DataHoldingUserToken
{
internal Mediator theMediator;
internal DataHolder theDataHolder;
internal Int32 bufferOffsetReceive;
internal Int32 bufferOffsetSend;
internal Int32 lengthOfCurrentIncomingMessage;
internal Int32 receiveMessageOffset;
internal Byte[] byteArrayForPrefix;
internal Int32 receivePrefixLength;
internal Int32 receivedPrefixBytesDoneCount = 0;
internal Int32 receivedMessageBytesDoneCount = 0;
internal Int32 sendBytesRemainingCount;
internal Int32 sendPrefixLength;
internal Byte[] dataToSend;
internal Int32 bytesSentAlreadyCount;
private Int32 sessionId;
public DataHoldingUserToken(SocketAsyncEventArgs e,
Int32 rOffset, Int32 sOffset,
Int32 receivePrefixLength, Int32 sendPrefixLength, Int32 identifier)
{
this.theMediator = new Mediator(e);
this.bufferOffsetReceive = rOffset;
this.bufferOffsetSend = sOffset;
this.receivePrefixLength = receivePrefixLength;
this.sendPrefixLength = sendPrefixLength;
this.receiveMessageOffset = rOffset + receivePrefixLength;
}
public Int32 TokenId
{
get
{
return this.idOfThisObject;
}
}
internal void CreateNewDataHolder()
{
theDataHolder = new DataHolder();
}
internal void CreateSessionId()
{
lock (Program.lockerForSesId)
{
sessionId = Program.mainSessionId;
Program.mainSessionId++;
}
}
public Int32 SessionId
{
get
{
return this.sessionId;
}
}
}
Collapse
class Mediator
{
private IncomingDataPreparer theIncomingDataPreparer;
private OutgoingDataPreparer theOutgoingDataPreparer;
private DataHolder theDataHolder;
private SocketAsyncEventArgs saeaObject;
public Mediator(SocketAsyncEventArgs e)
{
this.saeaObject = e;
this.theIncomingDataPreparer = new IncomingDataPreparer(saeaObject);
this.theOutgoingDataPreparer = new OutgoingDataPreparer();
}
internal void HandleData(DataHolder incomingDataHolder)
{
theDataHolder = theIncomingDataPreparer.HandleReceivedData
(incomingDataHolder, this.saeaObject);
}
internal void PrepareOutgoingData()
{
theOutgoingDataPreparer.PrepareOutgoingData(saeaObject, theDataHolder);
}
internal SocketAsyncEventArgs GiveBack()
{
return saeaObject;
}
}
Collapse
class IncomingDataPreparer
{
private DataHolder theDataHolder;
private SocketAsyncEventArgs theSaeaObject;
public IncomingDataPreparer(SocketAsyncEventArgs e)
{
this.theSaeaObject = e;
}
private Int32 ReceivedTransMissionIdGetter()
{
Int32 receivedTransMissionId;
lock (Program.lockerForTid)
{
receivedTransMissionId = Program.mainTransMissionId;
Program.mainTransMissionId++;
}
return receivedTransMissionId;
}
private EndPoint GetRemoteEndpoint()
{
return this.theSaeaObject.AcceptSocket.RemoteEndPoint;
}
internal DataHolder HandleReceivedData(DataHolder incomingDataHolder,
SocketAsyncEventArgs theSaeaObject)
{
DataHoldingUserToken receiveToken =
(DataHoldingUserToken)theSaeaObject.UserToken;
theDataHolder = incomingDataHolder;
theDataHolder.sessionId = receiveToken.SessionId;
theDataHolder.receivedTransMissionId = this.ReceivedTransMissionIdGetter();
theDataHolder.remoteEndpoint = this.GetRemoteEndpoint();
this.AddDataHolder();
return theDataHolder;
}
private void AddDataHolder()
{
lock (Program.lockerForList)
{
Program.listOfDataHolders.Add(theDataHolder);
}
}
}
Collapse
class OutgoingDataPreparer
{
private DataHolder theDataHolder;
internal void PrepareOutgoingData(SocketAsyncEventArgs e,
DataHolder handledDataHolder)
{
DataHoldingUserToken theUserToken = (DataHoldingUserToken)e.UserToken;
theDataHolder = handledDataHolder;
Byte[] idByteArray = BitConverter.GetBytes
(theDataHolder.receivedTransMissionId);
Int32 lengthOfCurrentOutgoingMessage = idByteArray.Length
+ theDataHolder.dataMessageReceived.Length;
Byte[] arrayOfBytesInPrefix = BitConverter.GetBytes
(lengthOfCurrentOutgoingMessage);
theUserToken.dataToSend = new Byte[theUserToken.sendPrefixLength
+ lengthOfCurrentOutgoingMessage];
Buffer.BlockCopy(arrayOfBytesInPrefix, 0, theUserToken.dataToSend,
0, theUserToken.sendPrefixLength);
Buffer.BlockCopy(idByteArray, 0, theUserToken.dataToSend,
theUserToken.sendPrefixLength, idByteArray.Length);
Buffer.BlockCopy(theDataHolder.dataMessageReceived, 0,
theUserToken.dataToSend, theUserToken.sendPrefixLength
+ idByteArray.Length, theDataHolder.dataMessageReceived.Length);
theUserToken.sendBytesRemainingCount =
theUserToken.sendPrefixLength + lengthOfCurrentOutgoingMessage;
theUserToken.bytesSentAlreadyCount = 0;
}
}
Collapse
class DataHolder
{
internal Byte[] dataMessageReceived;
internal Int32 receivedTransMissionId;
internal Int32 sessionId;
internal EndPoint remoteEndpoint;
}
Collapse
internal sealed class SocketAsyncEventArgsPool
{
private Int32 nextTokenId = 1000001;
object lockerForTokenId = new object();
Stack<SocketAsyncEventArgs> pool;
internal SocketAsyncEventArgsPool(Int32 capacity)
{
this.pool = new Stack<SocketAsyncEventArgs>(capacity);
}
internal Int32 Count
{
get { return this.pool.Count; }
}
internal Int32 GetTokenId()
{
Int32 tokenId = 0;
lock (lockerForTokenId)
{
tokenId = nextTokenId;
nextTokenId++;
}
return tokenId;
}
internal SocketAsyncEventArgs Pop()
{
lock (this.pool)
{
return this.pool.Pop();
}
}
internal void Push(SocketAsyncEventArgs item)
{
if (item == null)
{
throw new ArgumentNullException("Items added to a
SocketAsyncEventArgsPool cannot be null");
}
lock (this.pool)
{
this.pool.Push(item);
}
}
}
Collapse
class SocketListenerSettings
{
private Int32 maxConnections;
private Int32 backlog;
private Int32 maxSimultaneousAcceptOps;
private Int32 receiveBufferSize;
private Int32 receivePrefixLength;
private Int32 sendPrefixLength;
private Int32 opsToPreAllocate;
private IPEndPoint localEndPoint;
public SocketListenerSettings(Int32 maxConnections, Int32 backlog,
Int32 maxSimultaneousAcceptOps, Int32 receivePrefixLength,
Int32 receiveBufferSize, Int32 sendPrefixLength,
Int32 opsToPreAlloc, IPEndPoint theLocalEndPoint)
{
this.maxConnections = maxConnections;
this.backlog = backlog;
this.maxSimultaneousAcceptOps = maxSimultaneousAcceptOps;
this.receivePrefixLength = receivePrefixLength;
this.receiveBufferSize = receiveBufferSize;
this.sendPrefixLength = sendPrefixLength;
this.opsToPreAllocate = opsToPreAlloc;
this.localEndPoint = theLocalEndPoint;
}
public Int32 MaxConnections
{
get
{
return this.maxConnections;
}
}
public Int32 Backlog
{
get
{
return this.backlog;
}
}
public Int32 MaxAcceptOps
{
get
{
return this.maxSimultaneousAcceptOps;
}
}
public Int32 ReceivePrefixLength
{
get
{
return this.receivePrefixLength;
}
}
public Int32 BufferSize
{
get
{
return this.receiveBufferSize;
}
}
public Int32 SendPrefixLength
{
get
{
return this.sendPrefixLength;
}
}
public Int32 OpsToPreAllocate
{
get
{
return this.opsToPreAllocate;
}
}
public IPEndPoint LocalEndPoint
{
get
{
return this.localEndPoint;
}
}
}
The server app
After downloading the zip file that contains the code, in order not to have problems using it in Visual Studio, before extracting it, right-click on the zip file and choose Properties, and then Unblock, then OK. Then extract it. If you do not do that, you may get security warning errors from Visual Studio.
Before running the server code the first time, you may want to change the folder where the logs are written. The default path is c:/LogForSaeaTest/, which will be created at server startup, if it does not exist. If you do not want to use the default folder, change the path in TestFileWriter class before running the app the first time. (TestFileWriter code is not included above, but is in the source code.) For the most part I have not set the server application up so that the SocketListenerSettings and other variables can be controlled from the Console. You'll need to change the source code and rebuild to make most changes during testing.
It's much better to run the client on one machine and server on another. If you try to run the client and server on the same machine, as the value of the "host" variable on the client, try to use the computer name first. If that does not work, try "localhost" as the value of the "host" variable on the client. When trying to connect the client to the server, if you get a "connection actively refused" message, check to see if you have a firewall that is blocking the transmissions on the incoming port of the server. You might have to allow incoming transmissions on that port on your local network. And if you have a firewall that blocks outgoing transmission on the client, then you would need to change settings for that too. When testing any network application, I suggest learn how to you use Wireshark or something like it.
The client app
A lot of the code in the client app is very similar to the server app. The server code is fully commented. So I did not always fully comment the client code. If in the client you find code that you do not understand and it is not commented on, then check similar portions of the server app for code comments.
The client app is NOT a normal client, but an app designed to test the server. It is set up to deliver as many connections as you want to throw at the server. The client app is set up to build all the messages for all of those connections before the test is started. It sends a different message each time for each client. And all the messages are put in memory before the test starts, so message creation won't be a drag on the client app during the test. If you choose to have 5000 connections sending 5000 messages per connection, then that is 25 million messages. So that is too many for memory probably. If you want to do a long test like that, then in the client app change runLongTest
to true
. In that case, instead of sending a separate message for each message from each client, it will send the same array of messages over and over for each client. That way the messages can fit in memory. (If you are doing a long test like that, also set runLongTest
to true
on the server. That will keep the server app from writing the received data to a dictionary. Otherwise, you'll run the server out of memory on the server probably.)
How many connections can this application handle? It depends on your hardware and configuration. In some testing I did, when running the server on an older single processor Dell desktop with Windows XP Pro 32 bit on a wired local area network with 100 MB NICs, it could handle a few thousand connections sending/receiving messages continually.
- The client app is set up so that you can do the following from the Console: (a) put in the method for finding the network address of the host machine, either machine name or IP address, (b) put in the correct string for the host machine name or IP address, depending on what method you chose for getting the network address of the host, (c) type in the port number of the host app, or accept the default, (d) specify a folder name for the log file to be written to, or accept the default, (e) specify the buffer size, or accept the previous value, (f) specify the number of client connections to attempt, or accept the previous value, (g) indicate the number of TCP messages to send per connection or accept the previous value.
- In the downloadable source code, there is plenty of capability to visualize what is happening by writing to a log and the Console. The things that you can visualize are (a) program flow, as it moves from method to method, (b) connects and disconnects, (c) the data which was sent from client to server and server to client, (d) threads, only in the server app. Save the thread watching for last.
Simple process to understand the code well
First, start the server app, and make sure you see "Server is listening" in the console. Then start the client app. You'll be asked to specify the host in the console. For the other items it will ask you, hopefully you can just accept the defaults. When it displays "Press Enter to begin socket test," press Enter. It should finish quickly. Then close both client and server, to make the logs finish writing. You just sent one message from one client connection to the server, and a response message from the server back to that client connection.
Now look at the log files from both server and client. (It's easier if you print them.) Compare them to the code, and think about it a long time. You are looking at one message sent from one connection, and the response to that message. You should be able to see and understand the program flow very well from it.
Now go thru that same for each of the tests below. Start the server, and then the client. The client will ask you about buffer size, number of connections and number of messages. You'll make some selections for those things on the client. When you change the number of connections you are changing the number of simulated users. One connection is like one client user running on one machine on the Internet. Run the test. And then after each test close client and server apps to write the logs. Look at the logs and make sure you understand what is happening after each test. Then go to the next test in the list.
test 1) buffer size stays same (25), number of connections = 2, number of messages = 2.
test 2) buffer size stays same (25), number of connections = 3, number of messages = 5.
test 3) buffer size = 5 on client, number of connections = 3, number of messages = 5. (In this test the buffer is smaller than the message with its prefix. So the client will require multiple send ops to send one message. And the client will require multiple receive ops to receive one message. You can see that in the client log. What happens in the server logs can vary, since one send op from client does NOT necessarily correlate with one receive op on the server, due to the way that TCP works.)
test 4) buffer = 3 on client, number of connections = 3, number of messages = 5. (In this test the buffer is smaller than even the prefix by itself. Multiple send ops will be required.)
test 5) put buffer back to 25 on client, number of connections = 3, number of messages = 5. Now on the server, in the Program.cs code changetestBufferSize
from 25 to 5. Build the server app. Run the test. You'll see that multiple receive ops are required to receive a message from the client. And multiple send ops are required to send from server back to client.
test 6) On server put testBufferSize
back to 25 on server, change watchThreads
from false to true, and build. Leave settings the same as they were on the client, and run the test again. Now after you close the server app the server log file will show info on threads. In Notepad, or whatever program you use to look at the logs, search for the phrase "New managed thread". You'll see that phrase every time a new thread starts. Usually there are only 2-4 managed threads running for the app. The managed thread numbers and socket handle identifiers are displayed throughout the log file now.
test 7) On server change watchThreads
to false, watchProgramFlow
to false, maxNumberOfConnections
to 10,000, and build. On client change watchProgramFlow
to false, and build. In client console at startup make number of connections = 1000, number of messages = 500. Run the test. (If this crashes your client app, change runLongTest
to true on the client, and build it again. If you change runLongTest
to true on the client, there will not be a huge array of arrays of messages created on the client before starting the test. Instead just one array of messages will be created and it will be sent over and over, the same array being used for all of the connections. When runLongTest
is false on the client, every connection has its own unique array of messages.)
test 8) To run bigger tests, I suggest you change runLongTest
to true on the server also. Try setting number of connections = 8000, number of messages = 5000 on the client, and see if it crashes your server app. On the client there is also a tickDelayBeforeNextConn
variable which is set to delay the next connection by 1 millisecond (10000 ticks). You can play around with that some. If you send too many new connection requests at once, you'll overwhelm the server. It's kind of fun to do once or twice.