Introduction
In trying to keep up to speed with .NET 2.0, I decided to do a .NET 2.0 version of my CodeProject article "DIME Buffered Upload" which used the DIME standard to transfer binary data over web services. The DIME approach was reasonably efficient but the code is quite complex and I was keen to explore what .NET 2.0 has to offer. In this article, I use version 3.0 of the WSE (Web Service Enhancements) which is available for .NET 2.0 as an add-in, to provide a simpler and faster method of sending binary data in small chunks over HTTP web services.
Background
Just a re-cap on why you may need to send data in small chunks at all: if you have a large file and you want to send it across a web service, you must understand the way it all fits together between IIS, .NET, and the web service call. You send your file as an array of bytes, as a parameter to a web service call, which is all sent to the IIS web server as a single request. This is bad if the size of the file is beyond the configured MaxRequestLength of your application, or if the request causes an IIS timeout. It is also bad from the point of view of providing feedback of the file transfer to the user interface because you have no indication of how the transfer is going, until it is either completed or failed. The solution outlined here is to send chunks of the file one by one, and append them to the file on the server.
There is an MD5 file hash done on the client and the server to verify that the file received is identical to the file sent. Also, both upload and download file transfers are included in this article.
Adventures with MTOM
MTOM stands for SOAP "Message Transmission Optimization Mechanism" and it is a W3C standard. To use it (and to run this application), you must download and install WSE 3.0, which includes MTOM support for the first time. If you look in the app.config and web.config files in the source code, you will see sections referring to the WSE 3 assembly, and a messaging clientMode or serverMode setting. These are necessary to run MTOM in the application.
The problem with DIME was that the binary content of the message was sent outside the SoapEnvelope of the XML message. This meant that although your message was secure, the Dime Attachment may not be secure. MTOM fully complies with the other WS-* specifications (like WS-Security) so the entire message is secure.
It took me a while to realise that when MTOM is turned on for the client and the server, WSE automatically handles the binary encoding of the data in the web service message. With DIME and WSE 2.0, you had to code your app for DIME by using DimeAttachments. This is no longer necessary, you just send your byte[] as a parameter or return value to a WebMethod, and WSE makes sure that it is sent as binary, not padded by XML serialization as it would be in the absence of DIME or MTOM.
The User Interface
The client application should be fairly straight forward. There are options at the top of the form to indicate if you want the file hash check done at the end of the transfer. You can also manually set the chunk size, or you can tick the box to AutoSet to let it regulate itself. In general i like using status bars more than message-boxes, as they are less intrusive, so keep an eye on the status bar to see what's going on.
Then there is the panel for uploading a file, simply click browse and click the button and you're away.
Similarly the panel for downloading files is easy to use. It has a listbox showing all the files in the 'Upload' folder on the server, there probably won't be any there by default. Just select a file and click Download. You can refresh the files list at any time with the Refresh button. To change the save folder, enter a new path in the text box provided.
Lastly, there is a textbox at the bottom which will contain the file hashes of the local and remote files after the transfer is complete.
Use the File menu for some additional options like manually resuming a transfer and manually invoking a file hash check.
A Forms Authenticated web service?
You can also tick the box for 'Login Required' if your web site is configured with forms authentication. Normally you can't use a web service if it is protected by forms authentication. This is because forms authentication is performed via a login aspx page and an auth cookie is given to the client browser, these conditions are not web-service friendly. There is a work-around to allow you to protect the web service via forms-authentication, it sends a HttpWebRequest to the login.aspx page and captures the cookie and places the cookie in the web service objects. Look in the code-behind of login.aspx.cs to see the minor modification that was needed to accept a login via a querystring (i.e. HttpWebRequest). To use forms auth, just change the authentication section of web.config (it is set to Windows auth by default). Then you will need to enter a username/password (admin/admin by default) in the client app to upload or download a file.
How the code works
The web service has two main methods, AppendChunk is for uploading a file to the server, DownloadChunk is for downloading from the server. These methods receive parameters for the file name, the offset of the chunk, and the size of the buffer being sent/received.
The Windows Forms client application can upload a file by sending all the chunks one after the other using AppendChunk, until the file has been completely sent. It does an MD5 hash on the local file, and compares it with the hash on the file on the server, to make sure the contents of the files are identical. The download code is very similar, the main difference is that the client must know from the server how big the file is, so that it can know when to stop requesting chunks.
A simplified version of the upload code is shown below (from the WinForms client). Have a look in the code for Form1.cs to see the inline comments + the explanation of the code. Essentially, a file stream is opened on the client for the duration of the transfer. Then the first chunk is read into the Buffer byte array. The while loop keeps running until the FileStream.Read() method returns 0, i.e. the end of the file has been reached. For each iteration, the buffer is sent directly to the web service as a byte[]. The 'SentBytes' variable is used to report progress to the form.
using(FileStream fs = new FileStream(LocalFilePath,
FileMode.Open, FileAccess.Read))
{
int BytesRead = fs.Read(Buffer, 0, ChunkSize);
while(BytesRead > 0 && !worker.CancellationPending)
{
ws.AppendChunk(FileName, Buffer, SentBytes);
SentBytes += BytesRead;
BytesRead = fs.Read(Buffer, 0, ChunkSize);
}
}
Setting the chunk size
In many windows forms applications, regular feedback to the user is very important. Having a responsive and visually communicative application is usually worth a small sacrifice in performance. Feedback for file transfers is typically done via a progress bar and/or status bar message. Obviously the web services aspect to a chunked file transfer is overhead, the client constructs and sends the soap message, and the server receives and parses it and sends the response. If the chunk size is very small, i.e. 2Kb, then there is a lot of messaging going on and not much data transfer. It should be clear then that we should aim for the highest possible chunk size that is within our requirements for quick user interface feedback. I have aimed for each chunk to be completed in 800 miliseconds. You can adjust this setting programatically before the file transfer, see the PreferredTransferDuration variable in the file transfer object. The client regulates the chunk size automatically, to ensure that each chunk is completed in the desired time. This is done by sampling every 15th chunk (also adjustable, ref ChunkSizeSampleInterval) and adjusting the chunk size based on the time it takes to transfer this chunk. The overall result is a self-controlled file transfer which will adapt to changing network conditions during the transfer.
One useful feature is that the web service provides the MaxRequestLength setting on the server, which the client retrieves before the transfer in order to stay within acceptable request sizes on the server. The client indicates "Max" in the status bar if the chunk size is at its maximum.
Resume Transfers Supported
The application also supports resuming a failed upload or download. I use this app to copy a system image (20 gigs+) from my web server to my home PC. Obviously there is a good chance of a connection being dropped during such a lengthy transfer, even with a 3Mbit bidirectional line. Resume support is a must when dealing with such large files, and fortunately it is very simple to include this in the application. Because data is only written to the file after it has been successfully received, we can be confident in resuming a file transfer based on the size of the partial file. In practice this works perfectly. Click File > Resume Upload or File > Resume Download etc to locate the partial file you wish to resume. After the file transfer, an MD5 hash is requested from the server and the client to compare the files. The timeout value is increased for the hash check, but it is always possible that it will timeout when checking such a large file, so you might have transferred the correct number of bytes but have no way to verify that they are the right bytes. I have included a manual MD5 hash check, available in the File menu. If the server is not giving the file hash within the timeout limit, you could run the client app directly on the server and check it locally, overcoming the timeout problem.
Incorporating into your own application
To use this code in your own client application, simply add a reference to MTOM_Library.dll, which is included in the article download. Then you should use the example client application as a starting point, it shows how to use the classes provided to perform the file transfer. Refer to the SetUpDownload and SetUpUpload methods of Form1.cs in particular. You use the FileTransferUpload class to do an upload, and the FileTransferDownload class to do a download. Each class has settings, such as ChunkSize, AutoSetChunkSize, LocalSaveFolder, IncludeHashVerification, etc. You can configure these as needed in your code. To change the location of the web service, make sure the app.config file is deployed to your run directory and change the MTOM_Library_MtomWebService_MTOM setting to the location of your web service.
Your asp.net application will also need to host the MTOM.asmx web service, and the Login.aspx code behind if you use forms authentication. To configure the web application for MTOM, just copy the settings from the web.config file included in this application. You can change the Upload folder on the server in web.config also, files are downloaded from and uploaded to this folder.
Points of Interest
The BackgroundWorker class in .NET 2.0
.NET 2.0 has a great new class called 'BackgroundWorker' to simplify running tasks asynchronously. Although this application sends the file in small chunks, even these small chunks would delay the WinForms application and make it look crashed during the transfer. So the web service calls still need to be done asynchronously. The BackgroundWorker class works using an event model, where you have code sections to run for DoWork (when you start), ProgressChanged (to update your progress bar / status bar), and Completed (or failed). You can pass parameters to the DoWork method, which you could not do with the Thread class in .NET 1.1 (I know you could with delegates, but delegates aren't great for thread control). You can also access the return value of DoWork in the Completed event handler. So for once, MS has thought of everything and made a very clean threading model. Exceptions are handled internally and you can access them in the Completed method via the RunWorkerCompletedEventArgs.Error property.
The code shown below is an example of the ProgressChanged event handler:
private void workerUpload_ProgressChanged(object sender,
ProgressChangedEventArgs e)
{
// update the progress bar and status bar text
this.toolStripProgressBar1.Value = e.ProgressPercentage;
this.statusText.Text = e.UserState.ToString();
// summary text is sent in the UserState parameter
}
I have used four BackgroundWorker objects in the application:
- one to manage the upload process,
- one to manage the download process,
- another to calculate the local MD5 file hash in parallel while waiting for the server result,
- and another to download the list of files in the Upload server folder to allow the user to select a file to download.
The reason I use a BackgroundWorker object for each task is because the code for each task is tied in to the events for that object, and i like this programming model.
A good example of Thread.Join()
When the upload or download is complete, the client asks for an MD5 hash of the file on the server, so it can compare it with the local file to make sure they are identical. I originally did these in sequence. But it can take a few seconds to calculate the result for a large file (anything over a few hundred MB), so the application was waiting five seconds for the server to calculate the hash, and then five more seconds for the client to calculate its own hash. This made no sense, so I decided to implement a multi-threaded approach to allow them to run in parallel. While the client is waiting on the server, it should be calculating its own file hash. This is done with the Thread class, and the use of the Join() method which blocks execution until the thread is complete.
The code below shows how this is accomplished:
// start calculating the local hash (stored in class variable)
this.hashThread = new Thread(new ThreadStart(this.CheckFileHash));
this.hashThread.Start();
// request the server hash
string ServerFileHash = ws.CheckFileHash(FileName);
// wait for the local hash to complete
this.hashThread.Join();
if(this.LocalFileHash == ServerFileHash)
e.Result = "Hashes match exactly";
else
e.Result = "Hashes do not match";
There is a good chance that the two operations will finish at approximately the same time, so very little waiting around will actually happen.
Common Problems / Questions
Visual Studio Compile Errors
There have been dozens of questions with people not being able to compile the solution in Visual Studio. You may get an error like this: The type or namespace name 'MTOMWse' does not exist in the namespace 'UploadWinClient.MtomWebService'. With an MTOM enabled web service, Visual Studio is supposed to generate 2 proxy classes, a standard one derived from System.Web.Services.Protocols.SoapHttpClientProtocol and a WSE class derived from Microsoft.Web.Services3.WebServicesClientProtocol, with "Wse" tagged on to the end of the proxy class name. Sometimes VS misbehaves and does not generate this class, i don't understand why but the work around is to Show-all-files in the win-forms project, expand the web service > Reference.map > Reference.cs. Edit this file and change: public partial class MTOM : System.Web.Services.Protocols.SoapHttpClientProtocol to public partial class MTOMWse : Microsoft.Web.Services3.WebServicesClientProtocol
Also make sure to update the constructor to match the new class name. Then it should compile fine.
Can you make a web client? No, No, and Double No!
I have got a ton of questions about people asking for a web client instead of a winforms client. This is fundamentally not possible because of the advanced file Input/Output required to achieve this solution. For good reason, browsers do not provide this level of access to the file system of the client. A guy called Brettle wrote a progress bar control for asp.net file uploading, this may be your best bet, although you must understand that web applications are very limited when it comes to sending large amounts of data to the web server.
Conclusions
I found that MTOM was about 10% faster than DIME in my limited testing. This is probably to do with the need to package up each chunk into a DIME attachment, which is no longer necessary with MTOM. Remember, if you want to send chunks larger than 4 MB, you must increase the .NET 2.0 Max Request Size limit in your web.config.
904
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
