Why VHD not working on my Kingston UFD

Why VHD not working on my Kingston UFD

Robbie (Decheng) Fan
2011-06-10

Yesterday I put my VHD file on my new Kingston 8G USB Flash Drive (aka UFD), and hope that Virtual PC can work with it if I turn on the -usehostdiskcache switch. I have tested the speed of the UFD the day before, and have confidence in its performance.

However, to my dismay, the VHD write performance was extremely poor. Not to mention massive file copy operation, even the normal operation is affected by the slow disk performance. I see the light of the UFD virtually flashes all the time. I wondered why. I'm sure of its write performance problem - as I paused the VPC, the flashing went on for quite a few seconds. And it's a proof that disk cache is on.

I didn't believe it. I checked write cache of the UFD device in Device Manager, and made sure it is turned on. I stopped the UFD, unplugged and plugged it in again, hoping the system to refresh its cache status. Anyway still nothing changed. I removed the -usehostdiskcache switch from VPC, and started the VM again. This time the write performance is even lower than when disk cache is enabled.

OK, I admit I lost. I tried copying a directory of many small files directly to the UFD again, and it's still very fast. I tried to copy the same bunch of files from inside the VM, it's extremely slow again. There is no quality problem found with the UFD, as I checked again for bad sectors.

My host operating system is Windows Vista. Today I finally got the Disk Performance counters fixed. I looked carefully at the Physical disk performance counter. The result revealed that the disk queue length (Current Disk Queue Length) is often 2 or more when I copy files on the VHD on it. The disk queue length is often 1 when I copy files directly to it. And the disk writes per second is dozens of times higher when I copy files directly to it. See screenshots "Why VHD not working on my Kingston UFD_files/Perfmon_Direct.png" and "Perfmon_VHD.png".

 

Figure 1: Performance Monitor monitoring direct file copy to the UFD (Perfmon_Direct.png)

 

Figure 2: Performance Monitor monitoring VHD operation when copying a folder inside VPC (Perfmon_VHD.png)

 

I don't know the root cause of it yet. It's quite strange: less writes per second causes higher disk queue length. Anyway it's the symptom. I wrote test programs, see "Why VHD not working on my Kingston UFD_files/file1.cs" and "file2.cs". I tried to use those programs to simulate the situation of disk access on VPC, but none gave the similar behavior. All, including the WriteThrough option, made disk writes very fast.

I finally guess it's because VPC writes to the disk directly (or some special mechanism), rather than using Windows file I/O. I tried further with process monitor. VPC often writes 4KB blocks to the VHD, which is reasonable as the guest operating system is Windows NT, and Windows NT uses 4KB page size and cluster size for NTFS.

I found a problem through the Process Monitor trace of virtual pc and other related processes. See Procmon_VirtualPC.png. It shows that process SYSTEM (process id 4) does the cache flush. However, because the positions to write are scattered in various places, the write operation is issued many times. However, when compared with xcopy/robocopy, there are also many write operations in small chunks. I don't understand now... The VPC when copying a lot of files the disk queue will sometimes be very very high...

And the small chunks of write isn't the root cause, either. As I tried with test3.exe, it worked fine, though it issued many 32K writes at 64K addresses, the write speed can still be at around 1MB/s. Verified in Process Monitor.

 

Figure 3: Process Monitor monitoring VPC operation on the VHD on UFD (Procmon_VirtualPC.png)

 

Figure 4: Process Monitor showing program test2.cs operation to a large file on UFD (Procmon_Test2.jpg)

 

Now my task list came to the following.

- [DONE] Performance Monitor disk counters
- [DONE] Python/C# test write prog (cannot mimick VPC successfully)
- [DONE] Try vhd on UFD and net share and then VPC (no use, same problem)
- Undo disk strategy: VHD on UFD, undo disk on HDD

Before trying the final approach, I tried this configuration: put the Undo disk on the UFD. And interestingly, it seems to be useable. The write bytes per second can be an optimal 1MB/s. See image "Perfmon_UndoUFD.png". However, the performance is not ideal. Thus, I finally persuaded myself to use an Undo disk on the hard disk drive.

Now, a problem is left. Since we're using the Undo disk, what's the performance of commiting the Undo disk onto the UFD? Some headache came up: from the image "Perfmon_UndoCommit.png", it is apparent that the disk queue is too high! And the average value of write bytes per second is also quite low, at about 500KB/s. Fortunately, the queue length finally dropped.

 

Figure 5: Undo disk operation on the UFD from VPC (Perfmon_UndoUFD.png)

 

Figure 6: Undo disk on the UFD being committed to the VHD on the UFD (Perfmon_UndoCommit.png)

 

Below is proven false: I guessed why the write queue length is very high: because there are a lot of write requests followed by a lot of read requests. However, it's proven false by test4.cs. Thus, it's quite strange why Undo disk commit is so slow.

Anyway, the case draws to a close. Let me use an Undo disk on the HDD, and a VHD on the UFD. It'll be interesting.

 

Source code: test1.cs

 

using System; using System.IO; namespace RobbieTests { public class DiskTest { public static void Main() { const int Size = 1024 * 64; FileStream f1; long mid; int i; byte[] empty; f1 = new FileStream("H://Test.tmp", FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read); f1.SetLength(1048576 * 600); mid = f1.Length / 2; Console.WriteLine("Mid: {0}", mid); empty = new byte[Size]; for (i = 0; i < 2000; i++) { f1.Seek(mid - i * Size, SeekOrigin.Begin); f1.Write(empty, 0, Size); if ((i + 1) % 50 == 0) { Console.WriteLine("Position: {0}", f1.Position); Console.WriteLine("Write {0} completed.", i); if (Console.KeyAvailable) { break; } } } Console.WriteLine("Write completed."); f1.Close(); } } }

 

Source code: test2.cs

using System; using System.IO; namespace RobbieTests { public class DiskTest { public static void Main() { const int Size = 1024 * 64; FileStream f1; long mid; int i; byte[] empty; /** * Try: * FileOptions.RandomAccess * FileOptions.WriteThrough * FileOptions.SequentialScan * Result: * Nothing like the VPC behavior */ f1 = new FileStream("H://Test.tmp", FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read, 1024, FileOptions.SequentialScan); f1.SetLength(1048576 * 600); mid = f1.Length / 2; Console.WriteLine("Mid: {0}", mid); empty = new byte[Size]; for (i = 0; i < 2000; i++) { f1.Seek(mid - i * Size, SeekOrigin.Begin); f1.Write(empty, 0, Size); if ((i + 1) % 50 == 0) { Console.WriteLine("Position: {0}", f1.Position); Console.WriteLine("Write {0} completed.", i); if (Console.KeyAvailable) { break; } } } Console.WriteLine("Write completed."); f1.Close(); } } }

 

Source code: test3.cs

 

using System; using System.IO; namespace RobbieTests { public class DiskTest { public static void Main() { const int Size = 1024 * 64; FileStream f1; long mid; int i; byte[] empty; /** * Special: * Compared to test2.cs, this version uses "skip write" - the * blocks written to are not contiguous. * Try: * FileOptions.RandomAccess * FileOptions.WriteThrough * FileOptions.SequentialScan * Result: */ f1 = new FileStream("H://Test.tmp", FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read, 1024, FileOptions.RandomAccess); f1.SetLength(1048576 * 600); mid = f1.Length / 2; Console.WriteLine("Mid: {0}", mid); empty = new byte[Size]; for (i = 0; i < 2000; i++) { f1.Seek(mid - i * Size, SeekOrigin.Begin); f1.Write(empty, 0, Size / 2); if ((i + 1) % 50 == 0) { Console.WriteLine("Position: {0}", f1.Position); Console.WriteLine("Write {0} completed.", i); if (Console.KeyAvailable) { break; } } } Console.WriteLine("Write completed."); f1.Close(); } } }

 

Source: test4.cs

 

using System; using System.IO; namespace RobbieTests { public class DiskTest { public static void Main() { const int Size = 1024 * 64; FileStream f1; long mid; int i; byte[] empty; byte[] readbuf; /** * Special: * Compared to test2.cs, this version reads a lot of data * after each write. * Try: * FileOptions.RandomAccess * FileOptions.WriteThrough * FileOptions.SequentialScan * Result: */ f1 = new FileStream("H://Test.tmp", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.Read, 1024, FileOptions.RandomAccess); f1.SetLength(1048576 * 600); mid = f1.Length / 2; Console.WriteLine("Mid: {0}", mid); empty = new byte[Size]; readbuf = new byte[1024 * 128]; for (i = 0; i < 2000; i++) { f1.Seek(mid - i * Size, SeekOrigin.Begin); f1.Write(empty, 0, Size); f1.Seek((i % 1000) * readbuf.Length, SeekOrigin.Begin); f1.Read(readbuf, 0, readbuf.Length); if ((i + 1) % 50 == 0) { Console.WriteLine("Position: {0}", f1.Position); Console.WriteLine("Write {0} completed.", i); if (Console.KeyAvailable) { break; } } } Console.WriteLine("Write completed."); f1.Close(); } } }