The clue is the Cache size for reading from SD Card. It's set to 128 KB, on some ROM's it's even set to 4 KB!
You can check for yourself your Cache size by looking into this file:
You can change it manually but it will be reverted to default on the reboot. The changes need to be done permanently by loading the script on the startup through the init.d.
Or you can flash the CWM zip files that I made for you.
For RA 1.7 Recovery zip look into this post. Many thanx to user laie1472 for making them.
You must find which Cache size fits best for your SD Card. In my case its the 3072 KB Cache size. At best, use the Root Explorer to change the value, then run SD Tools Benchmark - finally flash the CWM zip file that fits your SD Card.
Here are two screenshots - the first one shows the reading speeds with the default Cache 128 KB and the second one is with the 3072 KB Cache with my Class 4 SD Card on CM7 RC2 by Charnsingh.
The Benchmark is made with the SD Tools program.
If you want to convince yourself of the speed improvement after applying the fix, just open your Gallery and see how fast it loads the photos!
Suggestion: 2048 KB Cache size performs at best for most of the SD Cards!