I read that if you specify too large a heap (using -Xmx) then it will slow down the application, is

Yes. No. Maybe.
It could slow down the application (as opposed to definitely will slow down the application). Let me explain.

First off, you should know by now that the heap should be set to less than the available physical RAM on the machine, that is the RAM minus the amount of memory taken by any other processes that will run concurrently. Otherwise your JVM process will likely swap and that is bad. But let’s assume you know all about that. What about heap size apart from that?

There two types of garbage collection (GC) that execute in the system (at least for Sun JVMs): young generation GC and old generation GC. Young generation GC time is proportional to the number of live objects, so the size of the heap doesn’t matter for young generation GC. Old generation GC is proportional to the size of the heap. Most objects are collected in the young generation, some live long enough to flow over to the old generation. If the old generation gets full up, the JVM will execute an old generation GC. With a 3 Gig heap, the old generation GC would likely stop all processing for several seconds unless you are using the concurrent garbage collector (not on by default though will be in 5.0). That could make your overall GC take longer than is optimal - if the heap were smaller, the old generation GC would be shorter.

An ideal situation for Java applications is where the old generation does not fill up - in which case no old generation GC occurs. So, you probably want to try and tune your system so that no old generation GCs occur, in which case too large a heap size doesn’t matter; or if old generation GCs will occur in your application, you want to minimize the overhead these impose on your system in which case the size of the heap is important and too large a heap can slow down your application.