Optimization is often counter-intuitive

Optimization is often counter-intuitive

Anybody who's done intensive optimization knows that optimization is often counter-intuitive. Things you think would be faster often aren't.

Consider, for example, the exercise of obtaining the current instruction pointer. There's the naïve solution:

__declspec(noinline)
void *GetCurrentAddress()
{
  return _ReturnAddress();
}

...
void *currentInstruction = GetCurrentAddress();

If you look at the disassembly, you'll get something like this:

GetCurrentAddress:
    mov eax, [esp]
    ret

...
    call GetCurrentAddress
    mov [currentInstruction], eax

"Pah," you say to yourself, "look at how inefficient that is. I can reduce that to two instructions. Watch:

void *currentInstruction;
__asm {
call L1
L1: pop currentInstruction
}

That's half the instruction count of your bloated girly-code."

But if you sit down and race the two code sequences, you'll find that the function-call version is faster by a factor of two! How can that be?

The reason is the "hidden variables" inside the processor. All modern processors contain much more state than you can see from the instruction sequence. There are TLBs, L1 and L2 caches, all sorts of stuff that you can't see. The hidden variable that is important here is the return address predictor.

The more recent Pentium (and I believe also Athlon) processors maintain an internal stack that is updated by each CALL and RET instruction. When a CALL is executed, the return address is pushed both onto the real stack (the one that the ESP register points to) as well as to the internal return address predictor stack; a RET instruction pops the top address of the return address predictor stack as well as the real stack.

The return address predictor stack is used when the processor decodes a RET instruction. It looks at the top of the return address predictor stack and says, "I bet that RET instruction is going to return to that address." It then speculatively executes the instructions at that address. Since programs rarely fiddle with return addresses on the stack, these predictions tend to be highly accurate.

That's why the "optimization" turns out to be slower. Let's say that at the point of the CALL L1 instruction, the return address predictor stack looks like this:

Return address predictor stack:		caller1	->	caller2	->	caller3	->	...
Actual stack:		caller1	->	caller2	->	caller3	->	...

Here, caller1 is the function's caller, caller1 is the function's caller's caller, and so on. So far, the return address predictor stack is right on target. (I've drawn the actual stack below the return address predictor stack so you can see that they match.)

Now you execute the CALL instruction. The return address predictor stack and the actual stack now look like this:

Return address predictor stack:		L1	->	caller1	->	caller2	->	caller3	->	...
Actual stack:		L1	->	caller1	->	caller2	->	caller3	->	...

But instead of executing a RET instruction, you pop off the return address. This removes it from the actual stack, but doesn't remove it from the return address predictor stack.

Return address predictor stack:		L1	->	caller1	->	caller2	->	caller3	->	...
Actual stack:		caller1	->	caller2	->	caller3	->	caller4	->	...

I think you can see where this is going.

Eventually your function returns. The processor decodes your RET instruction and looks at the return address predictor stack and says, "My predictor stack says that this RET is going to return to L1. I will begin speculatively executing there."

But oh no, the value on the top of the real stack isn't L1 at all. It's caller1. The processor's return address predictor predicted incorrectly, and it ended up wasting its time studying the wrong code!

The effects of this bad guess don't end there. After the RET instruction, the return address predictor stack looks like this:

Return address predictor stack:		caller1	->	caller2	->	caller3	->	...
Actual stack:		caller2	->	caller3	->	caller4	->	...

Eventually your caller returns. Again, the processor consults its return address predictor stack and speculatively executes at caller1. But that's not where you're returning to. You're really returning to caller2.

And so on. By mismatching the CALL and RET instructions, you managed to cause every single return address prediction on the stack to be wrong. Notice in the diagram that, in the absence of somebody playing games with the return address predictor stack of the type that created the problem initially, not a single prediction on the return address predictor stack will be correct. None of the predicted return addresses match up with actual return addresses.

Your peephole optimization has proven to be shortsighted.

Some processors expose this predictor more explictly. The Alpha AXP, for example, has several types of control flow instructions, all of which have the same logical effect, but which hint to the processor how it should maintain its internal predictor stack. For example, the BR instruction says, "Jump to this address, but do not push the old address onto the predictor stack." On the other hand, the JSR instruction says, "Jump to this address, and push the old address onto the predictor stack." There is also a RET instruction that says, "Jump to this address, and pop an address from the predictor stack." (There's also a fourth type that isn't used much.)

Moral of the story: Just because something looks better doesn't mean that it necessarily is better.

Published Thursday, December 16, 2004 6:58 AM by oldnewthing

http://blogs.msdn.com/oldnewthing/archive/2004/12/16/317157.aspx