How to user SSE2 instructions to improve the performance of memory copy?

 

How to user SSE2 instructions to improve the performance of memory copy?

 

It is an ageless topic for programmers to discuss how to improve the performance to transfer data. For the common scenario, the memcpy provided by C/C++ lib is enough to handle the boring duplicating tasks. However, if the size of data is larger than the size of last data cache in your system, for example, the last data cache is L2 data cache in my system whose size equals 1Mbytes, what will happen when copying 10Mbytes data? Obviously, the speed will become slow because the cache is polluted after invoking memcpy.

As a result, we often ask whether there is a good solution that can decrease the polluting degree in the data cache, and improve its performance at the same time. Fortunately, it exists in the SSEx instructions’ set. This article only focus on the SSE2’s instructions, the advanced SSEx will be discussed in the following articles.

Before showing the source codes, we had better understand several basic instructions in the SSE, and they are:

a. PREFETCHNTA.

Non-temporal data—fetch data into location close to the processor, minimizing cache pollution

• Pentium III processor—1st-level cache

• Pentium 4 and Intel Xeon processor—2nd-level cache

b. MOVNTDQ.

The MOVNTDQ (store double quadword using non-temporal hint) instruction stores packed integer data from an XMM register to memory, using a non-temporal hint.

c. MOVDQA.

The MOVDQA (move aligned double quadword) instruction transfers a double quadword operand from memory to an XMM register or vice versa; or between XMM registers. The memory address must be aligned to a 16-byte boundary; otherwise, a general-protection exception (#GP) is generated.

d. MOVDQU.

The MOVDQU (move unaligned double quadword) instruction performs the same operations as the MOVDQA instruction, except that 16-byte alignment of a memory address is not required. Its efficiency is lower than MOVDQA.

e. SFENCE.

The SFENCE (Store Fence) instruction controls write ordering by creating a fence for memory store operations. This instruction guarantees that the result of every store instruction that precedes the store fence in program order is globally visible before any store instruction that follows the fence. The SFENCE instruction provides an efficient way of ensuring ordering between procedures that produce weakly-ordered data and procedures that consume that data.

 

The maximal advantage is that SSE instruction enhances the throughput to transfer data. One cycle of instruction can transfer 16 bytes’ data, but the original bandwidth of MOV only holds 4 bytes. Moreover, by cooperating with PREFETCHNTA, the hit rating of data cache takes few effects after SSE’s memcpy if comparing with the traditional way.

Ok, we have spent too much time in describing the un-core contents, and let’s enter the main branch.

 

[The block of source code]

 

void* sse2_fast_memcpy( void *pDst, const void *pSrc, size_t len ) {     void *pBegin = pDst;     int  offset;       if ( (offset = ((unsigned long) pDst) & SSE_ALIGNMENT_MASK) > 0 )         offset = SSE_ALIGNMENT_VAL - offset;       if ( len < (size_t) offset + 16 )     {         return memcpy( pDst, pSrc, len );     }       if ( offset > 0 )     {         memcpy( pDst, pSrc, offset );         len -= offset;         pDst = ((char *) pDst) + offset;         pSrc = ((char *) pSrc) + offset;     }       if ( SSE_CAN_ALIGN( pDst, pSrc ) )     {         _asm         {             mov ecx, len             mov esi, pSrc             mov edi, pDst               cmp ecx, 128             jb LA2             prefetchnta [esi] LA1:             prefetchnta XMMWORD PTR[esi + 16 * 4]             movdqa xmm0, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm0             movdqa xmm1, XMMWORD PTR[esi + 16 * 1]             movntdq XMMWORD PTR[edi + 16 * 1], xmm1             movdqa xmm2, XMMWORD PTR[esi + 16 * 2]             movntdq XMMWORD PTR[edi + 16 * 2], xmm2             movdqa xmm3, XMMWORD PTR[esi + 16 * 3]             movntdq XMMWORD PTR[edi + 16 * 3], xmm3               prefetchnta XMMWORD PTR[esi + 16 * 8]             movdqa xmm4, XMMWORD PTR[esi + 16 * 4]             movntdq XMMWORD PTR[edi + 16 * 4], xmm4             movdqa xmm5, XMMWORD PTR[esi + 16 * 5]             movntdq XMMWORD PTR[edi + 16 * 5], xmm5             movdqa xmm6, XMMWORD PTR[esi + 16 * 6]             movntdq XMMWORD PTR[edi + 16 * 6], xmm6             movdqa xmm7, XMMWORD PTR[esi + 16 * 7]             movntdq XMMWORD PTR[edi + 16 * 7], xmm7               add esi, 128             add edi, 128             sub ecx, 128             cmp ecx, 128             jae LA1 LA2:             cmp ecx, 64             jb  LA3             prefetchnta XMMWORD PTR[esi]             sub ecx, 64             movdqa xmm0, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm0             movdqa xmm1, XMMWORD PTR[esi + 16 * 1]             movntdq XMMWORD PTR[edi + 16 * 1], xmm1             movdqa xmm2, XMMWORD PTR[esi + 16 * 2]             movntdq XMMWORD PTR[edi + 16 * 2], xmm2             movdqa xmm3, XMMWORD PTR[esi + 16 * 3]             movntdq XMMWORD PTR[edi + 16 * 3], xmm3               add esi, 64             add edi, 64 LA3:             prefetchnta XMMWORD PTR[esi]             cmp ecx, 32             jb  LA4             sub ecx, 32             movdqa xmm4, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm4             movdqa xmm5, XMMWORD PTR[esi + 16 * 1]             movntdq XMMWORD PTR[edi + 16 * 1], xmm5               add esi, 32             add edi, 32 LA4:             cmp ecx, 16             jb  LA5             sub ecx, 16             movdqa xmm6, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm6               //add esi, 16             //add edi, 16 LA5:             sfence         }     }     else // Unalignment     {         _asm         {             mov ecx, len             mov esi, pSrc             mov edi, pDst               cmp ecx, 128             jb LB2             prefetchnta [esi] LB1:             prefetchnta XMMWORD PTR[esi + 16 * 4]             movdqu xmm0, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm0             movdqu xmm1, XMMWORD PTR[esi + 16 * 1]             movntdq XMMWORD PTR[edi + 16 * 1], xmm1             movdqu xmm2, XMMWORD PTR[esi + 16 * 2]             movntdq XMMWORD PTR[edi + 16 * 2], xmm2             movdqu xmm3, XMMWORD PTR[esi + 16 * 3]             movntdq XMMWORD PTR[edi + 16 * 3], xmm3               prefetchnta XMMWORD PTR[esi + 16 * 8]             movdqu xmm4, XMMWORD PTR[esi + 16 * 4]             movntdq XMMWORD PTR[edi + 16 * 4], xmm4             movdqu xmm5, XMMWORD PTR[esi + 16 * 5]             movntdq XMMWORD PTR[edi + 16 * 5], xmm5             movdqu xmm6, XMMWORD PTR[esi + 16 * 6]             movntdq XMMWORD PTR[edi + 16 * 6], xmm6             movdqu xmm7, XMMWORD PTR[esi + 16 * 7]             movntdq XMMWORD PTR[edi + 16 * 7], xmm7               add esi, 128             add edi, 128             sub ecx, 128             cmp ecx, 128             jae LB1 LB2:             cmp ecx, 64             jb  LB3             prefetchnta XMMWORD PTR[esi]             sub ecx, 64             movdqu xmm0, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm0             movdqu xmm1, XMMWORD PTR[esi + 16 * 1]             movntdq XMMWORD PTR[edi + 16 * 1], xmm1             movdqu xmm2, XMMWORD PTR[esi + 16 * 2]             movntdq XMMWORD PTR[edi + 16 * 2], xmm2             movdqu xmm3, XMMWORD PTR[esi + 16 * 3]             movntdq XMMWORD PTR[edi + 16 * 3], xmm3               add esi, 64             add edi, 64 LB3:             prefetchnta XMMWORD PTR[esi]             cmp ecx, 32             jb  LB4             sub ecx, 32             movdqu xmm4, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm4             movdqu xmm5, XMMWORD PTR[esi + 16 * 1]             movntdq XMMWORD PTR[edi + 16 * 1], xmm5               add esi, 32             add edi, 32 LB4:             cmp ecx, 16             jb  LB5             sub ecx, 16             movdqu xmm6, XMMWORD PTR[esi]             movntdq XMMWORD PTR[edi], xmm6               //add esi, 16             //add edi, 16 LB5:             sfence         }     } // End if ( SSE_CAN_ALIGN( pDst, pSrc ) )       offset = len & 0x0F;     if ( offset > 0 )     {         memcpy( ((char *) pDst) + (len - offset), ((char *) pSrc) + (len - offset), offset );     }       return pBegin; }

 

[Comments of source codes]

 

1) The memcpy of system is adopted when the length of data is small in the source codes. If you prefer all by yourself, small_fast_mempcy listed in the next session maybe is an ideal selection to handle the small block of data.

void* small_fast_memcpy( void *pDst, const void *pSrc, size_t len ) {     _asm     {         mov ecx, len         mov edi, pDst         mov esi, pSrc         rep movsb     }       return pDst; }

 

2) To ease to catch the emphasis of source code, the simply logical judegement is implemented by C syntax. In fact, it is very easy to convert them into ASM, too. If readers do not prefer hybrid way, why no to do it now by yourselfJ!

 

3) MACROs are utilized in the source code to eliminate the modification if instructions are 64-byte alignment(The partial instructions of SSE4x need 64-bytes alignment). Please refer the below block.

#define SSE_ALIGNMENT_VAL               (16) #define SSE_ALIGNMENT_MASK              (SSE_ALIGNMENT_VAL - 1) #define SSE_CAN_ALIGN( addr1, addr2 )   /     ((((unsigned long) (addr1)) & SSE_ALIGNMENT_MASK) == (((unsigned long) (addr2)) & SSE_ALIGNMENT_MASK))

 

4) The interlace of XMMx register is to decrease the reading and writing dependence, and improve the parallel ability.

 

5) The interlace of reading and writing operations is to add the delaying time to cover the current prefetching latency.

 

6) The prefetching size is 64bytes and it is my length of data cache line in my PC. It should be adjusted in the real environment, but it is appropriate size in normal case.

 

[Summarization]

1. It only can acquire the improvement of speed when the data’s size is larger than the size of last data cache. Otherwise, the performance becomes worse. By testing it for big data, there is +50% improvement.

2. Be careful to use prefetch instruction. Its abusing operation can heavily disturb the performance of system.

3. It is not the best solution by using SSEx instruction. If the higher SSEx’s instructions are used, +1% gain is available if compared with SSE2.

 

Anyway, I hope that readers can give me any feedback to improve/correct the usage of SSEx. I think that it will help me understand inside deeply.