2008.08.06_RolfRolles_VMProtect

http://www.openrce.org/blog/view/1238/VMProtect,_Part_0:__Basics

VMProtect is a virtualization protector.  Like other protections in the genre, among others ReWolf's x86 Virtualizer and CodeVirtualizer, it works by disassembling the x86 bytecode of the target executable and compiling it into a proprietary, polymorphic bytecode which is executed in a custom interpreter at run-time.  This is unlike the traditional notions of packing, in which the x86 bytecode is simply encrypted and/or compressed:  with virtualization, the original x86 bytecode in the protected areas is gone, never to be seen again.  Or so the idea goes.

If you've never looked at VMProtect before, I encourage you to take a five-minute look in IDA ( here's a sample packed binary).  As far as VMs go, it is particularly skeletal and easily comprehended.  The difficulty lies in recreating working x86 bytecode from the VM bytecode.  Here's a two-minute analysis of its dispatcher.

  push  edi                        ; push all registers
  push  ecx
  push  edx
  push  esi
  push  ebp
  push  ebx
  push  eax
  push  edx
  pushf
  push  0                          ; imagebase fixup
  mov   esi, [esp+8+arg_0]         ; esi = pointer to VM bytecode
  mov   ebp, esp                   ; ebp = VM's "stack" pointer
  sub   esp, 0C0h
  mov   edi, esp                   ; edi = "scratch" data area

VM__FOLLOW__Update:
  add   esi, [ebp+0]

VM__FOLLOW__Regular:
  mov   al, [esi]                  ; read a byte from EIP
  movzx eax, al
  sub   esi, -1                    ; increment EIP
  jmp   ds:VM__HandlerTable[eax*4] ; execute instruction handler

A feature worth discussing is the "scratch space", referenced by the register edi throughout the dispatch loop.  This is a 16-dword-sized area on the stack where VMProtect saves the registers upon entering the VM, modifies them throughout the course of a basic block, and from whence it restores the registers upon exit.  For each basic block protected by the VM, the layout of the registers in the scratch space can potentially be different.

Here's a disassembly of some instruction handlers.  Notice that A) VMProtect is a stack machine and that B) each handler -- though consisting of scant few instructions -- performs several tasks, e.g. popping several values, performing multiple operations, pushing one or more values.

#00:  x = [EIP-1] & 0x3C; y = popd; [edi+x] = y

.text:00427251   and   al, 3Ch        ; al = instruction number
.text:00427254   mov   edx, [ebp+0]   ; grab a dword off the stack
.text:00427257   add   ebp, 4         ; pop the stack
.text:0042725A   mov   [edi+eax], edx ; store the dword in the scratch space

#01:  x = [EIP-1] & 0x3C; y = [edi+x]; pushd y

.vmp0:0046B0EB   and   al, 3Ch        ; al = instruction number
.vmp0:0046B0EE   mov   edx, [edi+eax] ; grab a dword out of the scratch space
.vmp0:0046B0F1   sub   ebp, 4         ; subtract 4 from the stack pointer
.vmp0:0046B0F4   mov   [ebp+0], edx   ; push the dword onto the stack

#02:  x = popw, y = popw, z = x + y, pushw z, pushf

.text:004271FB   mov   ax, [ebp+0]       ; pop a word off the stack
.text:004271FF   sub   ebp, 2
.text:00427202   add   [ebp+4], ax       ; add it to another word on the stack
.text:00427206   pushf                
.text:00427207   pop   dword ptr [ebp+0] ; push the flags

#03:  x = [EIP++]; w = popw; [edi+x] = Byte(w)

.vmp0:0046B02A   movzx eax, byte ptr [esi] ; read a byte from EIP
.vmp0:0046B02D   mov   dx, [ebp+0]         ; pop a word off the stack
.vmp0:0046B031   inc   esi                 ; EIP++
.vmp0:0046B032   add   ebp, 2              ; adjust stack pointer
.vmp0:0046B035   mov   [edi+eax], dl       ; write a byte into the scratch area

#04:  x = popd, y = popw, z = x << y, pushd z, pushf

.vmp0:0046B095   mov   eax, [ebp+0]      ; pop a dword off the stack
.vmp0:0046B098   mov   cl, [ebp+4]       ; pop a word off the stack
.vmp0:0046B09B   sub   ebp, 2
.vmp0:0046B09E   shr   eax, cl           ; shr the dword by the word
.vmp0:0046B0A0   mov   [ebp+4], eax      ; push the result
.vmp0:0046B0A3   pushf
.vmp0:0046B0A4   pop   dword ptr [ebp+0] ; push the flags

#05:  x = popd, pushd ss:[x]

.vmp0:0046B5F7   mov   eax, [ebp+0]  ; pop a dword off the stack
.vmp0:0046B5FA   mov   eax, ss:[eax] ; read a dword from ss


.vmp0:0046B5FD   mov   [ebp+0], eax  ; push that dword

完整版：VMProtect By RolfRollesF.pdf

http://pan.baidu.com/netdisk/singlepublic?fid=140670_1443998598