I'm running Windows 7 inside a virtual machine with 768 M of memory. I've set Parallels 5 to "Optimze for Mac OS X". This is because my host computer, an MBAir, only has 2 GB or RAM. If I optimize for the Virtual Machine a lot of pre-caching is going on and Parallels will eat a lot of my machine's memory (around 1.3 GB). This leaves very little space for using OS X. In Parallels 4, the setting "Optimize for Mac OS X" reduced Parallels' memory requirements significantly. In Parallels 5, there's absolutely no difference between the two settings (Optimize for Mac OS X vs Optimize for the Virtual Machine) - the memory requirements are horrendous regardless of the setting, and my Mac actually slows down as there is not enough real memory left to do any meaningful work. How much memory does your virtual machine require (look it up via OS X's Activity Monitor)? How much real RAM do you have?
I'm running PD4 / Snow Leopard (host) / XP SP3 (Guest). Just increased virtual memory to 512MB which is said by Parallels Configure screen to be optimal value - just as well given my ageing Intel iMac only has 2Gb memory.