Like I implied it's related to how much you have overall, if you have 4 GB RAM, and increase the VM to 2 GB it will likely make the whole computer run slow, as 2 GB for OS X will likely make it page, but if you have 8 GB you'll see an improvement. Depends also on what other things you are running on both sides.
Because you didn't actually specified any of these things my only possible answer is to talk about the generalities of it and suggest you try it yourself. Not trying to be rude, just pragmatic.
Last edited: Jun 19, 2013