You know it right
We have to load memory from the disk. I don't see any direct fix here.
But if you have a lot of operating memory and didn't reboot your machine since the time of suspend, there is a great chance that memory stays in the cache - wich means that restore will be way much faster...