Which guest resolution you have in your Openstep VM? We implemented dynamic screen resolution feature in Beta5 and some videomodes numbers were changed...
My openstep image hangs up as well, and it does so at boot time while trying to load some config file (probably it hangs after trying to apply things it read I think). Nothing to do with videomode apparently, as even booting in Default (eg 16 colors vga mode) hangs, as well as single user boot...