Not sure about where to find and revert all left-over changes applied by the agent, but since you mention `xorg.conf`, you might as well start there. I suggest you check out the `-configure` switch to `Xorg`.
Code:
-configure
When this option is specified, the Xorg server loads all video
driver modules, probes for available hardware, and writes out
an initial xorg.conf(5) file based on what was detected. This
option currently has some problems on some platforms, but in
most cases it is a good way to bootstrap the configuration pro-
cess. This option is only available when the server is run as
root (i.e, with real-uid 0).
You may also want to look at the CENTOS spec file and patches for the X server to see if they do anything fancier out-of-the-box.
Good luck.
PS: Of course, you can also (a) reinstall the guest or (b) give in and reinstall the agent using the CURSES interface without X, but these are things you're most likely trying to avoid.