When you try to configure xfdesktop and xfwm with xorg-x11, configure fails when it is checking for X libraries. Though 4.0.6 compiled fine.
no problem here (gentoo, xorg). what distro are you on, and where does it install X?
Works fine for me, can you try to describe the problem?
I'm using X.org on one of my systems, no problem either.
This seems to be your problem. What distro are you using? Could it be that you have to install the devel package for xorg-x11?
Works for most people.