I installed XFCE in the order as listed on http://www.xfce.org/documentation/requirements , but it would not start, claiming that it couldn't find libxfm-4.6.so.0. To be safe, I recompiled and installed xfce4-session, but it still complained. I ran ldconfig, which seemed to help, or it could have been coincidence. XFCE would start, but there were more serious problems (see compositing bug I submitted) that may be unrelated.
This was on Ubuntu 8.04, by the way.
So... you fixed your problem by fixing your broken library setup on your system...