xfce4-settings uses some symbols from libm and libX11 without explicitly linking to them. 

It should do so. 

/usr/bin/ld: xfce4_mouse_settings-main.o: undefined reference to symbol 'rint@@GLIBC_2.2.5'
/usr/bin/ld: note: 'rint@@GLIBC_2.2.5' is defined in DSO /lib64/libm.so.6 so try adding it to the linker command line

/usr/bin/ld: note: 'XSetSelectionOwner' is defined in DSO /usr/lib64/libX11.so.6 so try adding it to the linker command line

See: https://fedoraproject.org/wiki/Features/ChangeInImplicitDSOLinking for more info. 

Hacky patch (you will want to add it properly in automake files): 

diff -Nur xfce4-settings-4.6.4.orig/dialogs/mouse-settings/Makefile.in xfce4-settings-4.6.4/dialogs/mouse-settings/Makefile.in
--- xfce4-settings-4.6.4.orig/dialogs/mouse-settings/Makefile.in	2010-01-02 06:31:10.000000000 -0700
+++ xfce4-settings-4.6.4/dialogs/mouse-settings/Makefile.in	2010-02-14 11:52:59.000000000 -0700
@@ -177,7 +177,7 @@
 INTLTOOL_PERL = @INTLTOOL_PERL@
 INTLTOOL_UPDATE = @INTLTOOL_UPDATE@
 LD = @LD@
-LDFLAGS = @LDFLAGS@
+LDFLAGS = @LDFLAGS@ -lm
 LIBNOTIFY_CFLAGS = @LIBNOTIFY_CFLAGS@
 LIBNOTIFY_LIBS = @LIBNOTIFY_LIBS@
 LIBNOTIFY_REQUIRED_VERSION = @LIBNOTIFY_REQUIRED_VERSION@
diff -Nur xfce4-settings-4.6.4.orig/xfsettingsd/Makefile.in xfce4-settings-4.6.4/xfsettingsd/Makefile.in
--- xfce4-settings-4.6.4.orig/xfsettingsd/Makefile.in	2010-01-02 06:31:11.000000000 -0700
+++ xfce4-settings-4.6.4/xfsettingsd/Makefile.in	2010-02-14 11:54:03.000000000 -0700
@@ -173,7 +173,7 @@
 INTLTOOL_PERL = @INTLTOOL_PERL@
 INTLTOOL_UPDATE = @INTLTOOL_UPDATE@
 LD = @LD@
-LDFLAGS = @LDFLAGS@
+LDFLAGS = @LDFLAGS@ -lX11
 LIBNOTIFY_CFLAGS = @LIBNOTIFY_CFLAGS@
 LIBNOTIFY_LIBS = @LIBNOTIFY_LIBS@
 LIBNOTIFY_REQUIRED_VERSION = @LIBNOTIFY_REQUIRED_VERSION@