Please make Xfce run the .Xmodmap script after it loads new X keymaps.
*** Bug 5574 has been marked as a duplicate of this bug. ***
This should be fixed in git master. .Xmodmap is processed when starting xfce4-settings-helper and when the layout/variant is changed using the Xfce tools.