Then you close the tastkmanager the xfce taskmanager process is not kill. If you start the xfce taskmanger again a new process will be formed. So when you open and close the xfce taskmanager often you have to reboot the system or kill the process. OS: Debian 7.5 in Oracle virtualbox on a windows 8.1 host
Maybe that's because you have 'hide in systray' ticked in the preferences ? If so you can right-click on the icon in the systray and quit it.
Made hide in systray/show/hide status icon default to FALSE, should fix this behaviour by default.