I have an USB headset, and it works fine with the mixer plugin. I can control the sound properly.
But when I unplug the headset xfce4-mixer-plugin starts using all the cpu (100%).

I'm using Debian packages from debian.corsac.net and I'm running 4.6 RC1.

Steps to reproduce:

1 - Connect USB headset.
2 - Add mixer to panel
3 - In mixer configuration see if headset is selected, do not really select it.
4 - Unplug USB headset.

Please let me know if you need some more info to replicate this.