Currently, that setting is absolute (e.g. dim to 20% when inactive for X minutes). However, if I am already running at 10%, it will *increase* brightness of the screen after X minutes of inactivity. Either there should be a check against that, or the desired brightness should be handled relative to the current brightness (e.g. -20%)
There's a check already, added by the following commit: http://git.xfce.org/xfce/xfce4-power-manager/commit/?id=ab07509cdf0fa05c56ae61a020abce2ac1fb28cb (it's not part of any release yet, though)
*** Bug 8572 has been marked as a duplicate of this bug. ***
*** Bug 8329 has been marked as a duplicate of this bug. ***
It's fixed in git master and is probably even released in 1.2.0 (for sure in 1.3.0).