There is no way to indicate 'unselect' using the keyboard. I had expected Ctrl+space to work since space is generally 'select'. This is particularly irritating in Detailed List view since that view has no background to click to unselect all.

If I understand bug 3536, comment 1 correctly, this may not be possible since the functionality comes from Gtk widgets. Should this be filed upstream maybe?