Some programs use this as a keyboard shortcut (I need it in intellij IDEA), and there seems to be no way to make xfsettingsd not grab that key. Ungrabbing it with xdotool key XF86Ungrab works only for a fraction of a second before the program grabs it again.
Looks like it was just my stupid issue. I accidentally set a keyboard shortcut that didn't visibly do anything for the alt key.