Changing font anti-aliasing settings doesn't apply on-the-fly, only maybe for subsequent applications. But if we check or uncheck the 'custom DPI' box then the new anti-aliasing settings apply, so as a layman I suppose this could be trivial to implement. Current behaviour is misleading as there is no explanation (like tooltip or other). I thought those sub-pixel rendering settings didn't make any difference but when I learned how to apply the new settings I saw the effect and literally made my fonts go from a garbage rendering (was set to 'none') to a pretty acceptable rendering (set to 'RGB').
The changes are immediately applied to GTK2 applications. It needs to be investigated why the same does not happen for GTK3 ones.
Same behavior when using MATE or GNOME, so I suggest that you forward this issue to https://gitlab.gnome.org/GNOME/gtk . An instant refresh can be forced by temporary changing the DPI, font (size) or window scaling, but that is an ugly hack.
Opened https://gitlab.gnome.org/GNOME/gtk/issues/1995
To Theo or other interested developer, if you're willing to further on this issue I think upstream would need your feedback as it was said that GTK2 and GTK3 are different and thus the Appearance Settings app would need to use another approach to work now.
I really don't think this is our fault/bug, but who knows... - in any case I replied upstream.