Install the DEs manually instead of from metapackages so ,out don’t end up with their entire software suites being installed. Additionally, probably use Debian instead of Ubuntu if you’re gonna be doing stuff like that, less fingers in the pie make for an easier tinkering experience.
Have you considered doing stupid shit and used Bedrock Linux?
It’s great, but it’s still baking