You have a really beautiful house. Looks like a wonderful view.
I don’t think you need to know what is wrong… or what your objective is. By trial and error you can find out what it’s capable of. As soon I have optimized all wires (which of course is an expense)… then I go after the venue.
The answer to your question: all the things you say you cannot change (in your system description). Placement, floor and wall treatments. These can make large differences.
Is moving your system to a dedicated / or at least acoustically better room a possibility? That much glass and open flooring are the first thing I would go after. You can see my system and see what I did was incrementally work on these things.