Would you move to a bigger house if you could just to have a better listening room ?

Yes, and I did just that.

From being very content in a large, older building on Ft. Lauderdale beach where we were both very happy. I friend convinced me to buy a middle-road DAC and listen to some remastered Grateful Dead at 192Khz-24b....within a year I had acquired 600 watt monoblocks and then dual subs once the house and room were secured with a closing date. A ten month renovation followed and the "end-game" home surpassed my biggest dreams of the perfect listening room, less maybe some treatment panels....can't have it ALL.

My current house was chosen for its price and location. It just happened to have an area for my system that is the best I have ever had.

 17' X 22'  plus pony walls to kitchen and hallway, that increases the open area with an additional  11' X 22'.

The lanai has two large doors opening to the main room giving some "breathing room" of an additonal 14' X 22'.

Vaulted ceilings are 15' with the peak inside the  the main room a few feet.

The lanai walls are only screen and all windows and doors in all this area are always open. (Even though it is a neighborhood, I check with the people nearest me and they say they don't here my music. Maybe they are all hard of hearing.)

I went minimal on equipment after decades of many pieces and some quite expensive, but this is the best sound I have ever had in my own home.

Still, the Big improvement, most recently accomplished, cost me nothing except my own labor moving hard and soft surfaces and changing adjustments that the speakers have built into them.

Having this hobby for 50 years, it can still be surprising what rooms and room treatment will do.

Again, this home was purchased because it was a bank foreclosure in 2010 and it was exactly the area we wanted to live.  Luck on both accounts.

That's a little hardcore for me.  I make do with what I have.  Being an audiophile is figuring these things out as best you can.  Not like a dog chasing its tail.

I downsized from a 3600 ft 1854 home with 14 ft ceilings on an acre.  It was just too big for 2 people and down to I dog from 3.  My new home has a much smaller listening room, but I am good with my set up.

Yeah, sometimes big is too big and when downsizing you are in fact normalizing and upgrading.