Would you move to a bigger house if you could just to have a better listening room ?

In a heartbeat.  

When my wife and I moved from Nashville, TN to Charleston, SC in the Summer of 2021, one major deciding factor in the home we ultimately chose was indeed the 22"X25", dedicated listening room. It even has it's own private full bath and HVAC register.. It also helped that it is on the 6th tee box of our golf course and has a deck completely across the back. My wife got the pergola she always wanted.

I like things the way they are and I don't like large listening rooms! I'd probably move to a BIGGER property size so I can have my own livestock without neighbors complaining :-)

naw, I like to listen in the same space all the rest of my life resides in. I don't want to sequester myself in a little acoustically padded cell. 

I might be odd in that I consider music to be a social thing, while listening room seem decidedly antisocial.

I would not move on the basis of a single reason like audio room. But moving is typically predicated on many reasons or a big compelling one (like a job). Then, while looking for a new house, you keep your eye out for one with a good place for an audio setup.

I was really fortunate to have chosen a house 25 years ago during a corporate job move. The house had two floors and one floor turned out to be my man-cave… library, fireplace, audio room (one of the best I have ever heard), office, bathroom, huge storage room, changing room. The first three areas open and connected to make the acoustics excellent. I was lucky.