for better or worse, i might be the poster boy for this question.
i would do it, did do it, and glad i did it.
in 2003 after seriously considering adding a dedicated listening room to my previous home, which had a very nice sounding but small den where i had my system, i did move to a larger home with a barn mostly for HiFi purposes to build an ultimate room.
https://www.positive-feedback.com/Issue16/lavigneroom.htm
so guilty as charged.
now 20 years later it’s been a great ride.
OTOH such a large commitment is not for everyone. you get so far down the hole it starts to run your life. you need a wife/partner who understands you and can support that. and in my case my kids had not provided me any grandkids to dote on so my extra time was mine to use. and i just retired (likely delayed a few years by my audio habit) and had my 50th Wedding anniversary this past year. so life is good.
but careful what you ask for.