Well, lots of people love audio by definition, from the folks tapping their toes while on the train, white earbuds jammed in their heads, to people who drive around the block, because their favorite song is on the radio and only halfway done.
As to why don't more people love the hobby of audio, because it's a pain in the butt. It's expensive, maddening and the pot of gold at the end of the rainbow keeps moving.
Rich people want elegance and convenience, oftentimes. There are indeed rich folks who are totally into high end. But again, they tend to get stuff, and keep it. Done. I think most people understand that music is music, and reproduction is reproduction. Songs that make me cry, have that quality whether it's a download to my phone, or at home in the listening room.
It's that quality that makes most people say, why bother? It's all a question of what you're into. I'm a racing cyclist, who pedals in anger on a rig that, retail, would cost north of 10k. Why? It allows me to maximize my effort. For someone else, a $400 Trek hybrid bike is all they need.
It's the same with audio.