Start with a more basic question - why don't more people love music? It's certainly true, in my experience, that I find a lot more people willing to carry on a conversation about music than audio gear, and I'm sure there is a majority of people, perhaps even a vast majority, who value music to a point that if you tried to take it away from them, they'd yell.
Now, put those people in front of audio enthusiasts and watch what happens. Like somebody with a nagging spouse, any love they start out with gets chipped away at rather than reinforced - "Rap isn't music", "There hasn't been any good music made since I was 18", "Melody is more important than rhythm", "mp3's sound crappy". And on and on. Maybe it's just me, but I'm into music and high-end audio, but exposing myself to others of the audio mindset, these are the messages that just keep being emphasized. It's basically all detriment, no reinforcement.
Assuming that somebody starts with a love of music before "discovering" audio, then they move into figuring out what to buy. And they get more negative feedback - "CD sound is terrible", "Why would you want gear that emphasizes aesthetics", "You have to spend X thousands to get good sound", "Why aren't you sitting still listening instead of talking or paying attention to something else".
The high-end community does nothing but denigrate somebody who is less into audio or music than they are. Spend only low 4-figures and you're buying "mid-fi". Sheesh. I doubt, seriously, that this is significantly different from other hobbies - I can imagine a thread somewhere called "Why don't more people love performance bikes?", coupled with comments about how we tell our potential biking-loving friends that you really don't need to spend more than $3K to get a decent bike.
I have taken to starting as many conversations as I can about music, and I never start a conversation about audio gear. It's amazing to me where I find people who genuinely are into music - I wouldn't even begin to believe I could guess from anything other than asking them, as the patterns are undiscernible. But I find LOTS of people who really enjoy music and want to share their thoughts, turn me onto music, be turned onto music, etc. The only time I talk about audio gear is when somebody suggests that they're thinking of buying something - I offer that I know a lot about gear, and could offer advice on getting the most for their money.
If Audio == high-end in the sense that audiophiles tend to define it, then I think the answer to the original question is because it's so niche that it will never appeal to a wide audience. If Audio means music and music playback systems, then I'd contend that tons of people love audio, they just love it for different reasons than "pure" audiophiles.