Why Don't More People Love Audio?

It's not worth the effort. Make it simpler, with good sound, and they will come.
For example, are there any good one-box systems that can compete with separates, AND at reasonable cost? No. Instead your stuck with a cajillion wires, power cords, boxes. Is it worth the bother? For most people, the answer will be no.

macrojack -- Then there is Bose which populates big box stores and drives on the road it has paved with it's carefully managed reputation.

I object to the notion that it's too time consuming/requires too much work, particularly when it comes to the wealthy. To me, that's when it becomes easy. Walk into a well-known hi-fi shop, listen to their top setups, pick the one you like best, have them deliver and set it up. Working within a budget is the hard part.

I also disagree with the idea that most people's ears are "untrained", and find it a little arrogant to be frank. From the moment I first walked into a hi-fi shop and listened to that first note, I was floored. Why else would anyone of us have gotten into this hobby in the first place?

Personally, I think it's just lack of knowledge of its existence and financial priorities for most people. An iPod is a luxury audio purchase for many. How many are willing to spend double that amount for 36" of cable?

Throwing money at high end audio (or most things for that matter) does not gaurantee anything. I've heard plenty of uber-expensive stereos that sound horrible.
The rich, being that they are rich, aren't not buying because of a lack of money.

How many are willing to spend double that amount for 36" of cable?

Especially when the differences are virtually inaudible as no one can tell in double blind listening tests. Again, "most people" are not going to be willing to spend hours listening for differences between cables. Using measurements would simplify things, but "measurements can't be trusted".
People will spend money on something like an I-pod because it is tailored to their lifestyle. Going by stereo systems I see photographed on Audiogon, even many audiophiles are NOT going to put speakers out into the room even when it givers markedly better sound. And these are people who care about sound.

I think the vast majority of people don't enjoy music. Before retirement, I worked for Northern California's utility company servicing gas appliances. In 30 plus years of going into homes, I noticed that very few people have any type of audio equipment. If they did, it was an old record player, in a corner, collecting dust. It is as they they got older, their interest in music waned. I've seen many records packed away, the owners seemingly not caring to play them. More people have modest HT systems than 2 channel equipment. In my visits to approximate. 30 to 40 thousand homes, I've seen probably 10-12 high end systems. So here in San Francisco, the number of people with any type of music system is extremly low. Younger people enjoy music but they seem to get by with iPods and those little earbud things. Also, the few music lovers that are out there do not realize that a good system can be put together with relatively few $'s.