Why are audiophiles perceived as being wackos?

I have been pursuing this wonderful hobby of high end stereo equipment and music for almost 30 years. I do consider myself a passionate audiophile who loves to listen to music on a daily basis. The reason for this post is because it has been my experience that the great majority of people who do not pursue this hobby think of audiophiles as being wackos/screwballs because of the amount of time, money, and passion they put towards their systems. I find it very interesting that individuals who spend tremendous amounts of money and time pursuing esoteric tastes such as wines, watches, coins, etc, are considered connoisseurs or aficionados with a serious passion that is often respected even if it is not the other person's cup of tea. Another example would be people who love high end cars/boats who read all the magazines, go to the shows and invest large sums of money to purchase and tweak their cars or boats to get the last bit of ultimate performance out of their prized possessions. So I don't believe the negative viewpoint towards passionate audiophiles revolves around the amount of money that they can invest in their equipment. So my question for all you GON members is what would be your explanation for people outside our hobby having such a negative or condescending attitude towards our passion, where they infrequently would not have the same attitude towards other people's hobbies and passionate but unique pursuits? It would be a pleasure to hear about your experiences and what your explanation would be for this phenomena towards our hobby.
Pbb's explanation that "High fidelity is an ersatz", as convenient as it seems, may be a pretty good insight. Re-creating music via high fidelity equipment is rather sort of an ongoing science experiment to which there is no conclusive answer. There is no one definitive solution and all results are highly subjective. Evaluation involves extended and focused listening - and not necessarily to music but to 'sound quality'. How frustrating and cumbersome it must seem!

The appeal of a high performance car or a fine wine seems rather straightforward by comparison.

There is one audio manufacturer that stands out as breaking the audio = geek rule - Bang & Olufsen. It's a very high prestige product and it's even an instrument of seducing women. But audiophiles are generally not interested in it because it represents poor value in terms of sound quality for the dollar and you can't tinker with it. It is first a work of art and a visual statement. It plays music rather well, too, and can set quite an ambience it the right minimalist space.

I think the level of geekiness depends on how obsessive one gets about gear (how much time one spends switching out power cords) - vs. - buying excellent, unobtrusive gear and letting music enrich your life.
Because they seek to justify their existance, rather than just existing. I can't beleive there are audiophiles who try to justify their purchases to their friends.

What would'philes think of me if I tried to justify the purchase of a $400 bottle of wine, or $700 worth of wallpaper for my 10'x14' office?

They do not have the point of reference to be able to understand audiophilia.

Or maybe it's just because it sounds too much like pedophile!?!
Most people have no true understanding or passion for music they accept what is shoved down their throats by the radio stations. They are satisfied simple beats and catchy tunes and really do not listen to the little things or take time to think about the fact that music is is about heart and soul.They just cannot understand why someone would be so passionate about listening to music since they really can't hear what we do.