I think its cyclical, with a long, multi-year period.
I remember as a kid in the 1960s I saw some component systems on display at a department store when my parents went to buy a console stereo. (We eventually bought a Magnovox console.) When we asked the salesmen about the components, they pooh-poohed them by saying they were for audio enthusiasts and greatly higher priced. While we didn't buy, I was intrigued -- here was something new, beyond the realm of my prior experience.
When I got to high school, I had to walk past a stereo store every day on my way to school. Inevitably, I went in and was drawn to the quality of the components. I don't think I even heard them well, but the esoteric appearance and dedicated technology of the products on display captured my enthusiasm. This was the real deal! I saved my lunch money for a year and a half and bought the cheapest Fisher three piece system (two speakers and a receiver with a turntable mounted on top of it).
For me, the key was the first exposure at the department store, when I learned that there was something better, much much better, way above the whole mass of choosing between different consoles -- Motorola, Magnavox, RCA, etc. To be able to rise above the crowd, above the mundane, was something that kept me going.
Today, home theater is the draw. Customers go in to buy the TV, and many are made aware that there is a whole heirarchy of audio components. They understand that they are not buying the high end, but they now know it exists, and that there is something there, and there are people (us) who are enthusiastic about it and can tell the difference.
Like my family, they don't want to spend the money up front, but maybe the concept has some appeal, a little hook. I think they'll be back, at least some of them. I have faith the cycle will swing back to us, sometime.