I agree with Rec and Gregm. The overall customer base is relatively small, there is very little support infrastructure, and something not mentioned by either - a fair number of us (part of your customer base) are natural-born cheapskates. I, too, would highly recommend starting your efforts as a "paying hobby." As Gregm noted, the "value proposition" of high-end audio is "weak." The most successful dealers I have run into have something which makes them "sticky" as counterparts. They cater to their customers like personal servants, they specialize in one particular kind of item (only independent manufacturers, only analog, etc), or one particular kind of sales (low margin high turnover, low turnover but one-of-a-kind (or very rare) items), or they have a personality which drives them to be mavens. In each case, the "value" is in the person offering the goods, not the goods themselves.
As a personal aside... Every now and then I try to think of myself trying to make a business out of my hobbies. I have been able to imagine success in a few different hobbies. Audio ain't one. There is A LOT of competition out there.
That said, dreams are there for a reason... best of luck!