The single most expensive part of speaker manufacture is the laber by a huge margin. As an example my subwoofers use about $1000 worth of parts, drivers, and materials. One subwoofer enclosure has 168 hours of labor start to finish. Shop time is now $200/hour. This comes to $33,600! Retail is around 3 X parts and labor = $103,800 for one passive subwoofer! With modern tech in a shop set up specifically to make these subs will find efficiencies that could cut these figures in half. The best you could do would be $50,000 MSRP. This is why with rare exceptions you can not find a well built subwoofer in the commercial market. The entire construction process will be cataloged on Imgur when I am finished.
As for regular loudspeakers, the secret of fantastic audiophile speakers is not in the drivers. They are a dime a dozen. The secret is in the enclosure and crossover design. The DIYer has limited access to the kinds of test equipment needed to design SOTA loudspeakers but even now, advancements in computer and digital technology are making the speaker designer obsolete. You can have a preamplifier with a full 4 way digital crossover, room control and EQ. All the computer has to do is measure the system via a microphone and in 10 seconds you can have a perfectly designed loudspeaker for your room. You still have to build an enclosure that is functional and hopefully attractive, so there is work for the DIYer to do and be proud of.