Speaker manufacturers or box manufacturers with off the shelf drivers?

I Think it comes down to being a smart consumer an educated audio enthusiast and training your ear/brain. Speaker design is an art and a science and its up to you to decide if a speaker represents a good value for your budget and tastes.

Also don’t forget that driver designers are experts at what they do and make them often at an economy of scale that would be much more if they were bespoke for each speaker company.

To use a car analogy, a lotus Elise
I once rented had a Toyota engine and transmission. It was a fantastic exhilarating ride and if I didn’t live in a NYC I’d probably own one. Did the Commuter car components (though tweaked) take anything away from the car? No, in fact it made it affordable and reliable.

This is fascinating.  I am a small speaker manufacturer and drivers to me are one of the thing that stressed me less.  There is a lot of work that goes into choosing the right driver in terms of testing with your cabinets and materials but there are lots of bigger manufacturers that use stock drivers that make AMAZING speakers.  
To name a few, Wilson (small mods in some cases) Avalon, Marten, Joseph Audio  all use stock drivers.  These guys make some of the best products in the world and they use stock drivers from Scanspeak, Eton and Accuton, Accuton and SEAS from the companies above. 
I have been told that Revel has since moved production internally but the drivers in The Performa3 line were SB Acoustic when it launched.  Doubt this?  I still have a pair from years ago with SB drivers in them.  The tweeter had a waveguide designed by Revel.  That was what made it their own.  
There are crappy speakers out there and you can spot them and avoid them.  Check out this series of articles from audioholics:

https://www.audioholics.com/loudspeaker-design/high-fidelity-loudspeakers 

See links at the bottom of the page for other relevant topics like this one on the explicit elements of a driver:

https://www.audioholics.com/loudspeaker-design/loudspeaker-drivers

in the series they touch on all aspects of speaker design other than utilization of exotic cabinet materials (aluminum, fiberglass, carbon fiber, stone, etc...) but those materials are clearly superior to MDF based on guiding principles outlined in this series.  
I will admit that i designed my products so that if Audioholics took them apart, they would be pleased with what they found.  
Good luck to you in your shopping.  

Most of the speakers I have owned and liked, with one exception, make their own, especially Magnepan.  The only speakers I have owned with off the shelf drivers were Dahlquist DQ-10,  

I am sure if I were to start a "speaker company" tomorrow, it would be easiest to build a box and use some ready made dynamic drivers. 

Lets bad rap the world since were home an board.

It's mostly about economy of scale and available resources. Most speakers companies are small compared to main steam manufacturing. It's impractical for most speaker companies to build drivers in house especially when considering they use eight to ten unique drivers in their lineup. Furthermore, the skill-set and tool-set are somewhat different. There are people and companies that can do it all but that's rare and usually yields expensive drivers and products. Every manufacturer has vendors. No speakers company makes everything. It's much more expensive building your own drivers, unless you can build and sell many thousands of them, than buying from a vendor.

We are a small speaker company that decided to design, develop and build drivers for our own use. We build our own AMT drivers. It was a major investment in many ways. We went through the process because we wanted something very specific which didn't exist. We have had seven speaker companies (4 well known) contact us about using our AMT drivers. We politely declined because we don't want to be a driver vendor.

As with most things, it's more about good design and implementation than the actual parts. Aluminum, wood, MDF, carbon, carbon fiber, concrete, gold, unobtainium, plastic, plywood or whatever all have their place with their own properties. Anyone stating one material is better than the other without showing multiple materials optimizes for that specific application is engaged in marketing. Cost is part of the equation.

There are some brilliant speaker designers that can build excellent speakers using off-the-shelf parts at many price points. Buy what sounds best to you. What's inside the box is secondary.