What matters most in speaker design?

Weight.
The weight of all audio components relative to their function i.e only compare an amp to amp. The heaviest or most massive is always the better one.
Note I think the amount of wood in a wood or wood composite speaker can only be compared with another fundamentally wood speaker. Those granite thud rock slab speakers are cheating they get disqualified. Same with the lead lined cheaters thats not game.
So think about it which is better a thick highly braced enclosure or a thin non reinforced speaker enclosure. My JM Lab Focal Electra 938s are made with the same drivers (per JM) as the Utopias of that vintage. The Utopias weigh 3X as much.
Consider the drivers . Do you want that skinny excuse for a magnet always ferrite BTW compared to Alnicos magnets that weigh 10 pounds for a 1 inch tweeter and 30 or more pounds for a 10 -12inch woofer. Which is better. The bigger magnet. It makes for a more powerful motor in the driver it's true.
Finally crossover Skimpy three wind inductor a few small caps or....
So in sum, size or mass counts. Sorry fellow man, most woman are kind and reassuring but when they are alone...the size of your.... bank account really does matter.

No enclosure is best, I agree with Muralman1. That means planars, or certain special designs such as those buy MBL.

Soundlab, Sanders Sound, Quad EletroStats, Analysis Audio, Magnepan, Emminent Tech, Soundlab, GalloAcoustics, lots of cool choices.

Then the drivers, then the crossover network.

Resell value and brand recognition to be able to resell locally. ;)

Well the thing is, loudspeaker design is system design. Many an enthusiastic amateur has started out with "the best woofer, the best midrange, the best tweeter, the best crossover, and the best enclosure", and ended up with only an okay speaker. The professional designer takes everything into account at once, including in particular how it all works together.

I would say a good match between the drivers is necessary as a starting point, but in my opinion crossover design is the heart and soul of loudspeaker design. Cabinet design and construction matters, but it's not that big of a challenge in most cases.

Duke
dealer/manufacturer

If you are aiming to design one, go for it. Start from low cost speakers first around $20-40 each then enjoy and be proud of your hard work. Anyhow, a great completed set DIY from PartExpress can cost up to $900/pair while you could preferably buy a nice used set here and you can easily resell later.
Well, as you already know, a great set of DIY speakers can sound awful with a bad cross over design. A decent set of speakers can sound nice with the right x-over design. Guys from PartExpress.com can help you with more details and ideas. Sorry I can't give you a direct answer to your Q because I find that each one of them is crucial and somewhat related to each other.
Good luck.