What matters most in speaker design?

If you are aiming to design one, go for it. Start from low cost speakers first around $20-40 each then enjoy and be proud of your hard work. Anyhow, a great completed set DIY from PartExpress can cost up to $900/pair while you could preferably buy a nice used set here and you can easily resell later.
Well, as you already know, a great set of DIY speakers can sound awful with a bad cross over design. A decent set of speakers can sound nice with the right x-over design. Guys from PartExpress.com can help you with more details and ideas. Sorry I can't give you a direct answer to your Q because I find that each one of them is crucial and somewhat related to each other.
Good luck.

Duke, a lot of the Klipsch Heritage speakers used to have the 3 dB point at 17.5 kHz. I assume this level helped smooth out the sound and mitigated some of the horn harshness. I noticed your speakers quote the same number under the category of system bandwidth. I think this top end is mostly a tube "limitation" and you mostly recommend tubes for your speakers don't you? Is that what you're getting at - a speaker for tubes? We're always quoted 20 to 20 as the range of human hearing. Are those last 2.5 kHz nothing but irritation? :) I always read with interest your posts about speakers and the obvious expertise. I'd like to hear a pair out here in the southeast.

C - crossovers.

Cabinetry, ala veneers and finish rule the day, with exotic drivers following closely behind. Cetainly a smart strategy, as this hobby is one where people listen with their eyes and their wallets.

However, in my own experimentation, by varying the cap/coil ratio, I could make my own loudspeakers become as forward and white sounding as a pair of Lumenwhites or as lush and full-bodied as Vandersteens, and of course, anything in between those two points.

Along those lines, the unfortunate thing is in this day and age, we are as far away from having competent people designing crossovers as we have ever been. Why? So many of the people designing loudspeakers today are deficient in mathematics; to the point where they run away from it.

For example, one loudspeaker manufacturer who I know, and has won awards at shows got his crossover design from a "friend" in the business. When I looked at it, the numbers were so far off, I jumped up to ask him how he came up with such off the wall(being kind) values, which is when he told me. Knowing the friend's talents lie in many areas of loudspeaker optimization, but not in designing (although he surely can upgrade/mod them to improve sonics) crossovers or mathematics, I approached the friend about the design, and he admitted he knows almost nothing about how to design a crossover. Where did he get these values? He kind of mimicked another speaker on the market he liked, but wasn't 100% sure if he was using exactly the same values. My money says that given the difference between commas and periods on either side of the Atlantic, a decimal point got lost in translation. What's worse is that even after a fundamental and major redesign of the speaker, the crossover has not changed one iota, "I have a crossover that works, and am afraid to mess with it." Really? I've told him the more appropriate values several times, but hey, the parts have already been bought and paid for.

I was thinking about speakers that use relativly cheap
drivers, but are executed in a genius enclosure with a good (but not exotic)
crossover - and they sound absolutely amazing.

Unless the manufacturer starts with high quality drivers then they won't ever be
accurate even though distortion and coloration can sound pleasant for some
provided the design is cool looking and has a nice finish. Distortion is
cumulative. Of course the drivers also need to be matched appropriately and the
woofer to the cabinet - so there are still many ways to mess things up even
when starting with good drivers.

I have a friend who designs speakers. I'd describe him as brillant - his dad is a department head at UVA. My friend has advanced degrees and has written at least one very thick book on signal processing.

I watched him build some speakers a few years ago. Very thick cabinet, the finest drivers, a lot of labor. I watched him work for months and months on the cabinet and later the crossover. He approached the cabinet design and x-over design from several angles eventually using some commercially available software for the x-over. He ended up with a perfectly flat frequency response and speakers that didn't sound too good.

Next he found someone with a lot of experience designing crossovers and worked with him for a few more months. They sounded better and eventually they got it to sound very good. He told me at one time he was thinking about adding some foam or rubber around the tweeter to take the edginess off the highs.

He took his speakers to a DIY show. I went as well. Basically he didn't win with the basic gist of it being the speakers weren't lively or forward enough. I didn't get it then but now I figure he'd built the kind of speakers you could actually sit down and listen to for a good, long while. They just didn't pass the "sip" test. The "sip" test is the first few minutes with the "boom and sparkle" jumping out at you.

Two important points I got out of this experience. First, all the parts he used were among the finest available and used in some very expensive speakers. But the drivers and crossover parts didn't cost much relatively speaking. Lesson 1: you pay as much or more for the design of the speakers as for the materials.
Lesson 2: A flat frequency response is no indicator of how good a speaker sounds.