In the opening, the OP said out loud that "We need to reject hard-to-drive speakers as being Hi-Fi". Now the tone was evolved to "... It's fine with me if you buy a car that gets 10 MPG. What I disagree with is the awe and bragging rights associated with that..." and followed by "No, your car is not better to ride in or more exclusive than high MPG Cars just because it's low MPG."
In reality, in the hi-fi world, I think very few audiophiles (if there is any) will procure speakers just because it's low efficiency. The speaker designed to have lower efficiency has its good reason. Take ATC speaker/driver as an example, if one desires a good low frequency extension out of a small-to-medium sealed enclosure (which I prefer) design speaker with a given size of driver, efficiency has to be sacrificed. Manufactures/designers could achieve higher efficiency by utilizing vented enclosure like tube/flat ports, lighter material for moving parts, horns, etc., but the sound reproduced from those designs may not suit some audiophiles' taste. In addition, quoted from the ATC designer, "The low mass parts are often not suited to the forces and resonances generated when reproducing audio at high SPLs and the structures that behave well at low levels can become unstable. When the driver was reinforced for its stability and high internal damping, reduced sensitivity is unfortunately a trade-off.