If you think about what is going on, the effects of a neutral cabinet motion are critical to getting the best from your speakers.
For example, look at a high-end scan speak tweeter, the one's I call the cone-heads. That big spike in the middle of the tweeter is designed to make sure the CENTER of the tweeter move about when the EDGES of the tweeter, where the voice coil is attached, moves. And, to beam the energy more sideways so it's arrival time is more room "reflections than direct, improving resolution. A mechanical device is not infinitely stiff, so the middle of a round object, with the "push" coming from the edges will have a ripple effect towards the tweeter (in this case) center. If the middle of the tweeter is mechanically "soft" it will actually lag behind and dimple inward compared to the edges and add significant Doppler / intermodulation distortion.
Some tweeters use central phase plugs to damp this out, but all of them are aware of the shortcomings of a too soft structure and manage it somehow...apparent or not. The stiffer a structure is however, the harder it is to damp once it gets going.
Look at the distance a tweeter moves...Well, you do it, I can't see that well anymore. Its motion is virtually invisible to the naked eye. Want crisper sharper imaging? Add a bunch of cabinet resonance to the speaker front panel that is orders of magnitude worse than the tweeters full excursion. The tweeter doesn't stand a change to be at it's best.
Bass is more controlled by the cabinet itself as the wavelength / excursions of the driver are at the opposite end of the spectrum, much longer than the cabinet resonance. Lets hope so anyway. Still important, but not the pinnacle of perfection the mids and tweeter demand.
Look at speakers as they get larger, and harder to damp. A HUGE part of the price is stopping the darn cabinet from moving. Big speakers carry a lot of extra "loose" baggage that has to be managed, at a big cost. Internal braces and exotic shapes all to simply get the drivers to stand still. When the braces are added, it steal cabinet volume from the drivers and the cycle feeds on itself as the cabinet is made bigger to account for the lost internal volume the braces removed. OK, use carbon fiber and it feeds on your bank account. You pay all that money, and then let the speaker wobble all over the floor? Want to know why so many fantastic smaller speakers exist? Well, now you know...the cabinet is order of magnitude stiffer as they get smaller all things being the same in the design. A panel half as long is four times stiffer.
But, once you select a speaker what ever it takes to keep that front panel from moving back and fourth (the speaker cabinet itself is a done deal) is as important as any component in your system, probably in the top three (and it's actually pretty cheap to optimize).