Should we pay more attention to delays in sound?

Sounds like we already do, like it or not. 

It depends. Half a millisecond is, I believe, around 7" worth.

However, interferometry teaches us that small misalignment between drivers and crossover implementations would be completely audible even at a few millimeters.

So, matching your tweeter to midrange to woofer electroacoustically is critical.

Despite many studies we have never seen consistent acceptance or demand for speakers with perfect phase/delay response.

What is my point to all of this?  That delay mismatches are complicated.

I've been saying for years speakers must be precisely equidistant and symmetrical. The researchers could have saved an awful lot of time and money simply asking me. Seven inches, ha! Even a fraction of an inch has an audible effect on image focus.  

Don't need no anechoic chamber. Don't need no measurements at all. Simply set your speakers up with tape measure and framing square to be perfectly symmetrical and equidistant. Listen. Then move one an inch or so. Listen again. 

Once again the reality of listening proves superior to measurebating.

millercarbon, you need to make up your mind. In regard to subwoofers you have stated that "time does not matter." 

When it comes to crossovers it matters a lot. Group delays can be caused by positional factors and electronic factors. You can adjust any set of group delays to be exactly the same. This can be done by moving the slow drivers towards the listener or digitally delaying the early drivers to match the late ones. 1 ms = 1 foot (at sea level). I can usually tell something is wrong by 0.2 ms at low frequencies but have never blinded myself to this so it could be longer. I can tell the difference in bass impact but I am the one adjusting the delay. Of course I set them all to zero.
The vast majority of audiophiles do not have this capability which IMHO is a shame. It is vital for proper subwoofer management and imaging in general. It is the difference of hearing one vs two voices. 

I waited a couple weeks to get my Herron, almost two months to get my Moabs, and now waiting on DHL to deliver my Origin Live Sovereign turntable. Oldhvymec waited even longer on his Decware ZP-3. And that is nothing, after saving for more than a year for a Raven Reflection when finally ready to order was told it will be more than a year! So yes someone somewhere definitely needs to pay more attention to delays in sound!