@unsound I'm not sure why any single-driver cone speaker wouldn't deliver reasonably accurate step and impulse response. Sure, they may be compromised by dispersion, frequency extremes, SPL limitations, but so long as they're free from a crossover and, ideally, sealed or open-baffle (ie no port tuning), I see no reason for them not to mirror the amplifier's time response within the limits of their motor and diaphragm assembly.
I see why large single-panel planar speakers may be compromised as you describe.
Headphones are apples and oranges to me. Two very different and equally valid listening experiences, especially with the relatively recent advancements of higher-end headphones and dedicated amps. YMMV. My only point was that nearly all headphones are perfectly time- and phase-coherent, with none of the compromises required by single-driver or Thiel-like multi-way speakers.