If one defines fidelity as when the output of the acoustical waveform matches what the signal is... Then it seems like it cannot happen to be correct until the time and phase response is correct. All things being equal, it will will not hurt.
A lot of people believe that we hear in the frequency domain. It is certainly easier to describe a system’ s frequency response in the frequency domain.
it is also also easy to describe a "system response" using the transcient response.
But a lot of scientific work suggests that in reflective spaces, Phase is not the primary issue.
Maybe swap your red and black wire on both the left and right speakers and test whether it sounds any different when the signal is reversed in phase... (some people easily hear it and some people do not.)