The fact that all the data packets arrive in the proper order and number isn't the key thing here. Right?
Isn't the real issue the quality and speed of the clocking that occurs within a switch? The noise is certainly an issue and also is worthy of attention.
And another key issue is the bit rate through the ports. Audiophile Grade switches I believe better control the ports. Basic switches leave the ports wide open Vs. Higher grade switches for sound are scaled down to a much lower Data rate and by doing so it reduces noise. Not sure why but it sort of makes sense.
But key Is the clocking which when done well favorably impacts jitter and improve sound obviously. Basic stuff I guess.