There are a lot of nice comments here. One thing I think needs to be emphasized is how important the match is between the amplifier and loudspeaker!
For starters, all amps make less audible distortion when driving higher impedances. To this end, if a 4 ohm speaker were somehow magically made into an 8 ohm speaker, it would immediately sound more relaxed and detailed regardless of the amplifier driving it, simply because the amp would be making less distortion and the speaker cable would be having less detrimental effect.
There is a similar benefit going to 16 ohms- with the added benefit that the speaker cables become far less critical.
There is also the issue of the use of loop negative feedback in amplifiers. While loop feedback tends to suppress a lot of distortion, it also adds some of its own. Unfortunately the kinds that it adds are easily heard by the human ear, which translates all forms of distortion into tonality. In this case, the tonality is 'brighter' and 'harsher'.
(FWIW, the Ayre and some of the Pass amplifiers are zero feedback and that IMO and IME should not be regarded as coincidence that they are some of the best-sounding solid state amplifiers made.)
This is why loop feedback is avoided by many designers and used reluctantly by others. Of course, loop feedback also helps the amplifier control the load by reducing the 'damping factor' of the amplifier. You may have noticed that I used quotes there. I do that with 'output impedance' as well as both are charged terms that don't mean what they sound like they mean, which leads to a lot of confusion in the marketplace and indeed with designers themselves.
(Loop feedback is said to reduce output impedance, but what is really happening is there is a servo loop that occurs between the amplifier and loudspeaker. If the output impedance were really being lowered, the amplifier would be able to drive progressively lower loads without strain. This would violate a fundamental rule of electricity known as Kirchoff's Law
https://en.wikipedia.org/wiki/Kirchhoff%27s_circuit_laws.
The only way you can drive lower impedances is with more output devices, larger output transformers and or heatsinks, bigger power supplies, etc, else the impedance of the output section might not be any different from that of the load, meaning that much of the power it makes will be dissipated by the output section itself rather than the load. Adding feedback to an amplifier does not change how much power it can make into a lower impedance, ergo its output impedance is unaffected. This is what I mean by a 'charged term'; in audio, 'output impedance' is not a measure of the actual impedance of the output circuit and is instead a measure of its **response** to lower output loads.)
Anyway, loop feedback is really there more to control the frequency response of the amplifier when presented with a non-linear load (IOW, all real-world loudspeakers... this idea was originally proposed and championed by EV and MacIntosh in the late 1950s and into the 1960s). The idea is that if there is a peak in the impedance curve of the speaker, it represents a resonance, so the amplifier will make less power into that peak. Similarly the amp will be asked to make more power into lower impedances. You can see right away that this rule has a lot of application with box speakers.
The problem with this is that not all loudspeakers (many of which are highly respected) work properly with this scenario, called the Voltage Paradigm (as the ideal amplifier for this task is one which can make constant voltage into any load).
One of the problems is that all speakers have non-linear frequency response, so even if the amp is giving the right response the speaker will not. The other problem is not all loudspeakers have impedance peaks based on resonance. Electrostatics are a good example. The Voltage Paradigm relies on the idea that the impedance curve represents an efficiency curve; the simple fact is that such is not universal.
So the result is that a given amplifier won't sound right on all loudspeakers.
The Power Paradigm is what was around prior to the Voltage Paradigm. In this scenario, the amplifier tries to make constant power rather than constant voltage into all loads. None of them are successful at this, just as no Voltage Paradigm amplifier is 100% successful either. However, an advantage of the Power Paradigm is the recognition of how the human ear works- in that distortion is converted into tonality. The Power Paradigm avoids the brightness and harshness caused by loop negative feedback. It trades off this advantage for the disadvantage of non-linear frequency response, but it is understood that the ear has tipping points, one of them being that it can easily favor tonality from distortion over actual frequency response errors. We see this all the time with the brightness issue.
This is why many older (Power Paradigm) speaker designs and a number of current ones employ level controls for the tweeter or midrange. They are there not for room correction but to adjust the speaker to match the voltage response of the amplifier which is an unknown.
There are a good number of Power Paradigm devices for sale today in high end audio. SETs are a good example. Horns are another; ESLs and planar magnetics being more. Now its a simple fact that if you mix Power Paradigm technology with Voltage Paradigm technology, you are going to likely get a tonality aberration. An example is the brightness encountered when a transistor amplifier with lots of feedback is installed on an ESL. Another is the 'honkiness' experienced with many horn designs when their crossovers don't work as expected when the 'output impedance' of the amplifier driving them is too low (which can cause the woofer and midrange drivers to operate outside of their expected range).
I've noticed in this conversation that very dissimilar amplifiers are mentioned in the same breath, as if one were somehow able to compare them on the same speaker, which is likely to do one of the amplifiers a disservice! That is why I brought this up- matching between amp and speaker is pretty important and makes comparison really tricky at best!!
more info:
http://www.atma-sphere.com/Resources/Paradigms_in_Amplifier_Design.php