It is true that distortion goes up a little bit at lower levels, but it is still very low and below the noise level: 0.05% at 100 mW. Look at graph 16 in particular, for the output range of 480mW to 7.7 W rms. In that range it never gets above 0.015% and rapidly descends to 0.01%. There is no way you can hear this.Sure you can- if you have high sensitivity loudspeakers!
What you are not taking into account is the nature of the distortion (mostly higher ordered harmonics) and the fact that the human ear uses the higher ordered harmonics to sense sound pressure. We are more sensitive to them than almost anything else! So they manifest in amounts this small as brightness. This is what has fueled the tubes/transistor debate for decades.