Some tracks sound best at a certain volume for the simple reason they were mastered to be that way. This is due to our hearing not being linear but corresponding to equal loudness contours mapped by Fletcher and Munson. We do not hear all frequencies equally well at all volume levels. Low bass and high treble in particular need to be at a fairly high level to be heard at all. This is why the Loudness control used to be so common. All audiophiles should know this but whether they do or not all recording engineers certainly do.You are perfectly right ...
But we must not so much compensate this difficulty of equal perception between frequencies with higher volume than with an adequate room acoustic control to make easier the clarity of perception...
The better a room is controlled relatively to the speakers/room response the less we need to increase the sound volume to perceive details and music..