I agree - avoid ambient and background noise - but NOT the ambient sound in the recording, which is there for a reason, like a band playing *together* and not each player on their own, the "good ambience" which is often pro-tooled away in modern remasters.
And adjust your speakers (work with this). And some - not too much - treatment of the room. the lower your volume, while still hearing the music clear and good - the better.
Another underestimated factor is the RELATIVE volume, or gain, in your system. If you have two or more gain stages, and can adjust the volume on these, it pays to pay attention to which volume settings that give the best sound. If your system chain is tuned in this way, chances are, you can turn the overall volume down, and still have good sound.
Note that, in my experience, this relative volume tuning varies a lot. Generally if the source is turned too low, the music sounds dull and insipid. If too high it becomes too loud, insistent and harsh. However it also depends on the quality of the gain stages. So for example, for many years, I ran digital music from a pc to active speakers in my home office. I discovered that including a tube headphone amp between the source and the speakers gave better sound. Instead of two gain stages - lowly solid state in the pc, and similar in the active speaker amp, I now had three, a higher quality component in between - and discovered a new rule of thumb - make the best gain stage carry as much of the load as possible.