For me, the ideal is to hear playback at the same level I would experience hearing the same performance live, except for some rock. Anyone hearing very much live music should be able to adjust levels by ear, without need for a meter. For symphonic music, I prefer row M, not row B, so my levels would be lower than some others might like. For jazz, folk, etc., I prefer 20 to 30 feet from the stage. This provides better integration of sound when there is more than just a solo artist. Rock will be played back at lower than live levels since I don't want my ears ringing for days afterwards.
But reality means many recordings will be listened to at lower than ideal levels. As others commented, recording/pressing quality becomes very important here. For example, choral music can seldom be listened to at live levels without distortion.
But a real mystery to me is why so many audio friends tend to listen to all their music at the same level (individually, not necessarily the same as each other)? They find one volume setting and listen to everything at that level. The advantage of a remote volume control is not pure laziness, it is the ability to fine tune volume from the listening seat, even changing from one track to the next when necessary.