How loud are you typically listening?

I can speak to the specific case of Classical music. I usually set the volume so that the peak crescendo hits roughly 86db on my RadioShack sound pressure meter. I find this seems to mimic a real world concert listening experience.  This puts the softer sections down in the 50-60db level. 

Indeed. One of my favorite moments at a concert is when everyone pulls out their SPL meters to check the crescendo level. I look around at the readings and this way know where to sit for next time. But then at home I worry, am I setting the volume for where I like it way in back? Or should it be higher for where the recording engineer thought the level should be? These problems worry me so much I can never enjoy classical.

Depends on the music, but the loudest is just under the distortion level.

What do you mean? IM or THD?

https://youtu.be/uio1J2PKzLI?t=14

Casual listening - mid 70's

When the family is out - mid 80's

When I want to scare the neighbors - upper 90's