Fear of volume control

I have found that listening at 80-85 dB is where I get everything from most recordings. I have a slip of paper in each of my albums that indicates what level on my amp makes the recording sound the best. Some are at 48 on my Hegel H390 and some are at 56 with many falling in between. I just listen until it sounds the best and then I measure the dB level at my listening position. The average dB is almost always between 80-85.

The loudness switch is a really good tool.

Not only is Fletcher- Munson at work but also the volume the record was mixed at. Every recording has a "right" volume depending. If you play a string quartet at the volume of a rock record the strings will cut your throat. If you under cut the volume of a rock record it will sound dull and lifeless. If you have loudness compensation you have a little more flexibility but still. 

People who shy away from louder volumes generally have systems that start distorting at lower volumes. Distortion is at lower levels registered as volume. Obviously, they enjoy their systems at lower levels thinking that volume hurts. Not true. You can destroy your ears, long before clean music becomes painful. This is why I use a sound pressure meter not wanting to get carried away. Very clean systems with adequate headroom can play deceptively loud. People will try to start a conversation not realizing the other person can't possibly hear them. 

This is why distortion in loudspeakers and powerful amps are so important. Powerful means power relative to the efficiency of the speaker. and and the maximum power the speaker can handle. IMHO as a rule of thumb 105 dB is a reasonable target. So, you figure out the power required to get your speakers to 105 dB then multiply by ten ( to handle transients). This is a reasonable power for your speakers. Every 3 dB doubles the power. I am of the belief the more power the better as in my experience powerful amps have an effortless quality missing in lower powered amps. I have to admit there is probably some psychoacoustics in this as I have never been blinded for this evaluation.  

high quality acoustical treatment is the way to keep the volume levels down. And stay sane.

Increase the dynamic range and reveal the more accurate shape and subtleties of the music --- by removing the noise at the bottom.

Not by cranking the volume levels, a tactic which merely makes it scream over the inherent noise levels in the room, and simultaneously pollute your low level signal subtleties. (you know, the subtleties in dynamics and signal that we are here -in this particular world of audio- to try and resolve)

room shook,

if not exaggerating, that's when Fear of Hearing Damage should set in.

it's worth learning about Fletcher & Munson's discoveries, 

https://en.wikipedia.org/wiki/Equal-loudness_contour

It has always been poorly: named, implemented, mis-understood.

the use of 'loudness' control actually should be called 'Low Volume Listening'. 'Progressively' reduce the volume and boost the lows and highs so that we hear the content as enjoyably as when it is loud (as you discovered). The 'loudness' controls on vintage equipment did/do just that, much modern equipment skips tone controls, balance, loudness filters ...
....................................

You can easily put this unit with Automatic Loudness Control in front of any featureless equipment:

https://www.ebay.com/itm/274870895426?hash=item3fff950342:g:iiEAAOSw0tJg8tiE

The implementation needs to be done so that:

1. all or chosen sources into Chase Unit.

2. Chase RLC-1 default volume level (just unplug it, it goes to defaults). F-M is NOT engaged.

3. Sound System's Volume set at normal listening level. Leave it there.

4. Chase RLC-1 to remotely adjust volume. Up, just less attenuation by the chase unit. When you lower the volume, the Chase unit Automatically begins to progressively implement F-M curves.

thus progressively adjusting for our hearing system's properties. i.e. you still hear the Bass Player in a Jazz group, the highs are still there. You will find the music maintains it's Involving quality, rather than fade to background music.

5. Oh yeah, the Chase unit has remote balance, gotta love that if your system has terrific imaging ability. 

................................

Misunderstood: some people used to defeat the Chase loudness feature, they 'hated' the bass bloat. well, that's because they lowered it into F-M implementation and boosted that correction with the amp's volume, iow, Misunderstood Again. So poorly misunderstood that the feature has disappeared. 

IMO, if you call yourself an audiophile, you ought to fully understand our hearing system and go get yourself some 'Loudness'. And avoid needing to get some hearing aids if possible.