There certainly are more fitting volume levels depending on the source material, as has been pointed to - for some musical content more rigidly than others. Watching movies the reference volume level is adjusted from the most natural dialogue reproduction SPL-wise around which the rest of the soundscape then "settles" itself, and this can lead to rather staggering macro-dynamic outburst while also accommodating proper intelligibility in quieter scenes.
The sheer breadth in sound design and the variety presented with movies, both with regard to dynamic bandwidth as well as the specific sound design elements can be an impressive experience to witness when capably reproduced, while also being a tough test for one's speaker setup and its abilities into versatility; where for most audiophiles music may be the exclusive tool to evaluate system performance, I include movies (Blu-ray's/4K UHD's) as an additional testing means, finding this to reveal more effectively reveal the truer potential and possible limitations of one's speaker setup (+ remaining gear and acoustics), and the qualities I deem important here.
Unless as a background experience I prefer movies and well-recorded classical music reproduced at their more or less rigid reference volume levels, whereas other musical genres would seem more accommodating for enjoyment at lower levels. I love listening to jazz piano (like Keith Jarrett) at lower levels in the late evenings, as well as jazz in general, and fortunately I have the setup to make it come alive at lower levels. While as a system ability sonic proficiency of lower levels is vital to me I wouldn't want to be without the effortless presentation at the other end of the SPL-spectrum either.
To some loud playback may be regarded as the young individual's (fool's) game, but in measured doses and as mostly dynamic peaks emulating live events I find it to be indispensable reflecting musical (and movie) material. Indeed, since I started these audio endeavors my system has become better at playing at lower levels as well as loud beyond measure, which is what really matters in an effort to encompass most material.