That is the first time I have heard anyone expound on this subject. I truly believe that late night listening is partially psycho/acoustic in nature. For most, in addition to the factors that you mention, the human body tends to go into a more relaxed state, at least for most people anyhow. Even during the daytime I find that a largely darkened room where the mind loses visual contact with the sources from which the audio is being produced , tends to greatly enhance the clarity of detail, soundstage and depth perception. The mind tend to drift away from any sense of connection with the devices that produce the music and focuses more on the music itself, which is really what good audio was meant to produce.
One thing I have done since the realization of this concept came to me, is to always evaluate a new piece of equipment or media source, under the conditions discussed above. Also, interestingly enough, in the daylight hours, the same thing I was listening to, or evaluating, sounded much different. I have come to accept this as an absolute.
As an additional point of reference, I have also discovered that even daytime listening on SUNDAY, when the surrounding environment is settled down considerably; i.e. less external noises, less demand on the power grid, and usually, less demands at home, as an example, I almost always enjoy my listening time considerably more. I tend to feel this is all part of the psycho/acoustic phenomena.
I speak of course, as an urban dweller. I could quite easily accept that living isolated on a farm, for instance, or significantly far away from the urban areas, could also help change a persons perception of the day to day listening experience
Try it sometime. I would be interested in just how many audio enthusiasts would discover the same experience..
Of course I know there are those who would simply think I have a screw loose !
Maybe you're right !!.