Our ears have no direct depth perception mechanism like our stereoscopic eyes do. Our brain can simply measure the convergence angle of our eyes for things up close. Once the convergence angle gets close enough to zero everything is just far away. That’s why the moon looks like it’s the same size and distance as the sun. Or maybe you can see that the sun is over 200 times the diameter of the moon just by looking. I can’t tell.
But there are definitely indirect ways for our ears to perceive depth, and the most obvious methods are reverberance and tone. There’s also another potential way of perceiving depth with sound and that is lateral shift as you move slightly. As with visual perception, things close will seem to move quickly relative to your motion, while things that are far away will seem to follow you along. A strange effect of the phantom center image when using just two speakers is that sounds that are panned hard to each speaker will tend to stay put, so as you move your head left or right you will get proportionally closer or further to the apparent sound from each speaker. On the other hand, center panned images tend to follow your movement. If you move left, the center panned vocalist seems to follow you in that direction. This could conceivably create a sense of depth for some people, and may explain why they don’t like a center speaker for music. The center speaker will pin the vocalist to one specific location and that could make them seem closer. That’s a complaint I’ve heard and taken interest in because I am an oddball who absolutely LOVES a center speaker for 2 channel playback of music, but I try to commiserate with those who disagree with me. I really don’t care much about soundstage depth but I continue to perceive it on 2 channel recordings whether I’m just using 2 speakers or deriving a center channel.