But I'm still kind of stuck on why the preamp is so important if I'm using the "Reference Mode" which is supposed to be a straight shot through. Unless it isn't.
It isn’t. A passive preamp is closer to what you’re describing (although even those have an impact on sound), and as your prepro is active it’s not even close to a “straight shot through.” The preamp is amplifying a small signal, which is why it can have a significant impact on the sound that’s ultimately produced, and many (me included) feel the preamp can make an even bigger overall impact on system performance than an amp although this can be very system dependent. Further, prepros are obviously multifunctional and most are designed to meet an aggressive price point, and the necessary result is that critical/expensive parts (power supplies, volume control, etc.) are compromised to meet aggressive cost constraints. This is why your $2500 prepro costs the same as or much less than many dedicated stereo preamps despite including all the multichannel, digital, and video processing as well. I’d guess if all the added functionality were stripped out of your prepro and was sold as a stereo pre it’d cost closer to $1500, which is fairly inexpensive especially as your pre is purported to be fully balanced that nearly doubles the parts required versus a single-ended design. So while using “Reference Mode” may deactivate unnecessary functions for pure stereo listening and reduce added noise/circuitry to some degree, the signal is still flowing through cheaper electronics than it would through a comparably-priced dedicated stereo preamp, and that makes a big difference. Hope this provides some helpful info as to why “Reference Mode” doesn’t provide a get outta jail free card as far as preamps go.