Is digital contamination that big of a deal?

Yep, its a real issue in general so worth being educated about and alert to but lots of variables in play case by case so YMMV and it might in fact not be that big of a deal in most cases on the grand scale of things.

In my particular case, with multiple systems that play in multiple rooms, I find it always to be an issue to some extent but how much varies depending on exactly what I am listening to. The negative effects are typically greatest in cases where one is doing highly critical listening in good conditions with good gear that is well matched and performing well otherwise and a good recording to start with.

Wow! Thank you all for the thoughtful, insightful responses! It's great when you hear from good people with helpful input. What got me thinking about noise in my system and that I needed to get my system more quite was when I auditioned a completely passive (yet completely over the top expensive!) Stage 3 Concepts Kraken power cord. Brian Ackerman of Aaudio put just one Kraken power cord on my Pass Labs XP-30 preamp and man was I amazed! That one cord just to my preamp made such an impact on the sound of my system! I was completely blown away. The depth of the sound stage increased. One could hear instruments actually behind the other instruments - layering of the musicians - and the timbre of the instruments was so much more real. A wood drum stick hitting the metal rim of the drum actually sounded like wood! - not just a noise. These were things that I was trying to achieve with IC's and speaker cables. So this got me thinking.... For one thing, I realized that my system was actually pretty darn good. The information was always there it was just being masked. Now, my system sure seemed quite even before the Kraken power cord. I could never hear any noise when the system was on and no music playing. With the system playing music there was no indication of any "noise" either. My system appeared to be dead quite. So I started to think about what this passive power cord was doing to my system to make it sound the way that it did. The only thing that it could be was that the cord was some how making my system more quite/removing noise to where I could actually start to hear what my system was actually capable of. Yes, I think this whole area of making our system more quite, be it electrical or mechanical, are areas of big gains yet to be fully realized in the high end community. Thank you all for taking your time to respond.

As you've found out, a lower noise floor starts with your basic components.
The digital noise you asked about is yet another area of the system one needs to address.