DAC diminishing returns

Looking for some experienced input on the diminishing return as you go up the the DAC ladder. I mostly stream music and occasionally listen to CDs.  Currently, I’m using a Schiit Bifrost Multibit DAC.  It’s connected to my Naim Nait 5Si and that drives Zu Omen ii speakers. In my limited experience, I have not heard any substantial differences when changing DACs.  I’m considering a Denafrips Pontus DAC.  Does the significantly higher cost justify what might be a marginal increase in sound quality?  Is buying a more expensive DAC for better sound just splitting hairs, and making you feel better about your system?  Are different DACs just different coloration of the sound?

I don’t want to be duped into buying something I don’t really need. Thanks.

Post removed 

Lots of really good suggestions and advice here.  In my mind none are more important than trusting your own ears.  I upgraded my DAC less than a year ago.  I set a budget and started to audition.  I had a dealer come to my house with 4 different ones on tow all at varied price points.  I heard subtle differences between all of them but not one was good enough to make me want to get rid of what I already had in my system.  He then plugged in a Weiss 502 and, for me, it was like crossing the street and getting a completely new perspective on what I was hearing.  This wasn't a small upgrade, it was a leap.  It was double my budget but I went into the safe and peeled off the cash.  Every day I listen I'm happy I did.

@firstgrowth Well, I didn't want to read that but I can believe it. The 502 seems very impressive and I am probably right where you were at. I am currently running a Denafrips Venus II and am waiting on a new HoloAudio May KTE, hoping this will be my end game DAC as the 502's price point is double my budget. Fingers crossed. 

@firstgrowth , the 502 looks like a cool toy. I mean that in a good sense. Do you personally set and forget it or change the settings to match the music?

Until earlier this week I would have agreed that the law of diminishing returns does apply to DACs (as it does to other hifi equipment in my experience). I started my streaming life with a Logitech squeezebox with built in DAC, moved on to their Transporter, then to a Lumin D1 and from there to separate DACs. First a Chord 2Qute, then a TT2, then added an M-Scaler (upsampler). Last week, I managed to find a good deal on a used Chord Dave. I was expecting similar incremental improvement as with all my previous upgrades, like slightly better placement, separation, somewhat tighter bass perhaps, more natural voices, even less sibilance, better reproduction of ambient sounds/noises on the recording. But I have been absolutely blown away by this upgrade. While it is bringing improvements on all these fronts, it is in doing so in spades. It is as if I am listening to a whole new system, sometimes suggesting multi channel presentation even though it is stereo. It has by far exceeded my expectations. Now, in fairness, the Dave, even at the second hand price I paid, was about double what I paid for the previous TT2. I leave it to you to consider what you would find a reasonable expected return for that extra money, but I have been surprised and blown away by the improvement it has brought to my system. I’m afraid the only way to establish the improvement a DAC can bring is to try it out. Hope you will find an upgrade that brings as much listening pleasure as my most recent one!