Where should I go from here with my digital front end?

@emil I also use EERO and had Lumin U1 Mini. I highly recommend Purist ethernet cable. It’s a sleeper.
If you can pick it up used its insane value. New retail is around $370. I A/B’d it with Supra ethernet cable(stock and the telegartner terminated versions) and there’s a significant difference. I sold the Supra.
Purist gives you lower noise floor, much better details and soundstage and more vivid presentation. 
If you want to spend more, the Network Acoustics Eno streaming system is very good but it’s more of a different flavor than the Purist in that the Eno does everything that Purist does but in a slightly more relaxed way. I like both the Purist and Eno.

I tried the fibre optic solution with Lumin U1 Mini and while it did lower the noise floor and presented a more detailed soundstage, it didn’t sound as natural as the purist or the eno. I ended up sending that stuff back. I would not recommend a switch as the Purist or the Eno will be enough of an improvement and result in less clutter. 

Upgrade everything if possible , a good usb or Ethernet cable are at least $750 each , ether regen helps  most critical Ethernet cable is where the hub is 

buy the best dac you can afford ,by far the most important component 

for all detail and resolution, sound signature start here . For your budget I would Highly recommend the Denafrips Pontus2 , ifyou could come up with $4500 the 

the New Denafrips Terminator2 12 th anniversary  but for $2k the pontus 2 beats 

many dacs at 2x the cost.

Here’s the thread I started where I posted more details on the comparison between different ethernet configurations including the optical set up.

@emil I have the same LUMIN streamer, it’s excellent. I also manufacture the ENO filter, so if you’d like to get in touch I can tell you about my experience with the LUMIN.

As others have mentioned, get a switch with its own clock, an LPS for your U2, experiment with the fiber optic route or an ENO filter, better cables, and of course a better DAC eventually. IME every step in the digital chain counts, and ignore the "its just 0 and 1s" trolls.