I kind of know how you feel. I always think there's better sound out there. But not sure there is.
My system is somewhat similar to yours TP-Link Router to Netgear GS105 consumer switch (it's supposed to be a good one) with ifi power wart. Wired directly to a SOtM network isolator like the Eno wired to the streamer. In this case my streamer is the SOtM SMS200 ultra neo.
Actually, everything sounds great. I use Qobuz > HQ Player > Streamer > Chord Hugo TT2. So, it should sound good.
You didn't mention your DAC but that's certainly a key player.
I haven't any experience with the Zen Mini but I have read criticisms of it online. Folks that improved that one item and said it made a big difference. Does it? I don't really know.
I've read pros and cons to using a fiber connection, too. Have you tried eliminating that one part and seeing if it was a good addition or not?