You could benefit from dedicated server vs stock mini, but with no rendering things aren't as bad as they could be. Stock mini is noisy environment for a streaming component so there are some losses you'll never recover further down the chain, this even though doing no rendering. Certainly, there are nice benefits with a quieter network, but optimizing rendering is bigger bang for buck. You could continue with a streamer/dac combo or separate the two, either way has potential to be greatest improvement over what you have now.
Personally, I think you're on right track not doing any rendering within server, continue this into future. Still, after upgrade of dac or streamer this may become weak link in chain. Your mini is noisy, could be bested with dedicated server, but cost/benefit ratio is relatively low since you're not doing rendering. If I was in your position at this point, I'd think about bypassing switch with server with two network ports, second network port goes directly to streamer or streamer dac. Benefits would be twofold, simpler more direct network path best for eliminating noise, dedicated server with lps quieter environment vs mini. As I mentioned above, any dedicated server WITHOUT TWO ethernet ports would provide much less benefit.
The Roon Nucleus wouldn't provide enough benefit being designed as rendering server, and would need lps for best performance. Of the SMG, only the SMG I9 optical would be substantial upgrade, and here paying for capabilities you don't appear to need. Custom built or Antipodes K series server way to go down the road, server designed to be partnered with streamer is what you're looking for.