I had a similar issue with Tidal
Turned out to be that even though I was paying AT&T for the fastest internet possible in my area, they had "forgot" they needed to upgrade my router and switch.
Had been quite happy to take the money though......
Once they actually came out and "fixed" their issue Tidal HiFi has been just fine.
What internet speed are you getting on your server?