Once I put in bass traps and took care of direct reflections from walls and ceiling, I added diffusers towards the back of my listening room, which, given the other improvements I’d already enjoyed, seemed remarkable.
The main thing here, if you’re not doing some kind of professional room sound analysis anyway, is to try to fix the biggest problems first, and only then go after those smaller issue that remain. I don’t think one needs to spend a fortune.