While the TV would need to be addressed separately, I'd focus, to start with, on just the front wall alone and go ONLY with some form of quadratic diffusion. Vertically oriented at that. That's really important because that is what is going to help bolster and clarify the center image in particular. On their own, your dipole speakers will throw most of their energy into the area more or less directly behind each speaker, leaving the center image unreinforced (...unless you were to penalize the stage width by moving the speakers closer together, of course).
But vertically arranged quad diffusors will 'fan out' the rear projection horizontally (as horizontal diffusors spread the reflected sound vertically, it just depends on which problem you're trying to solve).
That 'flashlight beam' pattern the speakers inherently have (as most dipoles do), will be spread out more evenly across the entire width of the stage, including both the center image and also even the rear corners of the stage, while 'reducing' the buildup directly behind the speakers at the same time. Everything should even out considerably more.
Just don't settle for 'randomized' diffusors that spread equally in all directions or you'll lose out on the opportunity to 'steer' the sound where it will do the most good.
After that project is successful, then you can turn your attention to the side walls...if needed at all, or regardless of what type of solution that might require on its own.