Diffusion is best when you're farther away as the waves need proper time and distance to spread out. When you sit very close to a boundary, the biggest issue is actually bass buildup which requires absorption.
And I fully agree with that.
What I meant was that diffusion is more essential for close reflections at either end (Close to speaker or close to the listener - if you are unlucky enough to have a terrible setup). Of course, diffusion is much more effective as you get further away from it.
I also agree that bass is the biggest issue when you sit close to a boundary but that does not mean that higher freqency reflection are not a problem too even if they are smaller).
Sorry if I wasn't clear.