Everything will make something of a difference. Not only size of the rug, but its thickness, material it's made of, whether you have a rug pad underneath, and position in the room vis-a-vis speakers, walls and listener.
This opens up a whole new avenue of audio/home decor experimentation, sure to have a high WAF.