Does the volume control effect sound quality?

Yes the can, so you may not be imagining things.

The volume control has a big effect on SQ.

Volume pots are very critical, with regards to the performance of a preamp, or any other sound-reproductive device. Alps, TKD and Noble make some highly regarded ones: ( http://www.partsconnexion.com/controls_pot_alps.html ) See page 14 of: ( http://percyaudio.com/Catalog.pdf )

Big time . There's a reason high end manufacturers use stepped attenuater's .

Sowter attenuater transformer volume control. From their website:
"The transformer action of these attenuators assures the loading circuit sees a low driving impedance regardless of switch position. This ensures the maximum bandwidth potential and minimum distortion of the circuit can be realized at all gain settings."