Yes, dusty, worn POTS and oxidized contacts on switches can be a problem. But that is why we have DeOxit ;-)DeOxit has a dicey reputation in this regard. Its not exactly the best medicine if you are dealing with a part that has the resistive element on a phenelic substrate. In a nutshell its bad chemistry.
So people that are really concerned about longevity of such parts (usually collectors) use specialized vacuum cleaners to clean out the pots and see how they behave after that is done before resorting to control cleaners like DeOxit.
Us designer types that like to make high performance circuits have different concerns. For example, I like tubes for preamp circuits and tubes tend to have some fairly high impedances that cause stray capacitances to roll off the top end. If you really want your circuit to have 3D depth and width, you have to eliminate phase shift; to do that you need bandwidth (2KHz-100KHz at a minimum; 200KHz is better). When you have a switch to bypass a tone stack you're asking for trouble in this regard. Usually the presence of a tone stack suggests two things- first that the speaker is bandwidth compromised and second that getting the sound stage right is secondary to the goals of the listener.
In high end audio, getting the system to sound like the musicians are playing in the room is a pretty serious endeavor. It is rendered impossible if a tone stack is involved. You might be able to get it sound like a good stereo, but sound like its real?- nope, won't do that.
Once I sorted this out and got competent equipment, I found that I never felt like I needed any tone controls. The key is 'competent equipment' but its also important to get the speaker placement right and have a good match between all the bits in the system.