Headshell Washers : Nylon or Stainless Steel?

Few things are unimportant which are so close to the most critical interface in hifi (stylus >> groove)....

I've been using Nylon washers for nearly 18 months now, mainly for their protectiveness of the headshell finish. The nylon also performs 2 other potentially useful jobs : insulation (breaks possible GND loops - although I've never suffered one before) and constrained layer damping.

There's no denying that setting up a cart accurately is much easier with stainless steel (they don't change shape under compression and end up skewing the cartridge) although, as you know, if stainless steel is used we must be certain that it's 100% non-magnetic.
Another minor source of worry with washer choice are tales of cartridge bolts which have secretly loosened due to inadequate torque. (Although I'm certain the owners would notice this straight away...)

My main question is do the nylon washers have any ill-effects or disadvantages that you can think of?
(e.g could they compromise the rigidity of the "closed loop" - bearing in mind we are using them on top of the headshell not underneath?)
So what is the source of any sonic differences - the damping ability, or something else?

Personally I can't see them having any effect on the closed loop as the cart body is in intimate contact with the headshell and there is plenty of friction there(?)
Please discuss....