Whether there's an audible difference will of course be system-dependent. FWIW, it's clearly audible in mine. Your system isn't listed so I couldn't guess whether it would be audible in yours.
As to which is preferable, that's a matter of taste.
In my system, compared to using no washers at all:
- nylon washers cause an audible slewing of rise times (slowed responsiveness) and a diminishment of amplitudes (reduced dynamics). IOW, they smooth and soften the sound. This is a typical effect of applying "constrained layer dampening" close to the signal path.
- brass washers have an effect similar to nylon, though somewhat less pronounced
- stainless steel washers have virtually no audible effects
Since my partner and I prefer the most faithful reproduction of the source that we can manage, I use SS washers. I torque the screws quite tight and have never had an issue with them loosening. (Note: whether this is safe varies with the design of the cartridge body, some can be deformed or damaged by over-tightening of the mounting screws.)