Dependent upon what platter you are using, one problem you might encounter, with a better clamp, or an aftermarket clamp, like a BDR (way over $100 anyway) is Dust Cover clearance.
A small inconvenience, but with a thicker platter, like a MK-IV/TNT Platter, and any of VPI's better clamps, the inside of dustcover will hit the clamp if you leave it on the spindle.
As far as effecting speed, no, there should not be a problem with any clamp you use, even if it was 5 lbs. The Hurst AC Motor is fairly robust.
The added wear on the bearing's thrust plate, and ball might be accelerated due to the added downforce weight of a substantially heavier clamp. Mark