When do records damage a stylus

I hope this is not a dumb question. How bad does a record have to be before it damages your stylus? I have a bunch of old records. Most of them are in very good shape. Some have ticks and pops even after I clean. Some have some scratches that don't make the music skip but you can here a pop, pop, pop when the stylus hits there until it gets past the scratch. Everybody talks about hear some pops but how bad before you do damage to the stylus? I can't afford to replace all of my records but I can't afford to replace my stylus either.