Just because something is extremely hard doesn't mean that a softer substance cant abrade it. When playing a record you have a single tiny piece of diamond traveling quite a distance just to get through one side of an LP, without doing the math I bet an LP may be at least a mile or better long. Add a little dirt in the groove and some non vinyl particles in the vinyl and it will very slowly lap the contact portions of the diamond.
On a microscopic level there may be all sorts of stuff in that vinyl from the manufacture of it. One of the last steps of raw vinyl production is to extrude it like spaghetti and chop it into small pellets. That is the form of raw vinyl that companies purchase. It gets unloaded into silos at the manufacturing plants and long screws pull it up from the bottom of the silo and feed it into heated extruders that feed the presses. All of the contact surfaces slowly wear and shed microscopic metal particles.
No where is this more evident than in a big commercial bakery as flour can be very abrasive. Mixing paddles and feed screws, all made from very tough stainless steel worn down till they are no longer effective. All that metal winds up in the food we consume but the levels are extremely small so it isn't a concern. I was shocked the first time I went into my favorite baker that makes snack foods. After talking to the machinist and realizing how slowly they do wear it didn't bother me.
I think the main culprit though is just dirt when it comes to stylus wear and the thousands of miles it gets dragged through the groove.
By the way, how do you machine a diamond? With other diamonds of course. Crushed diamond is bonded together and formed into grinding wheels. The wheels are contoured with a diamond tipped dresser and used to grind your stylus to shape.
BillWojo