Am I out of my mind? #obsessed

Most likely anything you do to fix micro scratches is going to make the problem worse. The best coarse of action is limit the cleaning to twice a year and invest in protective, non abrasive covers. Then leave the speaker covered when not in use to prevent dust build up. I have a piano gloss wood finished speaker that I have owned now for over 20 years and they still look factory fresh. Although, if you look super close, a micro scratch might rear its ugly head but I don't worry about it.

Guitar makers and repair guys often use Novus as I do. The 3 versions go from serious scratch therapy to a great sort of anti swirl mark polish.

For years I have cleaned piano gloss speakers using the following method with perfect results:
1) Lightly dust the speakers with a Swiffer Duster.  This will remove the loose dust particles without abrading the surface.2) Lightly mist the speaker with any high-quality auto detail spray.  I had good luck with Mothers.3) Spray the Mother detail spray onto your NEW microfiber cloth, folded into quarters.  Gently wipe the finish until you see the detail spray start to disappear.  Flip over the cloth for a final buff and you're all set.4)  As long as the speakers are not touched, usually you can get excellent results from just going back to Step 1, above, and not having to use the detail spray steps.
To fix existing surface scratches on used speakers I purchased, I have used Turtle Wax Jet Black Car Polish  and plenty of elbow grease with good results. 

Hello,
I owned an antique Ferrari and a modern Mercedes E350 black in color. I was obsessed with the finish. Probably spent 2-3 hours per section or panel of the car. When I was done like MC said you can’t focus on it. It looked like it was a liquid finish. Its true glazes and waxes are finish coats. It is better to get the clear coat perfect and use the sealers, glazEs, and waxes to add depth, not to hide imperfections. My REL subs have wax on the finish from the factory. When using these polishes you need lubricant like water or car liquid detailer. Your speakers might have wax or cleaner on them so do clean them off so you don’t embed the dirt into the finish. When buffing the idea is to keep buffing until the product is completely broken down while you keep lubricating it. Use the highest # of grit or finish polish since you are trying to remove micro scratches or spiderwebs. To clean spray a good microfiber towel,$10-$15 with water until it covers most of it without dripping. Next grab two corners closest to you and drag the cloth across with no weight applied to the towel except surface tension and the weight of the towel itself. The liquid acts like a lubricant while the towel grabs the dust. If you apply pressure you can scratch the finish from the towel or the dust on the speaker. If you can slowly lift the towel off or higher as you slide across. This way the dirt you pick up doesn’t touch the finish. Most important. “If the towel hits the floor it’s done.” You can use it for dusting your house or cleaning your windows. 

It's best to avoid the scratches rather than try to remove them. Remove dust with compressed air, not a cloth. If you don't have a small compressor, compressed air is available in aerosol cans for very little money. You can get the cans at computer and electronics stores, or on Amazon.