I don't like the idea of using Windex. Most glass cleaners are a combination of water, isopropyl alcohol, ammonia, and lemon extract. Assuming the water doesn't damage anything, ammonia and lemon will have serious consequenses long term. The cardinal rules of restoration are to do no harm, and make sure all work is reversable. I strongly suspect that 10-30 years down the line, these album covers will be noticably browner, and more brittle. The arts supply store cleaning pad is suitable for archival purposes. If your only concern is making things look better temorarily so you can off the item on eBay, then maybe you also want to try Armor-all on your vinyl. It will make it so pretty!
The Cheapskate