Academy has their jazz, pop, folk etc (so not classical) at their other location on 10th between 4rd and lafayette (3rd ave), and they've got a good selection and you can listen before you buy.
Second Hand Rose is on 12th between University and Broadway, but I find them a tad pricey sometimes.
Footlight records is great for folk, they're on 12th between 3rd and 4th ave.
Jazz Record Center is in chelsea on sixth avenue, and available on the web (and in previous posts here).
Otherwise, google could probably turn up a slew.
have fun!
