For jazz, the best store in NYC is by far the Jazz record Center, 236 W. 26th Street. It has vinyl, thousands of new and used CDs, and a huge selection of jazz DVDs. The owner, Fred Cohen, is extremely knowledgable. If you are looking for something and don't see it, just ask. Academy, on 18th Street, is also pretty good, and the two stores are within 10 minutes from each other. Have fun.
Carrard
P.S. The Jazz Record Center is on the 8th floor. You must ring the bell from the street to get in.