I would definitely recommend a trip to the WFMU show this weekend (starting tonight, actually, for the $20 admission--tomorrow and Sunday are only $5). Don't expect to find many great bargains there, but do expect to find a lot of rare LPs priced reasonably (usually). I think I spent $250-$300 in about two hours last year...
The "little shop" in New Paltz is, I assume, Rhino Records. It recently moved from its former location on Main Street to a smaller space off Main Street further downtown. I haven't been to the new one yet, but have been told that they sold off most of their LPs before moving and don't have as many in stock at the new shop. I think this shop tended to be overpriced anyway, though they did have a lot of old jazz & classical LPs at the old shop, with very little turnover. Maybe some of these made the move? The rock/etc. sections were only occasionally interesting.
If you're willing to drive all the way up to Red Hook (on the opposite side of the river from W. Point), there's a small but very well-curated shop called "Left of the Dial Records" there, in the back of a collectible card shop. It has no sign and no phone number, but is open from 2-7PM every day, I believe. Very good prices considering the rarity of some of what he sells. Right at the main intersection in downtown Red Hook, on the west side of Rt 9, just north of the traffic light. Look for the card shop in a row of brick buildings there.
There's also supposed to be a place up in Hudson with 1000s of used LPs, but I haven't been up to check that one out yet. A lot of the used bookstores up and down the river have some LPs too.
There's also Last Vestige in Albany, which I've been to only once, but made out pretty well at.
Hope this helps,
Joshua