For some people, like Jjmali mentioned, hunting for vinyl is half of the hobby. It's the thrill of finding something great, or even just the prospect of finding something great. And when you do, the payoff means getting to listen to some of your favorite music, which is a real kick!
For others, like Taters, hunting for vinyl is a hassle. There's a lot of effort involved with inspection and seeking good used sources, and even then you will end up with some records that will sound like they were played with a dull thumbtack attached to a stick.
So it really depends on how you approach the hobby. Personally, I'm in the "thrill of the hunt" camp, and really enjoy the feeling of coming across a mint copy of something I was looking for (or something I didn't know I was looking for), even if I end up buying 75% duds to get there. I know that if I absolutely NEED a vinyl copy of something, I can afford to pay $20-$60 here and there as a special treat, but you will be very surprised at what kind of gems you'll uncover in used bins.
I'm with JohnnyB53 on the Technics SL-1200 turntable. You will find that the main opposition you will run into on this forum regarding the Technics turntable is that it is Direct Drive as opposed to Belt Drive, but you can do a search in the forums to find both sides of that argument thoroughly expounded upon. But if you do decided to pursue this route, I would also recommend finding the website of KAB USA and investing in some of their upgrades for the SL-1200 (or just buy a pre-upgraded table from them). This will put a turntable on your rack for under $700 and leave PLENTY of money left over for a cartridge (I say get the best Dynavector your budget allows) and vinyl hunting money.
As a personal aside and necessary disclosure, I started with a Linn turntable at the recommendation of posters who said that a belt drive was my only option as a budding audiophile, and since I've made the switch to the Technics SL-1200 I am playing a lot more records. I'm just one set of ears though, so if you get a chance to do some of your own comparisons then I would highly, highly recommend it.
It's always more fun to TRY something than to NOT TRY something, and the resell value on audio equipment is very decent so if you can afford to try it, then go for it!
Good luck with whatever you decide,
Dusty