Vinyl***What If***

What's "inexpensive"?  What's your budget?  Are you open to used? 

It sounds like the Cayin only does MM cartridges?  If so, the suggestion for a Hana SL (and many other MC cartridges) is not a good one.  You could get by with the Hana SH (the high output version) but might want to stick to MM.  There are some very good ones out there.

I would not buy a thing at this point if you are on the fence -- do you have a dealer who would be willing to loan you a better than entry level set up for a week or so? And where are you going to get records to listen to? Buy them and get rid of them if you decide it isn't for you? I assume you have some friends who'd be willing to provide some records for you to play temporarily....
I'm not trying to be negative about this-- I've been playing LPs all my life (well, since I was a child) and am fairly well 'invested' in the medium. Ultimately, that means money in records. Not just gear. Part of this is meant to be fun- looking for better sounding pressings, finding stuff in bins, not just hitting "BUY" on the latest remaster. 
I'd also be concerned that the total cheapy stuff probably isn't going to give you any sense of the potential of the medium. Nor will some records, no matter how how heavy the vinyl weight or fresh off the press remaster. (Nothing against re-do's, some actually do sound better than early pressings). 
In other words, I'm not sure you can get a real sense of what the potential is by dipping your toe in tentatively. So, see if you can get a dealer to loan you something decent and set it up for you and get a few good records to try. In the long run, it may be cheaper to pay for a dealer's time to do this than for you to buy something mediocre, declare every one who is into records delusional and then try to sell what you bought. 

What whart said, +1. Are you anywhere around Indiana? I’d be up for bringing my older/retired vinyl rig(and a few records) over, for a demonstration.

It seems you're under the illusion that there is something special about vinyl; that only happens if you're committed to putting in the money and effort to getting the best out of an analog rig.

Plug and play is a waste of money; when it comes to vinyl "Run with the big dogs or stay at home".

Bill is right, I think. Also, I don't quite understand your reasons for trying vinyl. If you seek the highest sound quality, I would suggest improving the rest of the system and then add analog source maybe.
No rush, records will still be here. Speaking of records' cost, in my case it varies from $1 to $150 plus shipping. And yeah not playing records at all is better than playing dirty records.