I'd go with a Mac Mini. Matter of fact, I did.
You could go with a Windows-based platform, but I think you'd spend a lot of time trying to squeeze performance out of it, and to figure out what went wrong. A dedicated Mac is just much easier and less prone to issues, IME.
Get a Mac Mini, set it up with an old monitor, keyboard, and mouse. Then put it on your rack to run headless, controlling from an iOS device such as your phone or iPad using Apple Remote, or from any other Mac in the house via Share Screen. (You'll need WiFi, of course.) Lots of info on the internet about how to do this, and optimization is easy.
For the Mac Mini, get one with an internal SSD drive to reduce mechanical noise. Use the SSD only for critical software and OS X. Then get an external HDD hooked up via Firewire 800 or better, or if you're feeling more ambitious, an NAS. Benjamin at Mojo Audio does a good job of using an AV drive in an OWC case with modifications.
After you've established your library -- in iTunes, most likely -- you'll want to use iTunes only for library management, as it's less than optimal for sound. I use PureMusic and love it; Channel D also has Pure Vinyl which includes PureMusic but is more geared toward vinyl reproduction. You could also use Amarra or Audirvana, but the thing to remember is that you want software to pull the files out of iTunes and handle them properly. BitPerfect is a clean and inexpensive option as well.
All in all, this is all about setting up a dedicated system to do one or two things well, and then tuning it to just do the basics. It's easy; it's a one-time thing, and the results are fantastic when done correctly. And you won't have to get up and down for each album side!
PM if you need more info.