From a hardware perspective:
I'd start with upping the RAM to as large as possible. With the 2014 Minis, this means getting the factory upgrade as the RAM is no longer user replaceable. Not an issue with the 2012 and earlier models.
I'd go with an SSD option (not spinning HDDs or hybrid/Fusion drives).
You can opt for a linear PSU transplant later on - the switch mode PSU inside is replaced with a cable or DC filter board and you'd need an external PSU.
Unlike Steve, I don't find the older Minis a good match - I've found in my PC builds that a faster i7 QuadCore sounded better than my Atom based servers so I'd say go for the fastest CPU you can get.
From a software perspective - it USED to be difficult. You needed to use sudo/terminal commands to disable a lot of the background processes.
But here's the good news - Amarra now includes a Wizard based optimiser that does this automatically for you. It disables Spotlight, enables Finder Quit, Spaces etc. The same Wizard allows you to re-enable the services if you wish.
Audirvana Plus also provides something similar but I think it is not as extensive as Amarra's. On the other hand, it is a lot simpler to use (just a check box and a restart).
=============
USB-SPDIF devices
I know the Offramp is highly regarded but up till the previous model, it still used an M2Tech USB converter IIRC which had two major downsides.
a. it wasn't USB Audio Class 2.0 compliant which meant it needed custom drivers from M2Tech for OS X. OS X from 10.6.8 and up had not needed drivers for USB Audio Class 2.0 drivers - they just worked when you plugged it in.
Mavericks (the previous version of OS X) broke compatibility with M2Tech's drivers for 10.8). And it took over 6 months for a new version to arrive. I'd just go for an Audio Class 2.0 compliant converter for the hassle-free experience. I don't know if M2Tech's driver works with Yosemite but I know Mytek (which also needed its own driver for Mavericks) doesn't work in Yosemite now either.
b. A consequence of USB Audio Class 2.0 compliance is that most apps like Audirvana Plus, BitPerfect and Pure Music can enable Integer modes which IMHO brings the performance up a fair notch.
If you have a Bryston BUC-1, I'd keep it. It is quite a well regarded converter as well.