In your case, by running from tape-out you're almost certainly creating an impedance mismatch that is rolling off your bass response big-time, and in general wreaking havoc on the net frequency response. The effect is exacerbated with headphones of lower impedance. The sound can be perceived as detailed or even "airy" this way, but it is not accurate. I've heard that result a few times, with a few different causes. If you're like me you'll tire of it and start craving a firm bass anchor to your music.
A properly engineered headphone amp will have a low enough output impedance to match with (most) headphones, and should thus yield a frequency response in keeping with the designer's intentions. Even those headphone amps with a higher output impedance (OTL tube amps) are usually voiced to end up sounding fantastic with certain headphones (e.g. many of those Singlepower amps and 300-ohm Sennheisers, or really almost anything before driven to clipping-- the Singlepowers should't sound nearly as good as they do, but they do).
In other cases, where you may have a headphone jack provided on say a $4K integrated, then you have to consider what's actually behind the headphone jack. What makes for a great line stage circuit is usually going to have too high an output impedance for most headphones...and rather than compromise the main circuit, it's common to see a crappy little opamp based headphone amp behind that jack. Even gear that boasts "discrete" headphone circuitry usually won't invest the right real estate, parts, and engineering to get it right (when headphones are an afterthought for most buyers of high-end speaker gear). Hence, a properly engineered $500 headphone amp may yield a much more pleasing result.
Now that's not to say that even most $600 headphone amps will sound particularly good, but it should be possible to find one (especially in the used market) for that price that will sound great. I think I remember buying an old used Headamp Gilmore Reference for like $650? That thing was awesome. And an old maxed-out PPA set me back a whopping $200 and can make other amps in the $1000 new range sound like garbage by comparison. If you're a gambling man, Singlepowers (with a rightfully deserved bad reputation thanks to their creator and their potential hazards) represent some of the most pleasing sounding amps around at any price (though a few do sound not so great...lots of unit-to-unit variation). That said, I'm not familiar with the newer crop of headphone amps from the last several years.
