Is it worth buying a headphone amp?

You will probably get more informed opinion over at headfi.

Sell the NAD and buy a Portal Panache; great integrated with great headphone jack, no remote or features, but great sound for the money.

Thanks for the responses, folks. It seems like a headphone amp will benefit me greatly, but I'm still wondering if I should buy better headphones first. I did a little experiment this morning and plugged my Senns into my Macbook (I'm not a computer audio guy yet) with its output impedance around 30 Ohms, and despite the crappy sound card the sound was worlds better than it was through my NAD. Still not a great impedance match, but it was better.

Though the bass was diminished when plugged into the laptop (I think the impedance mismatch with the NAD emphasizes the mid-bass because of the Senns' high impedance spike around 100Hz), the mids and treble were clearer. Cymbals sounded like metal rather than papery electronic hash, though resolution still wasn't excellent. A brushed snare on Lucinda Williams' "Fruits of My Labor" sounded like vague background clatter (think paperclips being shaken in an empty OJ-concentrate can), while it resolves perfectly through my speakers. Not sure if that's a limitation of the headphones, though I do see that they're capable of better sound than I was getting.

The Portal Panache is new to me and looks like a great product. That might be a good future upgrade for me. For now, it's a little out of my reach. I've read a bit on Head-Fi but haven't joined yet, so I appreciate everyone's input here. Still don't know if I should put the Schiit or headphones on the list first (ideally I'll eventually do both)--but I'm open to further suggestions! Admittedly I'm still not a huge headphone fan, as listening fatigue sets in pretty fast for me even at low volumes, but they're good for keeping peace with the neighbors.

I just upgraded from a Maverick Audio D1 DAC/Headphone amp driving Sennheiser HD 545's to a Peachtree Audio MusicBox driving Sennheiser HD 598's.My headphone usage has gone from must use so as to not disturb anyone during the wee hours of the mourning to can't wait to throw them on & rock out so I would answer a resounding YES to your question!
In addition to head-fi.org check out headphoneinfo.com,headfonia.com,onheadphones.com & headphonetalks.com...

Well, folks, new headphones it is. The HD555 are excellent for the money, but to start with I think I'm to going sell them and look for a used pair of AKG K702. I'll buy an amp later, when my budget allows. The Schiit Asgard is still at the top of my list, though I am concerned by the 6 Moons review, which characterizes it as being a bit lean--and lean is the opposite of what I want in a headphone amp. Any other suggestions around $250? Thanks for the input, all.