I doubt I'd hear any difference at all, and the convenience of having the remote would far outweigh any miniscule difference there could possibly be.
I've got a Herron Audio tube preamp, and I think at one point, Keith Herron did not offer remote, but there was enough demand for it that it now has it. For a great, great product, this is the cheapest remote I've ever seen in my life - all plastic, weighs almost nothing, and it's less than 4" long! Gets misplaced very easily, too! Maybe this was Keith saying 'OK, I'll give you a remote, but .... '