"I do notice better performance on the power amps plugged directly into the wall."You've probably just answered your own question. Maximum performance, in my opinion, trumps the minor equipment safety concerns.
"Or am I being too much of a worry wart."I'd say that it's self-defeating to purchase high end gear for big bucks and then compromise its performance with a surge protector or something. You MAY want to consider one of the supposedly "non-current limiting" big boys like Running Springs Audio or some of the others. I think those offer surge protection.
Also, the NBS website provides an interesting perspective on this issue (I don't sell NBS or work for them or any other power conditioner company):
What the people at NBS have to say