2ch amp and HT receiver connected to same speakers

I listen to 95% music and 5% home theater. My 2ch system is a BAT pre and a Bryston amp, and I have an Integra receiver for theater. My BAT has a HT pass-through function, so that I can run my LR speakers off my Bryston amp. This setup causes a ground loop, however, and I've given up trying to eliminate it. So I wonder if I can connect both my 2ch amp and the LR outputs of my receiver to my speakers. Would this harm my amp (especially the Bryston)? Would it hurt the sound quality of my music? Thanks for any comments.
02-03-11: Jamesg
Install banana plugs on your cables for a quick disconnect when you switch between systems. Never run with both connected at the same time.

This appear to be the best option of not destroying anything albeit a little inconvenient especially if one often switches between music and HT frequently. I am doing the same thing in my system.

I am not too convinced on the Niles switch box as I believe it may degrade sound quality with an extra box inserted into the chain. Bill is on the right track in contacting both Niles and Adcom as some amp manufacturers may not recommend doing this out of technical reasons(not associated with sound quality).

I'm still thinking to solve the ground loop. I've helped lots of buddies hook up using HT bypass, or just taking front preouts to an input on the preamp, then always setting it's volume control to the same position. It's hard for me to believe the ground loop cannot be solved.
regarding ground loop, have you tried an isolation xfrmr on your CATV (assuming that w HT bypass, you have a display that may be tied to the cable or sat co-ax)?
Good answer Swampwalker, this is the problem many times. Of course, the simple test is to have everything hooked up using the HT bypass, but with the incoming CATV from the wall disconnected from the system. If the hum is gone, an isolation transformer will solve the problem. A simple, inexpensive cure to the problem.