I have had the same experience. I think there is a fine line between politely asking the offending conversationalist to keep quiet and letting it be. If you think they would appreciate it, perhaps try the former, if you think you will offend, perhaps lean to the latter.
I was having a full-on listening session with two friends, and one was glued to his phone the entire time. While he was not making any noise, he certainly was not focusing on the music, nuances or otherwise, and I knew he was not getting out of the experience what my other friend and I were. I suggested that he put the phone in the other room, sit back, close the eyes, and get lost in the music. He protested at first, but he did this, and enjoyed it.
I often explain my hi-fi kit to the uninitiated as more than a fancy expensive stereo, but actually as high resolution equipment to recreate the reality of the recording, live or studio, to put the listener right there, in that room, to connect our ears to the musician's song. This is obvious to anyone on audiogon, but explaining it this way kind of sets the scene that this is not the same as the ceiling speakers in the rest of the house that we can chat and play cards to.