Regarding tone of voice--are you really sure you can read tone of voice accurately? Probably you can with people you know well, even on the phone. But with people you don't know, or especially with people whose native language is different from yours, I suspect things are quite otherwise.
I'll give you a few examples to explain what I mean. The first time I heard my own voice on a recording, I was shocked. It did NOT sound at all like it sounds to me, and therefore, it did (does!) not express the emotion I mean it to express when I speak. I've tried to learn to do this more effectively--I suppose actors must be good at it, but I'm a professor and need to communicate effectively every day, so I should be, too. Still, it's very disconcerting to hear my voice as I suppose others must hear it. As several philosophers have pointed out, we get our "selves" from others.
Here's another example. My wife is from Croatia, and we've spent a lot of time there. To my ear, Croatians always sound angry--and this carries over to a native Croatian speaker who speaks in a different language. I met my wife at a language school in Germany; even Germans, who speak a notoriously commanding language, tended to recoil from her when she was trying to sound friendly.
Conversely, I've spent time in Italy, although I don't speak Italian. Listening to Italians argue sounds to me like they're flirting--just as Croatians flirting sound to me like they're arguing.
Cultural conditioning again. There just aren't many "natural" cognitive judgments in anything, in my opinion--not in aesthetics, or politics, or even ethics (despite what we'd like to believe). What we see, hear and think is filtered through a complex system of neurons that are patterned in part by environments. What seems "natural" to us is highly structured, conditioned, mediated.