paranoid listener, bad recordings

I tend to listen apprehensively for some reason not understanding how durable speakers are at my listening level. So, I hear distortion from time to time coming out of my speakers and freak out thinking i have damaged them. But, I am beginning to think it is on the recordings.

Right now i have a pair of gallo 3.1's in my den that i am wrestling with room position (see other post) because of limited options. I put on some joni mitchell or typically pop recordings, sometimes older jazz, and will hear the distortion. I listen around 75-85db levels.

Am I paranoid, is the distortion in the recordings and should it sound like this on occasion? Is that too loud?
Are you playing CD or LP as source material?

If LP, the record may be damaged from either prior use, dirt or it may even be a defective pressing. Tone arm and cartridge setup can be an issue and so on. There are a lot of things to check.

CDs can also be poorly recorded or mastered.

Assuming there is not something wrong or broken in your system, your listening volume should not be a cause of concern for damage to the speakers.
There are some good suggestions above, however it is also possible, even likely, that the distortion is on your recordings, at least some of them. Recording engineers are by no means perfect, and many times the miking/mixing is less than desirable, even on normally very good labels, no matter what the format.
If you can, play the offending passages through decent headphones at low-moderate volume. If the distortion is in the recording you'll hear it.
Every time I play some CDs that I own I think; "Hey what happened to my stereo?".
I like the music but they sound pretty bad, some Bruce Springsteen comes to mind.
Then I put on something that sounds good and I get that "aah..." feeling.
It's funny though, I always forget for a moment and get panicky.
I'm not dismissing some of the other explanations but this could be part of it.
If you are listening to currently produced CD's, they as well as the MP3 files are victims of the loudness wars going on, and are often very distorted indeed.