There are some good points (& good humor) above. Music is like any other 'tradition'--film, literature, etc.--in that the broader your familiarity with the tradition, the more you'll appreciate individual contributions to it. Or to take Siliab/Photon46's language metaphor, your familiarity with a particular musical idiom or set of idioms will to a large extent govern your ability to understand what the music is doing. Same concept applies to a particular style, conductor, piece, whatever. I often find that the more difficult stuff grows on me, while music that is more easily accessible varies in the persistence of its appeal. In most cases, the latter-mentioned variability probably relates to greater formal complexity behind the beauty or tunefulness that first gets me (Mozart comes to mind here).
The suggestion to read some books is also a good one. I recently read Aaron Copland's _What to Listen for in Music_. I don't know what other people think about it, but for a guy like me, whose formal musical training ended with the fourth grade tonette choir, it could be a helpful starting place for a better understanding of music. Not that you really need to do anything but listen, though. :-)