To me, it’s a great example of how different music moves different people. I had a hard time even getting through it, with his horrible guitar strumming and the lyrics that sounded "poetic" but didn’t really seem to make sense.
I prefer "the Old Laughing Lady" off that album. I find it deeply evocative and emotionally engaging but don’t ask me what it’s about! The chord changes, melodic contour, use of space and resonance of the open tuning combine to shape my internal state in a pleasing manner. Dylan’s "Visions of Johanna" has a similar effect. Having a clear sense of narrative somehow isn’t of critical important for me in such songs.
However, if the formal elements are themselves off-putting, then it’s game over from the start. You mentioned "horrible guitar strumming". In my case, it’s the classically trained voice. You want to torture me? Lock me in a room and force me to listen to opera! As to why we have these preferences, who knows? Music affects us many different levels -- not all of them conscious-- and we can have very (!) strong likes and dislikes. We may try to justify our preferences with quasi-objective rationales but I can’t help but suspect we don’t fully understand what’s going on beneath the surface.