I have no problem with the word "Soul" as pertaining to music. Just like any other word, the meaning depends on how it's used.
Sam and Dave "I'm a Soul Man";
https://www.youtube.com/watch?v=ZVx2i6jGzf8
That pertained to an altogether different genre of music other than jazz, it was quite specific, and it was called "Soul music".
"Soul" in jazz is in reference to deep emotion that is generated by the music;
https://www.youtube.com/watch?v=VhgUUe5czxc
"Sandra's Blues"
Dave Brubeck's music falls into neither category; it has more of an intellectual appeal, but if you find it "soulful", you're one of a kind, and I will never dispute your perception.