First of all, Ginger’s drums and cymbals sound terrible---terrible! The drums are thin and ringy, no body or tone, like an unplugged Telecaster. The cymbals are dissonant (the overtones being out-of-tune with the fundamental) and clangy, without the percussive "click" of good cymbals, or their melodic pitch and tone. Listen to the cymbals of Jim Gordon (Derek & The Dominoes, Joe Cocker, Delaney & Bonnie, Traffic) and many Jazz drummers (as well as Don Lamonds in the aforementioned "Beyond The Sea"), then Gingers. The man had no taste! Likewise, his drums just don’t sound good. Just as with his cymbals, their overtones are out-of-tune with the fundamentals, creating dissonance. Ugly and unpleasant. Some drummers never learn how to tune drums. He also used really thin heads on his drums. Ask any guitarist about what super-thin strings sound like---no body, no substance. To hear what good drums and cymbals sound like, listen to Levon Helm’s on the Band’s albums, particulary the 1st and 2nd. Listen to the opening of "The Weight", the three simple quarter notes Levon plays on his mounted and floor toms. Fantastic! Then listen to his cymbals---beautiful musical instruments, very much like Jim Gordon’s.
As for the substance of Ginger’s "Toad" solo, when you take away all the repetitions of patterns and figures that he plays, you have only a couple minutes of actual ideas. He just repeats them over and over and over again in a row, making the solo sooo repetitious and (imo) boring. And those ideas themselves are just not interesting, at least not to me. They sound awkward and clumsy, without quality flow, structure, and development. Each to his own!
There is a story told of when Miles Davis went to a club to watch and hear The Buddy Rich Big Band. Miles expressed his surprise at discovering that Buddy wasn't just displaying his astounding chops, but that he was playing parts that set up the song for what the other players were about to play. That's called ensemble playing, and that's the way the best musicians play. Ginger Baker was not one of the best, whether he was soloing or playing a song. It doesn't bother me that others like his playing, I don't know why it should bother anyone else that I don't!
Regarding being a "professional musician and all", that does not make an opinion any more valid than that of anyone else, including non-musicians, professional or otherwise. I have known non-musicians with far better taste (imo) than many musicians I’ve known. And taste is what we’re talking about here. That, and the lack of it ;-).