Get a different bearing and replace the rubber belt with a thread and those issues will go away. It's not the motor torque it's the compliance of the belt and the relatively high friction bearing in your deck. Idlers and directs have their own challenges. Good examples of each sound more similar than different.
Cheers
Cheers