Yeah, perhaps the DAC has more influence. Say 70% DAC to 30% transport? Hard to quantify, but as bigkidz says the transport can certainly be heard.
Compare the build quality of the $10 Sanyo assembly and the out of production $20 Philips to the $105 current production, audio CD only Teac. That Tascam 200 is probably a good, low cost option.
When you move up in price you usually will get better, more robust power supplies and filtering, vibration control and dampening, better clocks and regulation circuits, SPDIF isolation, etc. All things that will provide a more refined sound. Audiolab 6000CDT might be another lower cost option.