My assumption is that with any of these formats, the bits being delivered are the same and timing is the real issue. I agree that in the past it was likely that a lot of activity on the PC could affect the sound, mostly because of timing (jitter) issues. But with the use of async USB the activity on the PC seems to be a real secondary issue. As long as the buffer is full and the activity on the PC is relatively low, it seems that the aysnc USB is the determining factor. I use a PC with a very low end processor, and running FLAC through J River, the CPU hardly goes about 5%. If the FLAC decompression is a problem on a system, it is probably because there are too many other things running. Although, as I said, using an async USB converter/DAC should minimize that effect. There can still be noise issues (like small ground issues) but that should not be affected by the minor differences in processing time to decompress files. I am of the mind that "everything matters", but with properly done async converters/DACs, it seems that the other effects can be extremely small. That said, there are always people who say they hear differences in everything. I would suggest converting a few of your wav files to FLAC and listening. dBpoweramp is a good converter and it has a free version.
What hardware/software are you using?