There was a reason CD was decided to be sampled at 44.1kHz. Many many many experiments were done and it was observed that 99.9% of the people could not hear any difference above that sampling rate (the other 0.1% descended from dogs).
Raising the sampling frequency will only push up the nasties from the filters to higher frequencies. So, up sampling does not improve the sound, it simply pushes up the filter artifacts to higher frequencies, hence giving the impression that the sound is ’cleaner’.
However, if very good designed and cleaner filters (more expensive to implement), are used, the 44.1 khz will sound almost exactly as good as your 23456234098634256 Mhz upsampled $34,000,000 music gear :-)
That is why most of the time, as well as is the case with you, you will not notice that much improvement with much higher up sampled music.
Buying a good quality CD player with a very good filter stage at its DAC will sound just as good.
Here is a good educational video: