iMac in a home Hi-Fi system.

In the setting of the audio input and output: how to select sampling rate and quantization?
44100, 48000, 96000 Hz? and 16, 20, 24 bit?

Yes Vlady, what Meiwan advised is correct. In essence from your appleloss source which is digital, you need to have it converted by a Dac to analog. Fom there you can plug it into your analog system pretty much normally. But as to your second question about upsampling.

Vlady:> How to select sampling rate and quantization?
44100, 48000, 96000 Hz? and 16, 20, 24 bit?

You definitely want 24-bit, so you will be ready when your source is also 24-bit. There is much controversy as to whether or not higher up sampling rates will be audible on your system.

See: High Definition Audio (HD)

There are many good USB DACs out there as well. But do your reading, as many DACs have poor USB implementation. However, good USB implementation has been found by many to be preferable to Toslink with many DACs.

You should also download a "more audiophile" music playback program than iTunes, although many of them still use iTunes as a front-end interface. This is not an exhaustive review, but Bitstream is the cheapest, and many people like it. You can download it for $10 from the App Store. I use Pure Music on my home computer and Audirvana on my laptop. They each have their pros and cons, but I think I like Audivarna slightly better. Decibel gets some good reviews, and Amarra is considered by many to be the best (or at least the most expensive).

When ripping discs with iTunes, make sure you have the error correction on. Some people prefer using other programs, like Max or XLD to rip discs. I think iTunes works fine.

Sumflow: You definitely want 24-bit, so you will be ready when your source is also 24-bit. There is much controversy as to whether or not higher up sampling rates will be audible on your system.

Yes, I would like to know this. Many say that the sampling rate and quantization should be the same as in the player.

I bought my first Apple desktop, the IMac and while I enjoy the SQ from the CD driver with or without my Audio Engine D1 DAC with my Shure SHR840, I am limited to the only Internet Music, Spotify.

I cannot play Pandora Premuium, MOG, Slacker on my IMac. It's frustrating since the HP desktop and the Sony laptop at much lower cost can play any of those formats.

Anyone has the same issue?