iMac in a home Hi-Fi system.

Apple lossless or Aiff are the best quality. Either one is fine.

You will need an external DAC using toslink(optical) via the headphone out jack on the mac. Then a set of RCA's from the DAC to any free inputs on the pre-amp.

Under 'sounds' select the external dac for output.

If you have a Cd player with 'optical digital in' you can use that in place of the DAC. Either way you will need a toslink cable - toslink to toslink rca mini.

In the setting of the audio input and output: how to select sampling rate and quantization?
44100, 48000, 96000 Hz? and 16, 20, 24 bit?

Yes Vlady, what Meiwan advised is correct. In essence from your appleloss source which is digital, you need to have it converted by a Dac to analog. Fom there you can plug it into your analog system pretty much normally. But as to your second question about upsampling.

Vlady:> How to select sampling rate and quantization?
44100, 48000, 96000 Hz? and 16, 20, 24 bit?

You definitely want 24-bit, so you will be ready when your source is also 24-bit. There is much controversy as to whether or not higher up sampling rates will be audible on your system.

See: High Definition Audio (HD)