What is the most challenging music to play on a stereo?

Skinny Puppy's entire catalog is a challenge for any system and every listener. Intentional distortion, symphonic synthesizers, plenty of fast low end bass, ominous synthesizers, distorted vocals. It invites loud listening but also easy to start clipping your amp(s) and your speakers will be tested as well as your ears. Not for the faint of heart. But a properly dialed system will reveal the depth and complexity of the recordings. Moderate/medium volume is the best way to listen. 

Skinny Puppy is indeed not for the (faint of heart).

;-)

For me, busy rock music with piano.  Large bands with many instruments competing for space are challenging to record and mix and then reproduce.  If this can be resolved then I'm always impressed.  

I started to say 'Piano' because of the dynamic range and impact of a Grand Piano. Tough to set up a system that can do this, even if you've found a recording where the engineer had succeeded.  But on reflection it has to be classical orchestral music where you have a much more apparent dynamic range and integration with a symphony hall as well as appropriate sound from different points in the hall. Too much to expect anything that is really live like. With a piano you can come a bit closer I think. One of the killers in orchestral recordings is the practice of spot lighting strings for example. Often bright to the extreme on a recording but rarely so in a live event. But its all good anyway.

The Dan! yes, the question could use refining though.  If the recording is bad, any kind of music. A bad pressing or mastering what ever.  If the recording is noisy or compressed or has background noise that makes it hard to play, or listen to.  If the recording is a good one,(any type of music, even country or Hair Metal), the stereo has a better time delivering it.  A quip comes to mind, "Don't shoot the messenger".  

So even the "Dan" as in Steely, can be a test for a system, but at what volume?  In what room, and so on.  

But to the assumed meaning of the question, I say Piano is the hardest to process for a stereo at any volume or in any room played by even a master!