Why can't all/most recordings be this good?

Changing components this weekend (swapping out my MF Trivista 300 for a Hovland hp 100/Bel 1001 Mk V fronting Meadowlark Blue Heron II’s), I picked some of my cd’s that had, what I consider, exemplary sound. I listened through the day & evening, marveling at the crisp, delineated, but musical sound but also began to question: Why aren’t all/most recordings done in this manner?
I am not a sound engineer nor musician. I have no knowledge of mixing, microphone placement, layering, etc. But some recordings are so outstanding in their clarity, instrumentation, and pure enjoyment that many others pale in comparison. Granted, I am dealing in only one aspect (redbook cd) of media, and only with a single genre of music. And I certainly didn’t compile an exhaustive list of recordings; just ones I listened to of late. I would certainly like to add to the list and hear other’s opinion.

Lenard Cohen Live in London, 10 More Songs
Rickie Lee Jones Rickie Lee Jones
Aaron Neville Warm Your Heart
Chris Izak Baja Sessions
Allison Krause and Union Station Live
Govi No Strings Attached
I HAVE Govi No Strings Attached AND Chris Izak Baja Sessions
It is sad but true that there seems to be no correlation between sound quality and record sales, therefore there is no economic incentive to do records better. I have heard of several instances in which record companies declined to use better manufacturing techniques even though the cost increases would be nominal. Some artists have both the power and inclination to demand good sound quality, most lack one or the other.
not everyone's cup of tea (it's 'artsy'), but Laurie Anderson's 'Life on a String' (nonesuch) is one of my favorite 'demo' CDs - astonishing sound for 'redbook'.
Most artists/producers could care less as long as the product sells.Just my opinion.
Generally the Doug Sax masters are in the top quartile (he has a huge list of credits on Artists Direct).

Other good engineers
Bill Schnee
Chuck Ainslay
Al Schmitt
Alan Parsons
James Guthrie
Mark Bishop
George Massenburg
Ed Cherney
Elliot Schiener

There are plenty of others of course but the above list probably accounts for more than 2/3 of the mainstream grammy award winning productions from big names.
CDs which are mastered for radio/ipod are usually compressed and can sound awful. Some artists are motivated this direction others to top sound. Engineer is directed by label or artist.
Lou Reed's Perfect Night in London is amazing. Best sound quality of any rock CD I have (and I have over 600).
Because they are, for starters, compressed. Maybe a new labeling requirement should be whether they are or not compressed. Often, if you compare an issued Cd againist the same music on an SACD, the difference will leap at you.

Lastly, some engineer's do it better, with perhaps Steve Hoffman being the tops. He is doing Nate Cole for SACD, worth checking out.
Larry Carlton - Greatest Hits Rerecorded
Gordon Goodwin - Big Phat Band (any of them)
Jimmy Buffet - JVC XRCD Songs you should know by heart
Allison Krause - New Favorite (SACD)
Neil Young - Prairie Wind; Live at Massey Hall
James Taylor - Live
John Mellencamp - Life and Death
Roy Orbison & Friends - Black and White Night
I suspected some were just stamped out for mass production and others were directed by the artist(s) to have a quality, audiophile flavor. Mlauner seems to agree. Will check out L. Anderson, don't have any of her work. Appreciate the names of the higher quality engineers, Shadorne; these are the guys doing the mixing and arranging of the final product, right?
Good suggestions on other cd's to try.
I think a lot of it does have to do with the fact that great recordings do not necessarily result in higher sales. And less than great recordings do not necessarily hinder high sales.
Because, hardly anyone in the recording business gives a hoot about a few audiophiles, the sound they want is the sound that will sound good and is catered for the masses that use cheapo cd players, alas they will sound awful in your mega buck system, and not surprisingly will not sound bad played through a ghetto blaster. We few, who require quality recordings to show off our gear are merely like a grain of sand in the Sahara compared to those that listen to music with lesser gear and could not give a chuff about speakers that cost more than a house.
Most classical music and possibly jazz also offer the best quality recordings for us few, well they do in my collection anyway.
NB-That's not to say all recordings of other genres sound bad.
+1 for Strange Angels by Laurie Anderson. Still love it after many years, and it also makes a great test CD - Monkey's Paw in particular.
"the sound they want is the sound that will sound good and is catered for the masses that use cheapo cd players"

Very true. An audiophile friend of mine once visited a well-known recording studio and complimented the recording engineer on the great speakers they had. The engineer pointed to an unnoticed pair of crappy little speakers and said that's what he mixed it for, because that's what most people heard it on.
For jazz, I've been having really good luck with the Highnote label. their recordings are excellent. Some are recorded in the Van Gelder studio, which - I'm happy to report - seems to have a new piano.