My guess is that, when tubes were used in everything, there were a lot of manufacturers competing, and this competition resulted in superior product. Also, we tend to now collect and praise the very best of the old lot, and in that sense, older tubes have been cherry-picked (the crap models having long ago been trashed).
In some cases, key elements are just no longer manufactured, either because the scale of manufacturing is no longer there, or the product is now deemed hazardous. There is some speculation that the Chinese have recently stopped manufacturing the 845 tube because they were relying on old stock of Thoriated Tungsten filaments that are no longer available. I have also heard that key parts of the currently manufactured Western Electric tubes are also "old stock" and manufacturing will cease when those parts run out.
Some preference for old tubes is just a matter of personal taste and system matching. For example, a lot of people covet red-base 5692s. In my system, and for my taste, they are NOT right (I like Tungsol roundplate 6sn7s). The much larger variety of older tubes (again the result of ubiquity of tube use) means a greater chance of finding the right match amoung older varieties than the limited choices of new tubes.
I personally have found a few new tubes to be very good. I am a BIG fan of EML meshplate 2a3s. These are the best sounding 2a3s I've heard, though they MUST be driven gently to survive.
As for the comment above that well designed components ought to be insensitive to variations in tubes, that make so sense at all. With good equipment, minor differences in any component, including tubes, SHOULD be discernable --if you can't hear the difference it is because detail is being lost in the muddle of crappy reproduction.
Also, the suggestion that people who like old tubes might be enjoying "gassy" tubes is completely unsupported. Where is the evidence that all older tubes are gassier than new tubes because of infiltration? Sure, some may go bad over time, but, on what basis does one conclude that those are the ones coveted by those who favor older varieties? I have a tube tester that measures for this condition. The older tubes I use are NOT gassy. As far as 6sn7s are concerned, it is actually some currently manufactured 6sn7 that measure as gassy.
