op, some good discussion on your post already - let me add my 2c to your points...
Q -- "And, it seems, consensus that certain brands, factories, vintages and specific types of tubes are better than their peers and are "worth" the extra money they command. Of course, the principle of diminishing returns applies here. Right?"
A -- yes, definitely and brent jesse’s writings and pricing of various brands on his website regarding the several common hifi audio tubes such as 6dj8, 12ax7/au7, 5ar4 detail this notion in spades
Q -- "Do measures of voltage gain, transconductance, plate resistance, noise and microphony tell the whole story? Are they sufficient for predicting performance?"
A -- necessary, but not sufficient (most common tube testers do not test the tubes in their full range of function), especially power tubes
Q -- "To what extent do they predict longevity?"
A -- only modestly well, to poorly sometimes, to the frustration of both seller and buyer
Q - "Or is longevity primarily a function of empirical collective experience? What about the materials science aspect of vintage vs. modern tubes?"
A - yes longevity is experiential on the part of the tube maker, and design, but also very much a function of the quality of materials used and screening done at the factory -- both were done a good bit better at the prime of tube production decades ago, rather than today, where much knowledge and experience has been lost and is being relearned, as mentioned in earlier posts - still, a learning curve, climbed again, is still a learning curve
Q -- "It seems that every company that produces tubes works hard to refine their design and balance innovation with "faithfulness to the vintage design", production processes, materials to make good sounding tubes. Even so, not all tubes from a given factory will perform the same, hence the testing and grading relative to the measures above and the consequent sorting into hierarchies of ascending levels of performance and price. "
A -- correct, there are youtube videos on tube making factories... watch those... you will understand -- screening and testing is key -- also as a tube making business, your production process quality and control is absolutely key, as rejects consume time and materials and do not yield sales revenues
Q -- "It also seems to me that performance will be closely tied to the circuit design and execution. A solid design would, it seems, be robust and produce excellent results with a wide range of tubes. A poor design might drive a tube in a way that is hard on the tube or requires a very specific technical characteristic for the circuit to perform well. Either of these situations could cause a user to come to completely different conclusions about what vintage/modern tube is "good" or excellent."
A -- correct - yes but in the day, we learned about how common tubes would perform in a variety of circuits and applications... everything was using tubes
"I have read some posts on this forum from certain individuals in which all "modern" or current production tubes or tubes produced by certain countries are dismissed out of hand. This seems like prejudice. Or it could, I suppose, reflect accumulated frustration with a given tube/brand/maker/etc as it performs in a given circuit."
A - such broad dismissals are usually based in prejudice, not fact, or based in old knowledge -- for example early chinese tubes were just terrible, in quality and in sound, but that was 20-30 years ago, they are trying to make a business work, so they improve and improve ... many of us have experiences over time, and in many many circuits, it becomes known certain circuits are hard on tubes, expose shortcomings sonically or in longevity quickly, but a bad tube is a bad tube and will show in many circuits over time - microphony is a trait that is exposed quickly, well before a tube actually ’fails’ to function
hope this point by point reply helps some, i tried to keep it pithy - will not get into my personal opinions of what tubes are good and poor, from where, where the sweet spots on the various tube type value curves are etc etc... that would be a book