Why have capacitors improved so much over the years?

Capacitors have improved in certain ways due to designers looking and wanting a different type of sound than previous versions. The change in sound is due to using different materials and construction. I can say that the early capacitors due not sound bad at all if given time to re form in an old component. You have to give some time before you judge an old capacitor especially in an amplifier or speaker because most of the time they are not bad at all they just need time to play and re form. If you do change the capacitor to a new one it can sometimes cause the gear or speaker in question to sound very different, especially bright at first, due to the new cap needing to form as well.

The best are still vacuum dielectric or air dielectric. They’ve been around forever, but they come in small values at high cost. I use these exclusively in my phono / pre.

Film and foil are next best; styrene is pretty neutral, teflon a bit bright, polypropylene a bit dull. Some examples are ’very’ rather than ’a bit’.

Metallized are constructed by depositing metal onto the dielectric. Quality appears to depend on the thickness of the metal and the junction of the metal to the wire leads. They can be almost as good as film and foil, or almost as bad as electrolytic. Mica and teflon are the best.

Ceramics tend to be inferior to metallized, the quality seeming to depend on the dielectric constant of the ceramic, lower being better.

I don’t have strong opinions on paper in oil, but I really don’t like leaks, so I avoid them.

Good luck!  YMMV

The big difference is in manufacturing tolerances.  Back in my early days in electronics (60s) 5% or even 10% tolerances in values were frequently used because it cost money to get to 1% tolerance or better.  MilSpec parts often had to measure far more accurately than consumer electronics.  The same was true for inductors and resistors.  The simple truth is that a 50 microfarad capacitor that accurately measure as 50 mfd will sound the same as any other that accurately measures as 50mfd.  Depending on the values of the other components in the circuit you cannot hear the difference, assuming they are all accurate to a given tolerance.  Usually, the differences in rated voltage and tolerance are used for the purposes of obtaining longevity and reducing failures due to arcing which results in a short circuit.

The other truth is that snake oil salesmen are far better psychologists than most university PHDs.  They have been around a long time because they know their craft and it works.  

WOW!  Short Answer!  "manufacturing process and technology, engineering !"  Hmm, Why are micro processors, HDD's and now SDD's Smaller Nowadays?? or the iPhone has a zillion more processing power than the computer that took us to the Moon..  Hmm, the Capacitor question is a Great one, I'll do a tad more research on that..