Now I get it...

Sometimes bad recordings can have endearing qualities that you will eventually recognize.

I wonder if you can purchase a more forgiving “musical” cartridge for poorer recordings?  

Consider ultrasonic cleaner if you have not already. Especially if you bargain bin and listen to older records. You'd be surprised how much ground dust accumulates in 40-60 years. Though that dust could tell some pretty cool party stories.

Congratulations.
I have done the same thing resently. But in a much smaller scale. And it goes in waves.
I have done most of the things I could do in the vinyl playback chain. And that is a lot over several years (!).
I almost gave up. The magic were there and were gone and now it is back in full effect.
To illustrate what I have done to get there:
Löfgren B Arc protractor
Dynamic adjustment and by ear SRA
Peter S anti-scating procedure
Speed adjustment while playing
Good ML stylus and cartridge(s)
Redraw tonearm cables with pure silver wire
Replaced RCA cables with known uF fully shielded
Upgrading dedicated power supplies
3 different RIAA Amps tested
Passive pre amp with coils
Build my own power amplifiers (driven by car battery)
Pure silver dual shielded ala chord DYI speaker wire
Bookshelves upgraded with top of the line crossover components
And dual 8" sealed sub
Dedicated room with sub 300ms decay times
Treated reflections points
Base traps
All lined up perfectly to the wall boundaries

Now I lend my RIAA to a friend. I had another RIAA in the meantime that the op-amps I had upgraded to audiophile grade developed for hi-fi MUSES 01. That were superior too the original 
OPA2111KP.

But here is the catch.. I ordered fully descret op-amps to try them out. And surprise they were better than the MUSES! And the they played one octave lower than the MUSES couldn't.
But they also made vinyl playback so much more enjoyable like your experience. 

So I have not powered on my new streamer and DAC yet since then.. and just enjoy my vinyl collection for the moment.

And it took all those years as in a roller coaster to get too this point.. 

Congratulations on your sonic breakthrough!! In my experience the most common reason for a recording to sound bad is that it is not a first pressing.  That is not to say that bad recordings don't exist-there are a lot of them.  But, the least crummy sounding will be a first press, and sometimes the differences are astounding. Discogs is a great site to determine true first pressings. I truly enjoy "the hunt" in finding replacement vinyl for my non-first presses. I agree with Uberwaltz that you need to stick to EX/ NM grading when purchasing used records on line, and even so, most grading is visual and doesn't always translate into a quiet record. I agree with the advice to largely avoid reissues.  With tape degradation many have lost the magic.  The Beatles reissues however are glorious!  Ultra clean records are also a must.  Besides removing a lot of noise, a veil is usually lifted revealing more detail and a better sense of "being there". Having gone trough the entire spectrum of cleaning options, I feel comfortable stating that Ultrasonic cleaning seems to work the best, but that is a topic for another thread.  Enjoy!!