First, a correction: Pet Sounds was McCartney’s inspiration for not Rubber Soul, but Sgt. Pepper. I get them confused because Rubber Soul was Brian’s inspiration for Pet Sounds, Brian feeling RS was the first album ever with no filler.
To put things in context, when Pet Sounds was released, the Beach Boys had already had three low-selling albums in a row---Today, Summer Days (and Summer Nights), and Party. After their last hit album, 1964’s All Summer Long, Rock n’ Roll had begun turning into Rock, and had moved in a decidedly tougher, harder direction. Even more than the Popish Beatles, The Rolling Stones, The Kinks, The Animals, The Yardbirds (featuring first Eric Clapton and then Jeff beck on guitar), and The Who were playing more "adult" music based on American Blues and R & B. Dylan's lyrics had completely transformed the nature of the music's lyrics, no longer being about romantic relationships, but about society, morality, hypocrisy, and values. The All-American, squeaky-clean, surf, cars, & girls teenage adolescent sound and image of the Beach Boys had become quite passe’, completely irrelevant to the about-to-emerge counter-culture. Pot had made it into the suburbs, and High School kids were adopting a cynical semi-adult attitude and posture. The Beach Boys were largely forgotten at the time Pet Sounds came out in ’66, already viewed as an Oldies act, as much so as 1950’s entertainers such as The Platters.
I didn’t know anyone who bought Pet Sounds when it was released. A new Beach Boys album? Who cares! A lot was happening at that time---a lot of new Groups, Bands, and records to keep up with. But then 1967 came around, and "Good Vibrations" was blasting out of every teenagers’ car radio. Considering how interesting that hit single was, the album it was on, Smiley Smile, needed to be heard. I was astonished when, on my Koss Pro 4AA headphones plugged into my Fisher X-100A integrated tube amp, I heard what was in the grooves of the LP being played by my Shure M44 cartridge mounted on my Garrard SL55 turntable. "Good Vibrations" was just the tip of a very weird iceberg!
Gone were the sun and fun of The Beach Boys I had seen live just three years earlier in 1964 (my first concert, one year before seeing The Beatles), replaced by a very dark, very odd, introverted creepiness. Hearing it in the dark made me feel like I had descended below ground (perhaps through a tree trunk, as did Peter Pan and the lost children), now being in a small earthen cave, dimly illuminated by candles, where none of the assembled persons spoke. The "songs" featured highly unusual chords and chord progressions (one subject discussed by Bernstein during the special, as well as Brian’s melodic and harmonic sophistication) played mostly by muted piano, harpsichord, bass, and snare drum. On top of that was surreal lyrics being almost whispered by strange hushed voices, very, very odd harmonies and lots of counter-point, primal chanting, and sound effects. It actually scared me, and still does. Very spooky, like walking into a Victorian mansion with a very musty smell at twilight, eerily quiet except for an old clock softly ticking off in the distance. It made the "experimental" Bands popular at the time seem very normal, very ordinary. I played the album for all my close friends (musicians, artists, intellectuals, weirdos), and Smiley Smile became the favorite album of the smartest people I knew---an elite club sharing a secret known only by it’s members. The album became a test for new friends; "get" the album and you were in.
Having heard Smiley Smile (though without yet knowing the whole Smile saga), it was only natural to go back and listen to Pet Sounds. Though not nearly as strange and interesting as SS, it too was a very welcome discovery. It’s songs are more formal, classically-structured Pop songs, and some real, real good ones. As much as I like them (especially "God Only Knows" of course!), I find Smile/Smiley Smile a more important/amazing album. Smile was never completed or released (Brian, as Icarus, flew too close to the Sun---via LSD etc. He had a complete mental/emotional/physical breakdown without completing the album, intended to be a musical representation of the American Manifest Destiny. Rather ambitious! His bother Carl patched together Smiley Smile for release in it’s place), but it’s bits and pieces were recently assembled into what it was to have been, and is available in a number of configurations. I find the 6-CD boxset a bit much for most people, and recommend the 2-CD version.