This has been an on-going topic for a very long time! @phantom_av makes a good point about the quality of the line stage; IME if a passive sounds better its telling you something about the line stage you used for comparison. IMO line stages have been in a sad state of affairs for a long time but if you get a good one there's no going back.
I don't like the problem were a passive sounds fine at full volume (which becomes the source driving the amp) but as you turn down the volume even a little bit the bass loses impact was well as the dynamic character overall.
If you never are able to turn the volume up all the way with a passive you may never find out what you are missing.
You can avoid this problem to a limited degree by using a control of a lower value for example 10K instead of 100K. But at that point a lot of sources will choke as they are not meant to drive impedances that low.
In addition you have to keep your cables short and be picky about what cables you use. I found out decades ago that if you can place your amps by the speakers and run short speaker cables you get an instant improvement in resolution across the board- but only if you can get the signal to the amps intact. I ran balanced lines for that and so had no problems. So for the last 30 years that is what I've done, and no worries about what cable I've used. They are 30 feet long!
That simply isn't something you can do if you use a passive control. If you have a smaller situation where a meter cable will be long enough then if you're careful (or lucky) you can get it to work quite well. My bedroom system employs a passive control built into the power amplifier which IME is the best place to put it.