Sound for my new theater?

"First priority is music but this is a full blown custom built theater with 12 seats and a fairly large volume."

Every time I've tried putting anything between my Eidolons the staging suffers.
I love my modest HT setup, thankfully it's in another room.

My suggestion is to change your priority to multi channel. Music videos are fun. Go with two smaller Velodyne DD subs for better system/room integration and room loading. Look into Wyred 4 Sound multi channel amplifiers. Blue Jean cable.

"Start over with a Magnepan based system using 3.6s as the mains with the Maggie center and surrounds. Also add sides for 7.1 and a Rel sub"

Have sold the Magnapan line, and have intimate experience with them. Down sides are going to be limited off-axis high quality sound (tiny sweet spot), VERY LIMITED dynamic output (low efficiency/sensitivity, more delicate speaker - not so great with heavy dynamic material, period). Also the Rel is going to do well in a small/medium room setup for modest music dubties, not a full blown HT system. The Rel will bottom out on you, and is a better music sub than it is a balls to the wall HT woofer (also, just one woofer? You'd need more!)

I'd recommend, yes, looking into more dedicated HT dynamic speakers, that are still musical, have a good focused propegation (Dappolitos), still offer excellent detail, and have tremendous dynamics! I'd be recommending possibly some sort of active speaker system if you could, otherwise something that's either higher efficiency, has a very simple passive crossover network (like 3 of the Triangle center speakers across the front - using smaller sides/rears, likely), maybe something with active woofers like Definitive makes, and forget about the linn system! Keep that in another system or whatever.
You must prioritize, and limit your compromises. I mean is it a system where mostly others will be joining you often, or mostly just you?
Keep in mind off-axis will compromise some stereo imaging, but multi-channel can make up for that, and you want balanced, high quality sound overall for lots of seats. Also, a center seat in such a room will still be in a bad acoustic spot. I'd setup for 4x4x4 seating if I were you, and you needed 12 seater.
Look at big centers from PSB, Dunlavy, Thiel, Triangle, Definitive (I like these with tubes in the system, FTR), and other higher efficiency systems. Also, horns like tubes, but do exceptional with dynamics and acoustical friendly applications!
Good luck

I had to make the compromise of HT and 2-channel occupying the same space. I can't stand programming HT technology with video based menus. I get enough of that from my PC, I want the audio to be simple. So for me, I went the most simple route I could while keeping 2-channel as priority. I use a Sony ES SACD/DVD, McCormack MAP-1 5 channel pre, Classe CAM-350 for the mains, Parasound A52 for center and rears. Thiel 3.6 for mains, MCS for center, and old JBL for rears (until I get Thiel in-walls). The system function very well as 2-channel. Turn on the Parasound and switch to TV for HT. I don't have bells and whistles or subwoofer slam for explosions. Most HT sound is recorded so poorly it doesn't matter.

Sgower

......"I don't have bells and whistles or subwoofer slam for explosions. Most HT sound is recorded so poorly it doesn't matter. "

If you mean it doesn't matter to you, I do understand.

If not though, Man! I really got to differ with you on the low level output quality in HT, movies, soundtracks, and lately cable TV shows… it’s getting better and better and has been good to very good, with any number of excellent ones, for some time. Get a sub and you will definitely see for yourself. Even if it’s not a great or expensive sub, adding one opens another door to video enjoyment you should not be missing out on.

The musical score in just about any adventure, action, deep drama, horror or sci fi, carries usually a very deep sub line or low freq accompaniment. Even next gen Star Trek, Stargate, and others have a pronounced low end component throughout the film for effect.

Low end effects are abundant and quite well laid in with many TV shows and films now. Even the Nat Geo, Smithsonian ch, Disc, History, Military, Science ch, and even the DIY ch, has low/sub effects with many of their programs.

Subs are a must for HT in a room of this proposed size. Like the very large screen size, subs will evoke as much of the sonic tables influence and impact. Cohesiveness of the surrounding Sonics provides pretty much just the involvement. In action adventure many cues too.

The only place where the musical end of things isn’t as up to par or important is with vintage films…. Pre 70s, mostly. Even some 80s as well. There I’ve found it best more often than not to utilize A Dolby Pro logic sound field, or an all ch audio/stereo feature. Many of today’s receivers & processors have built in to them… rather than just the center ch speaker doing all the work and being the focus.

In the classic films genre I’ll sometimes use a sound field choice such as theater, or church, which ever gives just a very slight reverb to emulate a large venue’s live characteristic, or introduce some ambience to an otherwise bland soundtrack.

Big screens need bigger if not better sound when possible, and in a space of size… spread it out. Even if it is only mono or just 2 ch. It is a nicer effect. IMHO

But there is a wealth of sub based info in TV and films these days for sure… having it in your bag of tricks is a must.

I think a sub adds value for music as well, especially if you enjoy pipe organ music. I supplement my KEF 104.2s below 80 Hz with a Velodyne HGS-15 and a Velodyne SMS-1 room correction module. It makes for a seamless integration that works well for music and HT. I use a vintage Proceed PAV/PDSD pre/pro with Proceed Amp 2 and Amp 3, so I use a separate HDMI switcher. The result is excellent audio and video.

db