@jaytor --
Kudos for venturing into outboard active configuration (it’s what I do myself). It will be interesting to learn of your findings (if you’re willing to share them) compared to your passively configured main speaker system once you’ve initiated the process and has gotten your head around it with actual impressions of the sonics to follow. Cool main setup, btw. - I’m sure it’s very capable.
My biggest reservations are 1) giving up my Denafrips Terminator+ DAC and nice-quality DIY preamp, and 2) using the DAC’s digital volume adjustments.
Your main priority is using the digital input on the dspNexus, I see, which - as a simplified system, with all that may entail - is what prompts above quoted concerns of yours. While on paper/in theory I understand worrying about introducing extra conversion processes, as an outset at least I wouldn’t be too concerned about them. What do you gain by avoiding them, while at the same time losing out on your Denafrips DAC and separate preamp?
Seems to me some effort of yours has been invested in sonically "shaping" the sound of your setup with these very components, so why not start with them remaining in the chain and instead use the analogue inputs of the dspNexus directly from your preamp? Yes, you’ll have an extra A/D conversion (the effect of which is blown out of proportion, if you ask me, if it’s even noticeable), but this way I’d argue you have better grounds for comparing the sound from the passive speaker config. and the one introduced with the active ditto on the basis of this alone.
There’s also the option of a software-based DSP preceding the DAC, which then necessitates a separate DAC channel for each of the amp dittos - like from the Okto Research dac8 PRO. This to some can bring into question whether a theoretical lack of overall signal symmetry of the DAC channels may have an effect on perceived sonics vs. only using 2 DAC channels that are then branched out on the output side of the digital crossover. I gather it’s in the splitting hairs department, but I have no experience to speak of here - not via my own setup, that is.
Regarding setting filter values good advice has been provided already. I’d experiment with steeper slopes than 4. order - myself I use 36dB/octave L-R throughout (BW slopestyle on the subs HP). Does the dspNexus use linear phase filters (FIR)? If not it’s not an issue, as I see it. In any case I gather you’ll be able to get even better integration between you sub and main towers fiddling with delay, although you may come about this by simply moving them back and forth to each other the way you've positioned them. With my speakers we used nearfield measurements of the MF/HF horns to set precise filter notches and make a light peak suppression. On top of that the real work was getting the delay right - not least between the woofer section of my main speakers and the tapped horn subs, where the "origin" of the front wave is somewhere inside the horn path, and not simply on the cone backside exposed at the mouth. With the tools at your disposal already I’d say you have a very good outset for interesting results.