"Is for the subs to be wired/operated in mono mandatory in a swarm sub system?"
Not at all.
"In other words, does the swarm not "work" (reduce/minimize bass peaks and nulls in the room) if the subs are run stereo?"
Works just fine in stereo. The only reason my commercial Swarm system is normally mono is, I can hit a more attractive price point by driving all four units with a single amplifier. As an option the Swarm can be supplied with a second amplifier.
"Or is it just a matter of that with a steep enough filter (say 4th-order, 24dB/octave), there is nothing to be gained by running the subs stereo?"
If the bass is summed to mono on the recording anyway, which I believe to almost always be the case, then imo there is nothing to be gained by running the subs in stereo. But imo there is something to be gained (greater sense of envelopment) by using introducing a roughly 90 degree phase difference between the subs on the left-hand side of the room and the subs on the right-hand side of the room, and doing so requires two amplifiers, in which case you might as well do stereo. That way if you know you have a recording with true stereo separation all the way down, you can quickly dial back in normal phase using the phase controls on the amps.
"Is running two subs at the front of the room (in the neighborhood of the loudspeakers) in stereo, and the third and fourth in mono, a viable option? THAT would make using a shallower low-pass x/o filter on the subs (for whatever reason) possible."
You can do that, but imo it doesn’t really make a shallower low-pass filter more feasible. Imo the main reason for using a steep low-pass filter is to prevent the subs from passing upper bass and lower midrange energy loud enough to betray their locations.
If your subs have fairly shallow (12 dB/octave) built-in lowpass filters, you might try this: Use a lower low-pass filter frequency for the subs that are closer to the listening area, since these are the ones most likely to be heard as separate sound sources in the upper bass/lower midrange region.
Duke