Have you called tech support?
I'm not sure how you have Audacity set up, but on my Windows PC I only have three choices for the audio host when recording audio - WASAPI, DirectSound, and MME. I use WASAPI.
Here's a link to the section in the manual about the Device Toolbar - https://manual.audacityteam.org/man/device_toolbar.html
And here's a link to the tutorial on recording computer playback - https://manual.audacityteam.org/man/tutorial_recording_computer_playback_on_windows.html#wasapi
Hopefully these will help you sort out your problem.
One other thing - I looked over the Nu Audio software page and I see it has options for SPDIF or speakers. Try using the speaker output and setting channels to two if you haven't already. Maybe that will give better results.