@shalmaneser – Thanks for responding with all that detail! If you want to stay with using your computer, that’s totally find. I can respect that. Though, I will say that I have been there with you. Many years ago, I used to run an HTPC for all my audio/video processing, with custom modified video cards using BNC RBGHV outputs and specialized sound cards. I have then moved on to dedicated components, which just worked so much better.
That being said, I have experience with what you are doing. I will say that the optical outputs on these motherboards are really poor. The reason is lack of good digital clock and lack of good local power supply filtering. Also, the optical interface is lacking when compared to digital coax. I would highly recommend getting a good sound card with spdif output. Since you are running Windows 10, the only one I can see that supports Windows 10 drivers is the Xonar Essence ST / STX / STX II. Any one of these will work. They all have two crystal clocks that are based on the two primary audio sampling rates (which are 44.1/88.2/176.4 or 48/96/192). I think I read the older ST PCI card had better digital clock section but they would require an open “PCI” slot. The STX models are the newer PCIE cards. These are going on ebay for $50-90. Then get that 6 foot Blue Jean Beldon COAX cable. I’ve tested the motherboard optical outputs and they just sound weak in comparison. The idea that “it’s just data until it hits the DAC” is not an accurate statement here. The VLC player should have a configuration that allows you to send “spdif passthrough” on the digital spdif output of the sound card (even for AC3 and DTS movie bitstreams).
At this point, I think you will need to decide if you are wanting to keep your Sony processor and get an amplifier, or look for a really high-end receiver. Just keep in mind that Sony is definitely a limiting factor because it won’t even do DTS (which is the format that is most used nowadays). The Bluray DTS-MA HD will be automatically downconverted to normal oldschool DTS before it is sent out on the spdif digital output. The only digital connection that supports hi-res bluray is HDMI. You could get a video card with HDMI output and run it through a new receiver with HDMI inputs if you wanted to truly decode hi-res bluray audio. Though, it may not sound better unless the receiver was up to par.