@neverlast With full output at only 300mV your amplifier is far too sensitive. Since it can only make 8 Watts, it has to be used on a high efficiency speaker. So you'll be fighting noise. So yes, that's a bit of a mismatch!
Usually SETs (or SEP in this case) have low gain so as to deal with this issue.
I rarely recommend a passive setup but this appears to be a situation where it would be handy. OTOH, finding an amplifier of similar spec that has a more real-world input sensitivity would help with the noise and you may notice greater low level detail as a result.