Subwoofer boom is too much for me...

Something like this is better. Available in most size.

 

less likely your port is the problem and more likely you have incorrect slope and x-over frequency relative to your main speakers. fixable to a certain extent. 

Mikeydee: Sure plug the bung hole. Let us know it it works on your subs. 

For music, non ported subs should be next on your list.

+1 @baylinor 

I switched from ported (for literally 25 years) to sealed and am much happier with the sound.