Outside of doing harm to the components, would there be any detrimental effect on the sound if I were to go outside of the manufacturers recommended specs?No. Just as analogluvr stated above, sometimes you just need to try it and listen.
Also, the load difference between 100 and 400 ohms is pretty small and you might not even hear a difference. Depends on the cart and all the associated gear.
I always try to get close to the recommended load, but you never know.
I have owned the same exact unit you have and many, many more that you can replace the resistors in. You can drive yourself crazy, if you get to concerned about it.