"What material would be best to use for an appropriate stuffing?"
I'm not sure. With the towel, what you did was add flow resistance to the port; you made it significantly harder for the low frequency energy to get out of the port. If the towel was filling the port completely, then you added a lot of flow resistance; the more airspace around it, the less flow resistance.
My suggestion is, trial and error. Bigger towel, smaller towel, see which is a step in the right direction. Maybe open-cell foam (when you "haaaaah" into the foam, can you feel the heat form your breath on the other side? If so, then it' open-cell foam), again I can't tell you how completely you should fill port, or how deep the foam should extend into the port.
Long-hair wool was mentioned. That's great for stuffing a transmission line, but imo the intention here is different - we're trying to reduce low bass output, rather than facilitate it. Unless you packed it tightly into a cloth bag and crammed it into the port - then, it would be behaving more as flow resistance than as gentle damping along the length of the line. Imo you can accomplish the same thing for less money with tightly packed polyester batting or fiberfill.
The variovent is itself a flow resistance device, and one with a rather strong flow resistance, maybe more like packing the port tightly with a towel than like open-cell foam. You'd want to turn the cabinet almost into a sealed box, with the only pathway between the inside and outside being through the variovent (which is dense fiberglass compressed inside a plastic housing that has airflow holes in it).
Duke
