It‘s precisely the initial pulse of the wave that defines phase, or for the purposes of this discussion more accurately polarity. Think of a wind instrument being blown rather than sucked ( in analogy to a speaker Diaphragm moving out rather than being sucked in with the corresponding air flow. And it is audible despite what certain theorists claim ex cathedra.
Just as an aside: this is the major advantage of a point source over multiple speaker units interfering with each others‘ phase.