Rape is sex in the absence of consent, so you do not have to say no, its rape if you don't say yes. Some state laws, however, say there must be force used, but others do not, in NJ it would definitely be rape because no force or threat of force is required here.