I was an RA in an all-girl dorm building for a year, and we did not put up with random guys wandering the halls. Granted, it was also a Catholic school, and would have been harder to enforce in a coed building, but there should have been ways for you to express your concerns. :/