Castle Black is depicted as a single blocky monolithic building on the map, but it isn't a castle in the traditional sense. In the book it is described as a pretty random collection of a variety of buildings that sprawl out around the gate through the Wall.