Racist white people didn't want to live in neighbourhoods with non-white people, and feared that non-white families in the neighbourhood would negatively impact property values. If I'm not mistaken, this is the root of the colloquial expression "there goes the neighborhood".