You seem to be forgetting that 1. Virginia shares custody with her ex, so there are probably periods of time when her kids are with him, and when she left her kids with Shelley that night when Bill met him it was because Lillian was, you know, dying and all? and 2. the show is based on actual people who actually did…