Supposedly she also refused to come back for flashbacks in Season 3, which stunted the plan for Libby (maybe she was meant to be Annie?)